Hugging Face's logo

IDEA-CCNL
/
Erlangshen-UniMC-RoBERTa-330M-Chinese

Fill-Mask
Transformers
PyTorch
bert
classification
zero-shot
Inference Endpoints
6 papers
Model card Files Community
1
Erlangshen-UniMC-RoBERTa-330M-Chinese / README.md
suolyer's picture
suolyer
Update README.md
bdf698b
preview code
|
raw history blame
8.05 kB
metadata
license: apache-2.0
tags:
  - classification
  - zero-shot

Erlangshen-RoBERTa-330M-UniMC-Chinese

简介 Brief Introduction

将自然语言理解任务转化为multiple choice任务，并且使用 48 个 NLU 任务进行预训练

Convert natural language understanding tasks into multiple choice tasks, and use 48 NLU task for pre-training

模型分类 Model Taxonomy

需求 Demand 任务 Task 系列 Series 模型 Model 参数 Parameter 额外 Extra
通用 General 自然语言理解 NLU 二郎神 Erlangshen RoBERTa 330M Chinese

模型信息 Model Information

我们为零样本学习者提出了一种与输入无关的新范式，从某种意义上说，它与任何格式兼容并适用于一系列语言任务，例如文本分类、常识推理、共指解析、情感分析。我们的方法将零样本学习转化为多项选择任务，避免常用的大型生成模型（如 FLAN）中的问题。它不仅增加了模型的泛化能力，而且显着减少了对参数的需求。我们证明了这种方法可以在通用语言基准上取得最先进的性能，并在自然语言推理和文本分类等任务上产生令人满意的结果。

We propose an new paradigm for zero-shot learners that is input-agnostic, in the sense that it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense reasoning, coreference resolution, sentiment analysis. Our approach converts zero-shot learning into multiple choice tasks, avoiding problems in commonly used large generative models such as FLAN. It not only adds generalization ability to the models, but also reduces the needs of parameters significantly. We demonstrate that this approach leads to state-of-the-art performance on common language benchmarks, and produces satisfactory results on tasks such as natural language inference and text classification.

下游效果 Performance

Few-shot

Model eprstmt csldcp tnews iflytek ocnli bustm chid csl wsc Avg
Finetuning 65.4 35.5 49 32.8 33 60.7 14.9 50 55.6 44.1
PET 86.7 51.7 54.5 46 44 56 61.2 59.4 57.5 57.44
LM-BFF 85.6 54.4 53 47.1 41.6 57.6 61.2 51.7 54.7 56.32
P-tuning 88.3 56 54.2 57.6 41.9 60.9 59.3 62.9 58.1 59.91
EFL 84.9 45 52.1 42.7 66.2 71.8 30.9 56.6 53 55.91
UniMC-110M 88.64 54.08 54.32 48.6 66.55 73.76 67.71 52.54 59.92 62.86
UniMC-330M 89.53 57.3 54.25 50 70.59 77.49 78.09 55.73 65.16 66.46
UniMC-1.3B 89.278 60.9 57.46 52.89 76.33 80.37 90.33 61.73 79.15 72.05

Zero-shot

Model eprstmt csldcp tnews iflytek ocnli bustm chid csl wsc Avg
GPT-zero 57.5 26.2 37 19 34.4 50 65.6 50.1 50.3 43.4
PET-zero 85.2 12.6 26.1 26.6 40.3 50.6 57.6 52.2 54.7 45.1
NSP-BERT 86.9 47.6 51 41.6 37.4 63.4 52 64.4 59.4 55.96
ZeroPrompt - - - 16.14 46.16 - - - 47.98 -
Yuan1.0-13B 88.13 38.99 57.47 38.82 48.13 59.38 86.14 50 38.99 56.22
ERNIE3.0-240B 88.75 50.97 57.83 40.42 53.57 64.38 87.13 56.25 53.46 61.41
UniMC-110M 86.16 31.26 46.61 26.54 66.91 73.34 66.68 50.09 53.66 55.7
UniMC-330M 87.5 30.4 47.6 31.5 69.9 75.9 78.17 49.5 60.55 59.01
UniMC-1.3B 88.79 42.06 55.21 33.93 75.57 79.5 89.4 50.25 66.67 64.53

使用 Usage 

git clone https://github.com/IDEA-CCNL/Fengshenbang-LM.git
cd Fengshenbang-LM
pip install --editable .

import argparse
from fengshen import UniMCPiplines


total_parser = argparse.ArgumentParser("TASK NAME")
total_parser = UniMCPiplines.piplines_args(total_parser)
args = total_parser.parse_args()
args.pretrained_model_path = 'IDEA-CCNL/Erlangshen-RoBERTa-110M-UniMC-Chinese'
args.learning_rate=2e-5
args.max_length=512
args.max_epochs=3
args.batchsize=8
args.default_root_dir='./'
model = UniMCPiplines(args)

train_data = []
dev_data = []
test_data = [
        {"texta": "放弃了途观L和荣威RX5，果断入手这部车，外观霸气又好开", 
         "textb": "", 
         "question": "下面新闻属于哪一个类别？", 
         "choice": [
            "房产", 
            "汽车", 
            "教育", 
            "科技"
            ], 
         "answer": "汽车", 
         "label": 1, 
         "id": 7759}
    ]

if args.train:
    model.fit(train_data, dev_data)
result = model.predict(test_data)
for line in result[:20]:
    print(line)

引用 Citation

如果您在您的工作中使用了我们的模型，可以引用我们的论文：

If you are using the resource for your work, please cite the our paper:

@article{fengshenbang,
  author    = {Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen and Ruyi Gan and Jiaxing Zhang},
  title     = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
  journal   = {CoRR},
  volume    = {abs/2209.02970},
  year      = {2022}
}

也可以引用我们的网站:

You can also cite our website:

@misc{Fengshenbang-LM,
  title={Fengshenbang-LM},
  author={IDEA-CCNL},
  year={2021},
  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}