Update README.md

README.md

@@ -10,15 +10,15 @@ tags:
 ---
 
 # Erlangshen-Albert-235M-UniMC-English
-
+- Paper: [Zero-Shot Learners for Nature Language Understanding via a Unified Multiple Choice Perspective](https://github.com/IDEA-CCNL/Fengshenbang-LM)
 - Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
 - Docs: [Fengshenbang-Docs](https://fengshenbang-doc.readthedocs.io/)
 
 ## 简介 Brief Introduction
 
-首个开源的中文CLIP模型，1.23亿图文对上进行预训练的文本端RoBERTa-base
+将自然语言理解任务转化为multiple choice任务，并且使用14个机器阅读理解数据进行预训练
 
-The first open source Chinese CLIP, pre-training on 123M image-text pairs, the text encoder: RoBERTa-base.
+Convert natural language understanding tasks into multiple choice tasks, and use 14 machine reading comprehension data for pre-training
 
 ## 模型分类 Model Taxonomy
 
@@ -28,62 +28,61 @@ The first open source Chinese CLIP, pre-training on 123M image-text pairs, the t
 
 ## 模型信息 Model Information
 
-我们遵循CLIP的实验设置，以获得强大的视觉-语言表征。在训练中文版的CLIP时，我们使用[chinese-roberta-wwm](https://huggingface.co/hfl/chinese-roberta-wwm-ext)作为语言的编码器，并将[open_clip](https://github.com/mlfoundations/open_clip)中的**ViT-L-14**应用于视觉的编码器。为了快速且稳定地进行预训练，我们冻结了视觉编码器并且只微调语言编码器。此外，我们将[Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/)数据集(100M)和[Zero](https://zero.so.com/)数据集(23M)用作预训练的数据集。在悟空数据集和zero数据集上预训练24轮。据我们所知，我们的Taiyi-CLIP是目前Huggingface社区中首个的开源中文CLIP。
+我们为零样本学习者提出了一种与输入无关的新范式，从某种意义上说，它与任何格式兼容并适用于一系列语言任务，例如文本分类、常识推理、共指解析、情感分析。我们的方法将零样本学习转化为多项选择任务，避免常用的大型生成模型（如 FLAN）中的问题。它不仅增加了模型的泛化能力，而且显着减少了对参数的需求。我们证明了这种方法可以在通用语言基准上取得最先进的性能，并在自然语言推理和文本分类等任务上产生令人满意的结果。
 
-We follow the experimental setup of CLIP to obtain powerful visual-language intelligence. To obtain the CLIP for Chinese, we employ [chinese-roberta-wwm](https://huggingface.co/hfl/chinese-roberta-wwm-ext) for the language encoder, and apply the **ViT-L-14** in [open_clip](https://github.com/mlfoundations/open_clip) for the vision encoder. We freeze the vision encoder and tune the language encoder to speed up and stabilize the pre-training process. Moreover, we apply [Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/) dataset (100M) and [Zero](https://zero.so.com/) dataset (23M) as the pre-training datasets. The model was first trained 24 epochs on wukong and zero. To the best of our knowledge, our TaiyiCLIP is currently the only open-sourced Chinese CLIP in the huggingface community.
+We propose an new paradigm for zero-shot learners that is input-agnostic, in the sense that  it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense  reasoning, coreference resolution, sentiment analysis.
+Our approach converts zero-shot learning into multiple choice tasks, 
+avoiding problems in commonly used large generative models such as FLAN. It not only adds generalization ability to the models, but also reduces the needs of parameters significantly. We demonstrate that this approach leads to state-of-the-art performance on common language benchmarks, and produces satisfactory results on tasks such as natural language inference and text classification.
 
 ### 下游效果 Performance
 
 **Zero-Shot Classification**
 
-|  model   | dataset  | Top1 | Top5 |
-|  ----  | ----  | ---- | ---- |
-| Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese  | ImageNet1k-CN | 55.04% | 81.75% |
-
-**Zero-Shot Text-to-Image Retrieval**
-
-|  model   | dataset  | Top1 | Top5 | Top10 |
-|  ----  | ----  | ---- | ---- | ---- |
-| Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese  | Flickr30k-CNA-test | 58.32% | 82.96%  | 89.40% |
-| Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese  | COCO-CN-test | 55.27% | 81.10%  | 90.78% |
-| Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese  | wukong50k | 64.95% | 91.77% | 96.28% |
+| Model   | T0 11B | GLaM 60B | FLAN 137B | PaLM 540B | UniMC 235M |
+|---------|--------|----------|-----------|-----------|------------|
+| ANLI R1 | 43.6   | 40.9     | 47.7      | 48.4      | 52         |
+| ANLI R2 | 38.7   | 38.2     | 43.9      | 44.2      | 44.4       |
+| ANLI R3 | 41.3   | 40.9     | 47        | 45.7      | 47.8       |
+| CB      | 70.1   | 33.9     | 64.1      | 51.8      | 75.7       |
 
 ## 使用 Usage
 
 ```python3
-from PIL import Image
-import requests
-import open_clip
-import torch
-from transformers import BertModel, BertConfig, BertTokenizer
-from transformers import CLIPProcessor, CLIPModel
-import numpy as np
-
-query_texts = ["一只猫", "一只狗",'两只猫', '两只老虎','一只老虎']  # 这里是输入文本的，可以随意替换。
-# 加载Taiyi 中文 text encoder
-text_tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese")
-text_encoder = BertModel.from_pretrained("IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese").eval()
-
-url = "http://images.cocodataset.org/val2017/000000039769.jpg"  # 这里可以换成任意图片的url
-# 加载openclip的image encoder
-clip_model, _, processor = open_clip.create_model_and_transforms('ViT-L-14', pretrained='openai')
-clip_model = clip_model.eval()
-
-
-text = text_tokenizer(query_texts, return_tensors='pt', padding=True)['input_ids']
-image = processor(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0)
-with torch.no_grad():
-    image_features = clip_model.encode_image(image)
-    text_features = text_encoder(text)[1]
-    # 归一化
-    image_features = image_features / image_features.norm(dim=1, keepdim=True)
-    text_features = text_features / text_features.norm(dim=1, keepdim=True)
-    # 计算余弦相似度 logit_scale是尺度系数
-    logit_scale = clip_model.logit_scale.exp()
-    logits_per_image =  logit_scale * image_features @ text_features.t()
-    logits_per_text = logits_per_image.t()
-    probs = logits_per_image.softmax(dim=-1).cpu().numpy()
-    print(np.around(probs, 3))
+import argparse
+from fengshen import UniMCPiplines
+
+
+total_parser = argparse.ArgumentParser("TASK NAME")
+total_parser = UniMCPiplines.piplines_args(total_parser)
+args = total_parser.parse_args()
+args.pretrained_model_path = 'IDEA-CCNL/Erlangshen-Albert-235M-UniMC-English'
+
+train_data = []
+dev_data = []
+test_data = [
+        {"texta": "Linguistics is the scientific study of language, and involves an analysis of language form, language meaning, and language in context. The earliest activities in the documentation and description of language have been attributed to the 4th century BCE Indian grammarian Pāṇini, who wrote a formal description of the Sanskrit language in his Aṣṭādhyāyī .", 
+         "textb": "", 
+         "question": "Based on the paragraph", 
+         "choice": [
+            "we can infer that Form and meaning are the only aspects of language linguistics is concerned with.", 
+            "we can not infer that Form and meaning are the only aspects of language linguistics is concerned with.", 
+            "it is difficult for us to infer that Form and meaning are the only aspects of language linguistics is concerned with."
+            ], 
+         "answer": "we can not infer that Form and meaning are the only aspects of language linguistics is concerned with.", 
+         "label": 1, 
+         "id": 0},
+    ]
+
+model = UniMCPiplines(args)
+
+if args.train:
+    model.fit(train_data, dev_data)
+result = model.predict(test_data)
+for line in result[:20]:
+    print(line)
+
+
+
 
 ```