Create README.md

README.md

@@ -0,0 +1,109 @@
+---
+license: mit
+---
+
+# MT-LLaMA Model Card
+
+## Model details
+
+**Model type:**
+MT-LLaMA is an open-source multi-task model trained by fine-tuning LLaMA on the massive tasks in [P3](https://huggingface.co/datasets/bigscience/P3) (i.e., T0 Train). Concretely, the used datasets during training and task taxonomy are listed below:
+* Multi-choice QA: CommonsenseQA, Cosmos QA, DREAM, QuAIL, QuaRTz, QASC, QuaRel, SciQ, Social IQA, Wiki Hop, WiQA  
+* Extractive QA: Adversarial QA, DuoRC, Quoref, ROPES  
+* Close-Book QA: Hotpot QA, Wiki QA  
+* Sentiment Classification: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp  
+* Topic Classification: AG News, DBPedia, TREC  
+* Structure-to-Text Generation: Common Gen, Wiki Bio  
+* Text Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum  
+* Paraphrase Identification: MRPC, PAWS, QQP  
+
+**Organizations developing the model:**
+The MT-LLaMA team with members from Alibaba Damo Academy and the Chinese University of Hong Kong.
+
+## Intended use
+
+You can try the codes from [this repo](https://github.com/DAMO-NLP-SG/MT-LLaMA).
+
+                                          
+## Zero-shot Evaluation
+We primarily follow the protocols of [Bigscience T0](https://openreview.net/forum?id=9Vrb9D0WI4) to assess the generalization capability of our Multi-task LLaMA to: (1) _**Unseen Datasets**_ (i.e., datasets from seen tasks); (2) _**Unseen Tasks**_.
+                                                     
+#### Prompt Format                                                     
+Extractive QA:
+
+1. XQuAD, TyDiQA, MLQA, SQuAD
+   ```angular2html
+    Input: Answer the question according to the context. Question: ${question}. Context: ${context}. Answer:
+    Output: ${Answer}
+   ```
+
+Sentiment:
+
+1. SST-2
+   ```angular2html
+   Input: ${sentence} Based on this review, would the user recommend this product? No or Yes?
+   Output: Yes / No
+   ```
+Multiple-Choice QA:
+
+1. OpenbookQA
+   ```angular2html
+   Input: ${question} Which is the correct answer? - (A) ${choiceA} - (B) ${choiceB} - (C) ${choiceC} - (D) ${choiceD}
+   Output: ${choiceA} / ${choiceB} / ${choiceC} / ${choiceD}
+   ```
+Sentence Completion:
+
+1. COPA
+   ```angular2html
+   Input: ${premise} {% if question == "cause" %} This happened because... {% else %} As a consequence... Help me pick the more plausible option: - ${text1} - ${text2}
+   Output: ${text1} / ${text2}
+   ```
+Coreference Resolution:
+1. Winogrande:
+   ```angular2html    
+   Input: ${sentence} In the previous sentence, does _ refer to ${option1} or ${option2}?
+   Output: ${option1} / ${option2}
+   ```
+Word Sense Disambiguation:
+1. WiC
+   ```angular2html
+   Input: Does the word "${word}" have the same meaning in these two sentences? Yes, No? ${sentence1} ${sentence2}
+   Output: ${sentence1} / ${sentence2}
+   ```
+Natural Language Inference:
+
+1. MNLI:
+   ```angular2html
+   Input: ${premise} Question: Does this imply that ${hypothesis}? Please response with 'Yes', 'No', or 'Maybe'.
+   Output: Yes / No / Maybe
+   ```
+2. RTE
+   ```angular2html  
+   Input: Given ${premise} Is it guaranteed true that "${hypothesis}"? Yes or no?
+   Output: Yes / no
+   ```
+#### Results on _Unseen Datasets_
+
+| Model       | XQuAD-en (F1/EM) | TyDiQA-en (F1/EM) | MLQA-en (F1/EM) | SQuAD (F1/EM) | SST-2 (Acc.) | OpenbookQA (Acc.) |
+|:------------|------------------|-------------------|-----------------|---------------|--------------|-------------------|
+| LLaMA-7b    | 9.5 / 2.0        | 14.3 / 2.6        | 13.4 / 3.3      | 29.4 / 11.5   | 50.5         | 32.4              |
+| MT-LLaMA-7b | 42.3 / 31.1      | 38.9 / 26.9       | 45.4 / 31.5     | 85.9 / 77.6   | 92.6         | 38.2              |
+#### Results on _Unseen Tasks_                                                     
+| Model       | COPA (Acc.) | Winogrande (Acc.)  | WiC (Acc.) | MNLI (Acc.) | RTE (Acc.) |
+|:------------|-------------|--------------------|------------|-------------|------------|
+| LLaMA-7b    | 56.0        | 49.3               | 51.7       | 30.2        | 52.7       |
+| MT-LLaMA-7b | 88.0        | 54.9               | 52.2       | 49.6        | 79.1       |
+
+## Acknowledgement
+* Our training codes are largely borrowed from [FastChat](https://github.com/lm-sys/FastChat)
+* We are also grateful for the efforts of [LLaMA](https://github.com/facebookresearch/llama) (from FAIR) and [T0](https://github.com/bigscience-workshop/t-zero) (from BigScience), which serve as the foundation of our work 
+
+If you find this resource useful, please cite the repo as follows:
+```
+@software{damonlpsg2023mtllama,
+  author = {Xu, Weiwen and Li, Xin and Bing, Lidong},
+  title = {Multi-task Instruction-tuned LLaMA},
+  year = 2023,
+  url = {https://github.com/DAMO-NLP-SG/MT-LLaMA}
+}
+```