Create README.md

README.md

@@ -0,0 +1,77 @@
+---
+language:
+- en
+---
+
+# Open-Instruct Flan V2 7B
+
+This model is a 7B LLaMa model finetuned on the Flan V2 dataset. *Please note this is a model diff - see below for usage instructions*.
+
+This was trained as part of the paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://arxiv.org/abs/2306.04751).
+The codebase used to train and evaluate this model can be found at [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct).
+
+This model is licensed under the AI model license given in LICENSE.txt along with the original Llama license (llama_license.txt).
+
+## Usage
+
+We assume you have access to a LLaMa model in HF format already. You can find details on getting access and converting the model here:
+[https://huggingface.co/docs/transformers/main/model_doc/llama](https://huggingface.co/docs/transformers/main/model_doc/llama)
+
+Clone [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct) and install the required dependencies, or just copy `scripts/weight_diff.py`
+and install the minimal requirements listed in `weight-diff-requirements.txt`. Then download or clone this model diff to the same machine.
+
+Then, run:
+```bash
+python scripts/weight_diff.py recover --path_raw ${hf_llama_path} --path_tuned ${output_path} --path_diff ${diff_location}
+```
+
+And you will have a recovered model! Note this takes up a decent amount of RAM, especially for the larger models.
+
+## Input Format
+
+The model is trained to use the following format (note the newlines):
+```
+<|user|>
+Your message here!
+<|assistant|>
+```
+
+For best results, format all inputs in this manner.
+
+## Performance
+
+Here is the performance of this model across benchmarks explored in our paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://arxiv.org/abs/2306.04751):
+
+| MMLU 0-shot | MMLU 5-shot | GSM Direct | GSM CoT | BBH Direct | BBH CoT | TydiQA Gold-Passage | TydiQA Closed-book | Codex-Eval Pass@1 | Codex-Eval Pass@10 | AlpacaFarm vs Davinci-003 | Average |
+|:-----------:|:-----------:|:----------:|:-------:|:----------:|:-------:|:-------------------:|:------------------:|:-----------------:|:------------------:|:-------------------------:|---------|
+| 45.4 | 47.1 | 3.5 | 13.0 | 38.6 | 36.1 | 45.0 | 8.3 | 9.6 | 12.9 | 4.6 | 22.4 |
+
+If you use this model, please cite our work, the llama paper, and the original dataset:
+
+```
+@article{camelevaluation,
+  title={How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources},
+  author={Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi},
+  year={2023}
+}
+```
+
+```
+@misc{touvron2023llama,
+      title={LLaMA: Open and Efficient Foundation Language Models}, 
+      author={Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie-Anne Lachaux and Timothée Lacroix and Baptiste Rozière and Naman Goyal and Eric Hambro and Faisal Azhar and Aurelien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample},
+      year={2023},
+      eprint={2302.13971},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+```
+@article{longpre2023flan,
+  title={The Flan Collection: Designing Data and Methods for Effective Instruction Tuning},
+  author={Longpre, Shayne and Hou, Le and Vu, Tu and Webson, Albert and Chung, Hyung Won and Tay, Yi and Zhou, Denny and Le, Quoc V and Zoph, Barret and Wei, Jason and others},
+  journal={arXiv preprint arXiv:2301.13688},
+  year={2023}
+}
+```