PEFT
English
hamishivi commited on
Commit
5772564
1 Parent(s): 0141809

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -7
README.md CHANGED
@@ -19,7 +19,12 @@ base_model: meta-llama/Llama-2-70b-hf
19
  Tulu is a series of language models that are trained to act as helpful assistants.
20
  Tulu 2 QLoRA 70B is a fine-tuned version of Llama 2 that was trained on a mix of publicly available, synthetic and human datasets.
21
  We release the QLoRA modules alone, which can be merged with Llama 2 70b base.
22
- We recommend **dequantising the model before merging**, as done in [this script](https://github.com/allenai/open-instruct/blob/main/open_instruct/merge_lora.py) (when using the `--qlora` flag).
 
 
 
 
 
23
 
24
 
25
  ## Model description
@@ -77,12 +82,13 @@ The following hyperparameters were used during training:
77
  If you find Tulu 2 is useful in your work, please cite it with:
78
 
79
  ```
80
- @misc{ivison2023changing,
81
- title={Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2},
82
- author={Hamish Ivison and Yizhong Wang and Valentina Pyatkin and Nathan Lambert and Matthew Peters and Pradeep Dasigi and Joel Jang and David Wadden and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
83
- year={2023},
84
- archivePrefix={arXiv},
85
- primaryClass={cs.CL}
 
86
  }
87
  ```
88
 
 
19
  Tulu is a series of language models that are trained to act as helpful assistants.
20
  Tulu 2 QLoRA 70B is a fine-tuned version of Llama 2 that was trained on a mix of publicly available, synthetic and human datasets.
21
  We release the QLoRA modules alone, which can be merged with Llama 2 70b base.
22
+ We provide [a merging script](https://github.com/allenai/open-instruct/blob/main/open_instruct/merge_lora.py) in our repository for easy merging.
23
+ We recommend **dequantising the model before merging**, done using the `--qlora` flag.
24
+
25
+
26
+ For more details, read the paper: [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
27
+ ](https://arxiv.org/abs/2311.10702).
28
 
29
 
30
  ## Model description
 
82
  If you find Tulu 2 is useful in your work, please cite it with:
83
 
84
  ```
85
+ @misc{ivison2023camels,
86
+ title={Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2},
87
+ author={Hamish Ivison and Yizhong Wang and Valentina Pyatkin and Nathan Lambert and Matthew Peters and Pradeep Dasigi and Joel Jang and David Wadden and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
88
+ year={2023},
89
+ eprint={2311.10702},
90
+ archivePrefix={arXiv},
91
+ primaryClass={cs.CL}
92
  }
93
  ```
94