Apel-sin commited on
Commit
e8ed5ed
1 Parent(s): 7c3421e

add measurement.json

Browse files
Files changed (2) hide show
  1. README.md +107 -0
  2. measurement.json +0 -0
README.md ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - EleutherAI/pile
4
+ - Open-Orca/OpenOrca
5
+ - GAIR/lima
6
+ - WizardLM/WizardLM_evol_instruct_V2_196k
7
+ language:
8
+ - en
9
+ license: llama3
10
+ tags:
11
+ - biology
12
+ - medical
13
+ ---
14
+ # Instruction Pre-Training: Language Models are Supervised Multitask Learners
15
+ This repo contains the **biomedicine model developed from Llama3-8B** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
16
+
17
+ We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. **In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.**
18
+
19
+ <p align='center'>
20
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/vRdsFIVQptbNaGiZ18Lih.png" width="400">
21
+ </p>
22
+
23
+
24
+ ## Resources
25
+ **🤗 We share our data and models with example usages, feel free to open any issues or discussions! 🤗**
26
+
27
+ - Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
28
+ - Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
29
+ - General Models Pre-Trained from Scratch:
30
+ - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M)
31
+ - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B)
32
+ - Domain-Specific Models Pre-Trained from Llama3-8B:
33
+ - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B)
34
+ - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B)
35
+ - General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora)
36
+ - Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora)
37
+
38
+
39
+ ## Domain-Adaptive Continued Pre-Training
40
+ Following [AdaptLLM](https://huggingface.co/AdaptLLM/medicine-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer).
41
+
42
+ ### 1. To chat with the biomedicine-Llama3-8B model:
43
+ ```python
44
+ from transformers import AutoModelForCausalLM, AutoTokenizer
45
+
46
+ model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/medicine-Llama3-8B")
47
+ tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/medicine-Llama3-8B")
48
+
49
+ # Put your input here, NO prompt template is required
50
+ user_input = '''Question: Which of the following is an example of monosomy?
51
+ Options:
52
+ - 46,XX
53
+ - 47,XXX
54
+ - 69,XYY
55
+ - 45,X
56
+
57
+ Please provide your choice first and then provide explanations if possible.'''
58
+
59
+ inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
60
+ outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]
61
+
62
+ answer_start = int(inputs.shape[-1])
63
+ pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)
64
+
65
+ print(pred)
66
+ ```
67
+
68
+ ### 2. To evaluate our models on the domain-specific tasks
69
+ 1. Setup dependencies
70
+ ```bash
71
+ git clone https://github.com/microsoft/LMOps
72
+ cd LMOps/adaptllm
73
+ pip install -r requirements.txt
74
+ ```
75
+
76
+ 2. Evaluate
77
+ ```bash
78
+ DOMAIN='biomedicine'
79
+
80
+ # if the model can fit on a single GPU: set MODEL_PARALLEL=False
81
+ # elif the model is too large to fit on a single GPU: set MODEL_PARALLEL=True
82
+ MODEL_PARALLEL=False
83
+
84
+ # number of GPUs, chosen from [1,2,4,8]
85
+ N_GPU=1
86
+
87
+ # Set as True
88
+ add_bos_token=True
89
+
90
+ bash scripts/inference.sh ${DOMAIN} 'instruction-pretrain/medicine-Llama3-8B' ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}
91
+ ```
92
+
93
+
94
+ ## Citation
95
+ If you find our work helpful, please cite us:
96
+
97
+ [AdaptLLM](https://huggingface.co/papers/2309.09530)
98
+ ```bibtex
99
+ @inproceedings{
100
+ cheng2024adapting,
101
+ title={Adapting Large Language Models via Reading Comprehension},
102
+ author={Daixuan Cheng and Shaohan Huang and Furu Wei},
103
+ booktitle={The Twelfth International Conference on Learning Representations},
104
+ year={2024},
105
+ url={https://openreview.net/forum?id=y886UXPEZ0}
106
+ }
107
+ ```
measurement.json ADDED
The diff for this file is too large to render. See raw diff