Adam commited on
Commit
03c5782
1 Parent(s): 9e1dbce

feat: updating model

Browse files
Files changed (6) hide show
  1. README.md +94 -0
  2. config.json +32 -0
  3. merges.txt +0 -0
  4. pytorch_model.bin +3 -0
  5. training.log +0 -0
  6. vocab.json +0 -0
README.md CHANGED
@@ -1,3 +1,97 @@
1
  ---
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - deepspeed
6
+ - chatgpt
7
+ - opt
8
+ - sft
9
+ - rlhf
10
  license: apache-2.0
11
  ---
12
+ ---
13
+
14
+ # ChatGPT OPT 1.3B DeepSpeed Reinforcement Learning from Human feedback
15
+
16
+ *fsalab-chat-opt-1.3b-rlhf-actor-ema-deepspeed*
17
+
18
+ This model consists of the final step of a modified pipeline the to the traditional training process of Chat-GPT models, which is comprised of a three-step procedure of [supervised fine tuning](https://huggingface.co/FSALab/fsalab-chat-opt-1.3b-sft-deepspeed), [reward model](https://huggingface.co/FSALab/fsalab-chat-opt-350m-reward-deepspeed) and **reinforcement learning from human feedback**.
19
+
20
+ This project's main goal was to make proper use of existing frameworks that revolve around the minimisation of training costs and thus the eventual improvements towards both the feasibility and usability of ChatGPT-like models. The framework selected here is DeepSpeed which has been instrumental in the development of this model and through this framework it was possible to train the ChatGPT-like model on much larger data-sets with a reasonable number of GPUs and consequently achieve significantly better performance.
21
+
22
+ This model follows the blog of ChatGPT and the paper of InstructGPT and especially the [Microsoft DeepSpeed Chat Blog](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat).
23
+
24
+ ## Our Training Methodology and Speedup Recipes
25
+
26
+ The training process simply involves a single python run of DeepSpeed-Chat which initiates the whole 3-step pipeline, saving all models in the process:
27
+
28
+ ``` bash
29
+ python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_node
30
+ ```
31
+
32
+ This pipeline can be broken up into three key steps:
33
+
34
+ 1. **Supervised fine-tuning (SFT):** See [here](https://huggingface.co/FSALab/fsalab-chat-opt-1.3b-sft-deepspeed)
35
+
36
+ 2. **Reward Model (RM) fine-tuning:** See [here](https://huggingface.co/FSALab/fsalab-chat-opt-350m-reward-deepspeed)
37
+
38
+ 3. **Reinforcement-learning from Human feedback (RLHF) fine-tuning:** At the completion of the prior two steps, the final RLHF fine-tuning can be initiated. This involves the collection of both the *fine-tuned model* from step 1 and the *reward model** from step 2 and train them on the data-set with comparisons. This generates both an [actor](https://huggingface.co/FSALab/fsalab-chat-opt-1.3b-rlhf-actor-deepspeed) and [critic](https://huggingface.co/FSALab/fsalab-chat-opt-1.3b-rlhf-actor-deepspeed) and **actor EMA** model.
39
+
40
+ To view the details behind each step head into their respective links and view the model card there.
41
+
42
+ ### Reinforcement learning from human feedback
43
+
44
+ **Model Configurations:**
45
+
46
+ | Parameter | Value |
47
+ |:-----------------------|:------|
48
+ | Parameters | 1.3B |
49
+ | Model type | OPT |
50
+ | FFN Dimensions | 8192 |
51
+ | Hidden Size | 2048 |
52
+ | Max Position Embedding | 2048 |
53
+ | Attention Heads | 16 |
54
+ | Hidden layers | 24 |
55
+
56
+
57
+ **Training Configurations:**
58
+ | Parameter | Value |
59
+ |:-----------------------|:------|
60
+ | Train Batch size | 32 |
61
+ | Train micro batch size | 4 |
62
+ | ZeRO stage | 2 |
63
+ | FP16 | True |
64
+ | Gradient clipping | 1.0 |
65
+ | Dropout | 0.1 |
66
+ | Attention Dropout | 0.0 |
67
+ | Attention Dropout | 0.0 |
68
+ | Prescale gradients | False |
69
+
70
+
71
+
72
+ ## Why did we choose DeepSpeed?
73
+
74
+ **DeepSpeed Training:**
75
+
76
+ The `main.py` Python code take the DeepSpeed config with the argument `--deepspeed_config ./ds_config.json`.
77
+
78
+ We read up on the DeepSpeed documentation and created a specific coniguration based on their work. The json file `ds_config.json` here is set to take the [ZeRO-2](https://www.microsoft.com/en-us/research/blog/ZeRO-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale/) stage and FP16, allowing must faster training and GPU memory saving. Note that ZeRO-2 is just one of the examples using our DeepSpeed. You may use ZeRO-1, Zero-3, ZeRO-Offload and ZeRO-infinity. For more information on DeepSpeed ZeRO family, please see this [tutorial link](https://www.deepspeed.ai/tutorials/zero/) for Zero-1/2/3 and this [tutorial ](https://www.deepspeed.ai/tutorials/zero-offload/)for Zero-Offload.
79
+
80
+ To enable the DeepSpeed Zero family training, we injected several lines of code in order to enable this i.e.:
81
+
82
+ ```python
83
+ model, optimizer, _, lr_scheduler = deepspeed.initialize(model=model, \
84
+ optimizer=optimizer, \
85
+ args=args, \
86
+ lr_scheduler=lr_scheduler, \
87
+ dist_init_required=True)
88
+ ```
89
+
90
+
91
+ ## **Acknowledgements**
92
+
93
+ We thank the following papers and open-source repositories. We especially thank DeepSpeed for their frameworks as well.
94
+
95
+ * [1] Schulman, John, et al. "Introducing ChatGPT", https://openai.com/blog/chatgpt (2022).
96
+ * [2] Transformers [Hugging Face (github.com)](https://github.com/huggingface)
97
+ * [3] DeepSpeed Chat [DeepSpeed Chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat)
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "../step1_supervised_finetuning/output/",
3
+ "_remove_final_layer_norm": false,
4
+ "activation_dropout": 0.0,
5
+ "activation_function": "relu",
6
+ "architectures": [
7
+ "OPTForCausalLM"
8
+ ],
9
+ "attention_dropout": 0.0,
10
+ "bos_token_id": 2,
11
+ "do_layer_norm_before": true,
12
+ "dropout": 0.1,
13
+ "enable_bias": true,
14
+ "end_token_id": 50260,
15
+ "eos_token_id": 2,
16
+ "ffn_dim": 8192,
17
+ "hidden_size": 2048,
18
+ "init_std": 0.02,
19
+ "layer_norm_elementwise_affine": true,
20
+ "layerdrop": 0.0,
21
+ "max_position_embeddings": 2048,
22
+ "model_type": "opt",
23
+ "num_attention_heads": 32,
24
+ "num_hidden_layers": 24,
25
+ "pad_token_id": 2,
26
+ "prefix": "</s>",
27
+ "torch_dtype": "float16",
28
+ "transformers_version": "4.29.0.dev0",
29
+ "use_cache": true,
30
+ "vocab_size": 50272,
31
+ "word_embed_proj_dim": 2048
32
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bdd303c6edb41793376b66d12011c0ade0ee99945d9f7349ddb20849b90f6307
3
+ size 2631660857
training.log ADDED
The diff for this file is too large to render. See raw diff
 
vocab.json ADDED
The diff for this file is too large to render. See raw diff