Sandiago21 commited on
Commit
b514467
1 Parent(s): 589f2de

commit base model - falcon-40b

Browse files
.gitattributes CHANGED
@@ -35,3 +35,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  finetuned_conversations.pth filter=lfs diff=lfs merge=lfs -text
37
  pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  finetuned_conversations.pth filter=lfs diff=lfs merge=lfs -text
37
  pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
38
+ tiiuae/falcon-40b/pytorch_model-00003-of-00009.bin filter=lfs diff=lfs merge=lfs -text
39
+ tiiuae/falcon-40b/pytorch_model-00005-of-00009.bin filter=lfs diff=lfs merge=lfs -text
40
+ tiiuae/falcon-40b/pytorch_model-00007-of-00009.bin filter=lfs diff=lfs merge=lfs -text
41
+ tiiuae/falcon-40b/pytorch_model-00009-of-00009.bin filter=lfs diff=lfs merge=lfs -text
42
+ tiiuae/falcon-40b/pytorch_model-00008-of-00009.bin filter=lfs diff=lfs merge=lfs -text
43
+ tiiuae/falcon-40b/pytorch_model-00001-of-00009.bin filter=lfs diff=lfs merge=lfs -text
44
+ tiiuae/falcon-40b/pytorch_model-00002-of-00009.bin filter=lfs diff=lfs merge=lfs -text
45
+ tiiuae/falcon-40b/pytorch_model-00004-of-00009.bin filter=lfs diff=lfs merge=lfs -text
46
+ tiiuae/falcon-40b/pytorch_model-00006-of-00009.bin filter=lfs diff=lfs merge=lfs -text
tiiuae/falcon-40b/README.md ADDED
@@ -0,0 +1,272 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - tiiuae/falcon-refinedweb
4
+ language:
5
+ - en
6
+ - de
7
+ - es
8
+ - fr
9
+ inference: false
10
+ license: apache-2.0
11
+ ---
12
+
13
+ # 🚀 Falcon-40B
14
+
15
+ **Falcon-40B is a 40B parameters causal decoder-only model built by [TII](https://www.tii.ae) and trained on 1,000B tokens of [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) enhanced with curated corpora. It is made available under the Apache 2.0 license.**
16
+
17
+ *Paper coming soon 😊.*
18
+
19
+
20
+
21
+ # Call for Proposals : Falcon 40B - World's Top Ranked AI Model Empowers Exceptional Use Cases with Training Compute Power in Call for Proposals
22
+
23
+ We get it. AI is everywhere! Is it taking over?
24
+
25
+ Before we debate the scant likelihood of a cyborg assassin from the future terminating humanity, let’s get to know the newbie that has soared to top-spot on the leaderboard – Falcon 40B.
26
+
27
+ Falcon 40B is the UAE’s and the Middle East’s first home-grown, open-source large language model (LLM) with 40 billion parameters trained on one trillion tokens. The brainchild of the Technology Innovation Institute (TII), Falcon 40B has generated a tremendous amount of global interest and intrigue, but what really sweetens the deal is its transparent, open-source feature.
28
+
29
+ TII is now calling for proposals from users worldwide to submit their most creative ideas for Falcon 40B’s deployment – allowing them to share their knowledge, enhance the software, and potentially transform their ideas into reality! Take that, ChatGPT!
30
+ Worth checking out? Give it a go and see for yourself!
31
+
32
+ Submit your proposal today! https://falconllm.tii.ae/call-for-proposal.php
33
+
34
+
35
+ 🤗 To get started with Falcon (inference, finetuning, quantization, etc.), we recommend reading [this great blogpost fron HF](https://huggingface.co/blog/falcon)!
36
+
37
+ ## Why use Falcon-40B?
38
+
39
+ * **It is the best open-source model currently available.** Falcon-40B outperforms [LLaMA](https://github.com/facebookresearch/llama), [StableLM](https://github.com/Stability-AI/StableLM), [RedPajama](https://huggingface.co/togethercomputer/RedPajama-INCITE-Base-7B-v0.1), [MPT](https://huggingface.co/mosaicml/mpt-7b), etc. See the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
40
+ * **It features an architecture optimized for inference**, with FlashAttention ([Dao et al., 2022](https://arxiv.org/abs/2205.14135)) and multiquery ([Shazeer et al., 2019](https://arxiv.org/abs/1911.02150)).
41
+ * **It is made available under a permissive Apache 2.0 license allowing for commercial use**, without any royalties or restrictions.
42
+ *
43
+ ⚠️ **This is a raw, pretrained model, which should be further finetuned for most usecases.** If you are looking for a version better suited to taking generic instructions in a chat format, we recommend taking a look at [Falcon-40B-Instruct](https://huggingface.co/tiiuae/falcon-40b-instruct).
44
+
45
+ 💸 **Looking for a smaller, less expensive model?** [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) is Falcon-40B's little brother!
46
+
47
+ ```python
48
+ from transformers import AutoTokenizer, AutoModelForCausalLM
49
+ import transformers
50
+ import torch
51
+
52
+ model = "tiiuae/falcon-40b"
53
+
54
+ tokenizer = AutoTokenizer.from_pretrained(model)
55
+ pipeline = transformers.pipeline(
56
+ "text-generation",
57
+ model=model,
58
+ tokenizer=tokenizer,
59
+ torch_dtype=torch.bfloat16,
60
+ trust_remote_code=True,
61
+ device_map="auto",
62
+ )
63
+ sequences = pipeline(
64
+ "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
65
+ max_length=200,
66
+ do_sample=True,
67
+ top_k=10,
68
+ num_return_sequences=1,
69
+ eos_token_id=tokenizer.eos_token_id,
70
+ )
71
+ for seq in sequences:
72
+ print(f"Result: {seq['generated_text']}")
73
+
74
+ ```
75
+
76
+ 💥 **Falcon LLMs require PyTorch 2.0 for use with `transformers`!**
77
+
78
+ For fast inference with Falcon, check-out [Text Generation Inference](https://github.com/huggingface/text-generation-inference)! Read more in this [blogpost]((https://huggingface.co/blog/falcon).
79
+
80
+ You will need **at least 85-100GB of memory** to swiftly run inference with Falcon-40B.
81
+
82
+ # Model Card for Falcon-40B
83
+
84
+ ## Model Details
85
+
86
+ ### Model Description
87
+
88
+ - **Developed by:** [https://www.tii.ae](https://www.tii.ae);
89
+ - **Model type:** Causal decoder-only;
90
+ - **Language(s) (NLP):** English, German, Spanish, French (and limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
91
+ - **License:** Apache 2.0 license.
92
+
93
+ ### Model Source
94
+
95
+ - **Paper:** *coming soon*.
96
+
97
+ ## Uses
98
+
99
+ ### Direct Use
100
+
101
+ Research on large language models; as a foundation for further specialization and finetuning for specific usecases (e.g., summarization, text generation, chatbot, etc.)
102
+
103
+ ### Out-of-Scope Use
104
+
105
+ Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.
106
+
107
+ ## Bias, Risks, and Limitations
108
+
109
+ Falcon-40B is trained mostly on English, German, Spanish, French, with limited capabilities also in in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish. It will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.
110
+
111
+ ### Recommendations
112
+
113
+ We recommend users of Falcon-40B to consider finetuning it for the specific set of tasks of interest, and for guardrails and appropriate precautions to be taken for any production use.
114
+
115
+ ## How to Get Started with the Model
116
+
117
+
118
+ ```python
119
+ from transformers import AutoTokenizer, AutoModelForCausalLM
120
+ import transformers
121
+ import torch
122
+
123
+ model = "tiiuae/falcon-40b"
124
+
125
+ tokenizer = AutoTokenizer.from_pretrained(model)
126
+ pipeline = transformers.pipeline(
127
+ "text-generation",
128
+ model=model,
129
+ tokenizer=tokenizer,
130
+ torch_dtype=torch.bfloat16,
131
+ trust_remote_code=True,
132
+ device_map="auto",
133
+ )
134
+ sequences = pipeline(
135
+ "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
136
+ max_length=200,
137
+ do_sample=True,
138
+ top_k=10,
139
+ num_return_sequences=1,
140
+ eos_token_id=tokenizer.eos_token_id,
141
+ )
142
+ for seq in sequences:
143
+ print(f"Result: {seq['generated_text']}")
144
+
145
+ ```
146
+
147
+ ## Training Details
148
+
149
+ ### Training Data
150
+
151
+ Falcon-40B was trained on 1,000B tokens of [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), a high-quality filtered and deduplicated web dataset which we enhanced with curated corpora. Significant components from our curated copora were inspired by The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)).
152
+
153
+ | **Data source** | **Fraction** | **Tokens** | **Sources** |
154
+ |--------------------|--------------|------------|-----------------------------------|
155
+ | [RefinedWeb-English](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) | 75% | 750B | massive web crawl |
156
+ | RefinedWeb-Europe | 7% | 70B | European massive web crawl |
157
+ | Books | 6% | 60B | |
158
+ | Conversations | 5% | 50B | Reddit, StackOverflow, HackerNews |
159
+ | Code | 5% | 50B | |
160
+ | Technical | 2% | 20B | arXiv, PubMed, USPTO, etc. |
161
+
162
+ RefinedWeb-Europe is made of the following languages:
163
+
164
+ | **Language** | **Fraction of multilingual data** | **Tokens** |
165
+ |--------------|-----------------------------------|------------|
166
+ | German | 26% | 18B |
167
+ | Spanish | 24% | 17B |
168
+ | French | 23% | 16B |
169
+ | _Italian_ | 7% | 5B |
170
+ | _Portuguese_ | 4% | 3B |
171
+ | _Polish_ | 4% | 3B |
172
+ | _Dutch_ | 4% | 3B |
173
+ | _Romanian_ | 3% | 2B |
174
+ | _Czech_ | 3% | 2B |
175
+ | _Swedish_ | 2% | 1B |
176
+
177
+
178
+ The data was tokenized with the Falcon-[7B](https://huggingface.co/tiiuae/falcon-7b)/[40B](https://huggingface.co/tiiuae/falcon-40b) tokenizer.
179
+
180
+ ### Training Procedure
181
+
182
+ Falcon-40B was trained on 384 A100 40GB GPUs, using a 3D parallelism strategy (TP=8, PP=4, DP=12) combined with ZeRO.
183
+
184
+ #### Training Hyperparameters
185
+
186
+ | **Hyperparameter** | **Value** | **Comment** |
187
+ |--------------------|------------|-------------------------------------------|
188
+ | Precision | `bfloat16` | |
189
+ | Optimizer | AdamW | |
190
+ | Learning rate | 1.85e-4 | 4B tokens warm-up, cosine decay to 1.85e-5 |
191
+ | Weight decay | 1e-1 | |
192
+ | Z-loss | 1e-4 | |
193
+ | Batch size | 1152 | 100B tokens ramp-up |
194
+
195
+
196
+ #### Speeds, Sizes, Times
197
+
198
+ Training started in December 2022 and took two months.
199
+
200
+
201
+ ## Evaluation
202
+
203
+ *Paper coming soon.*
204
+
205
+ See the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) for early results.
206
+
207
+
208
+ ## Technical Specifications
209
+
210
+ ### Model Architecture and Objective
211
+
212
+ Falcon-40B is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).
213
+
214
+ The architecture is broadly adapted from the GPT-3 paper ([Brown et al., 2020](https://arxiv.org/abs/2005.14165)), with the following differences:
215
+
216
+ * **Positionnal embeddings:** rotary ([Su et al., 2021](https://arxiv.org/abs/2104.09864));
217
+ * **Attention:** multiquery ([Shazeer et al., 2019](https://arxiv.org/abs/1911.02150)) and FlashAttention ([Dao et al., 2022](https://arxiv.org/abs/2205.14135));
218
+ * **Decoder-block:** parallel attention/MLP with a two layer norms.
219
+
220
+ For multiquery, we are using an internal variant which uses independent key and values per tensor parallel degree.
221
+
222
+ | **Hyperparameter** | **Value** | **Comment** |
223
+ |--------------------|-----------|----------------------------------------|
224
+ | Layers | 60 | |
225
+ | `d_model` | 8192 | |
226
+ | `head_dim` | 64 | Reduced to optimise for FlashAttention |
227
+ | Vocabulary | 65024 | |
228
+ | Sequence length | 2048 | |
229
+
230
+ ### Compute Infrastructure
231
+
232
+ #### Hardware
233
+
234
+ Falcon-40B was trained on AWS SageMaker, on 384 A100 40GB GPUs in P4d instances.
235
+
236
+ #### Software
237
+
238
+ Falcon-40B was trained a custom distributed training codebase, Gigatron. It uses a 3D parallelism approach combined with ZeRO and high-performance Triton kernels (FlashAttention, etc.)
239
+
240
+
241
+ ## Citation
242
+
243
+ *Paper coming soon* 😊. In the meanwhile, you can use the following information to cite:
244
+ ```
245
+ @article{falcon40b,
246
+ title={{Falcon-40B}: an open large language model with state-of-the-art performance},
247
+ author={Almazrouei, Ebtesam and Alobeidli, Hamza and Alshamsi, Abdulaziz and Cappelli, Alessandro and Cojocaru, Ruxandra and Debbah, Merouane and Goffinet, Etienne and Heslow, Daniel and Launay, Julien and Malartic, Quentin and Noune, Badreddine and Pannier, Baptiste and Penedo, Guilherme},
248
+ year={2023}
249
+ }
250
+ ```
251
+
252
+ To learn more about the pretraining dataset, see the 📓 [RefinedWeb paper](https://arxiv.org/abs/2306.01116).
253
+
254
+ ```
255
+ @article{refinedweb,
256
+ title={The {R}efined{W}eb dataset for {F}alcon {LLM}: outperforming curated corpora with web data, and web data only},
257
+ author={Guilherme Penedo and Quentin Malartic and Daniel Hesslow and Ruxandra Cojocaru and Alessandro Cappelli and Hamza Alobeidli and Baptiste Pannier and Ebtesam Almazrouei and Julien Launay},
258
+ journal={arXiv preprint arXiv:2306.01116},
259
+ eprint={2306.01116},
260
+ eprinttype = {arXiv},
261
+ url={https://arxiv.org/abs/2306.01116},
262
+ year={2023}
263
+ }
264
+ ```
265
+
266
+
267
+ ## License
268
+
269
+ Falcon-40B is made available under the Apache 2.0 license.
270
+
271
+ ## Contact
272
+ falconllm@tii.ae
tiiuae/falcon-40b/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alibi": false,
3
+ "apply_residual_connection_post_layernorm": false,
4
+ "architectures": [
5
+ "RWForCausalLM"
6
+ ],
7
+ "attention_dropout": 0.0,
8
+ "auto_map": {
9
+ "AutoConfig": "configuration_RW.RWConfig",
10
+ "AutoModel": "modelling_RW.RWModel",
11
+ "AutoModelForSequenceClassification": "modelling_RW.RWForSequenceClassification",
12
+ "AutoModelForTokenClassification": "modelling_RW.RWForTokenClassification",
13
+ "AutoModelForQuestionAnswering": "modelling_RW.RWForQuestionAnswering",
14
+ "AutoModelForCausalLM": "modelling_RW.RWForCausalLM"
15
+ },
16
+ "bias": false,
17
+ "bos_token_id": 11,
18
+ "eos_token_id": 11,
19
+ "hidden_dropout": 0.0,
20
+ "hidden_size": 8192,
21
+ "initializer_range": 0.02,
22
+ "layer_norm_epsilon": 1e-05,
23
+ "model_type": "RefinedWeb",
24
+ "n_head": 128,
25
+ "n_head_kv": 8,
26
+ "n_layer": 60,
27
+ "parallel_attn": true,
28
+ "torch_dtype": "bfloat16",
29
+ "transformers_version": "4.27.4",
30
+ "use_cache": true,
31
+ "vocab_size": 65024
32
+ }
tiiuae/falcon-40b/configuration_RW.py ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2022 the Big Science Workshop and HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ Bloom configuration"""
16
+ from transformers.configuration_utils import PretrainedConfig
17
+ from transformers.utils import logging
18
+
19
+
20
+ logger = logging.get_logger(__name__)
21
+
22
+
23
+ class RWConfig(PretrainedConfig):
24
+ model_type = "RefinedWeb"
25
+ keys_to_ignore_at_inference = ["past_key_values"]
26
+ attribute_map = {
27
+ "num_hidden_layers": "n_layer",
28
+ "num_attention_heads": "n_head",
29
+ }
30
+
31
+ def __init__(
32
+ self,
33
+ vocab_size=250880,
34
+ hidden_size=64,
35
+ n_layer=2,
36
+ n_head=8,
37
+ layer_norm_epsilon=1e-5,
38
+ initializer_range=0.02,
39
+ use_cache=True,
40
+ bos_token_id=1,
41
+ eos_token_id=2,
42
+ apply_residual_connection_post_layernorm=False,
43
+ hidden_dropout=0.0,
44
+ attention_dropout=0.0,
45
+ n_head_kv=None,
46
+ alibi=False,
47
+ **kwargs,
48
+ ):
49
+ self.vocab_size = vocab_size
50
+ # Backward compatibility with n_embed kwarg
51
+ n_embed = kwargs.pop("n_embed", None)
52
+ self.hidden_size = hidden_size if n_embed is None else n_embed
53
+ self.n_layer = n_layer
54
+ self.n_head = n_head
55
+ self.layer_norm_epsilon = layer_norm_epsilon
56
+ self.initializer_range = initializer_range
57
+ self.use_cache = use_cache
58
+ self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
59
+ self.hidden_dropout = hidden_dropout
60
+ self.attention_dropout = attention_dropout
61
+
62
+ self.bos_token_id = bos_token_id
63
+ self.eos_token_id = eos_token_id
64
+ self.n_head_kv = n_head if n_head_kv is None else n_head_kv
65
+ self.alibi = alibi
66
+
67
+ super().__init__(bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
68
+
69
+ @property
70
+ def head_dim(self):
71
+ return self.hidden_size // self.n_head
72
+
73
+ @property
74
+ def rotary(self):
75
+ return not self.alibi
tiiuae/falcon-40b/generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.27.4"
6
+ }
tiiuae/falcon-40b/gitattributes.txt ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tflite filter=lfs diff=lfs merge=lfs -text
29
+ *.tgz filter=lfs diff=lfs merge=lfs -text
30
+ *.wasm filter=lfs diff=lfs merge=lfs -text
31
+ *.xz filter=lfs diff=lfs merge=lfs -text
32
+ *.zip filter=lfs diff=lfs merge=lfs -text
33
+ *.zst filter=lfs diff=lfs merge=lfs -text
34
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
tiiuae/falcon-40b/modelling_RW.py ADDED
@@ -0,0 +1,1106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # port of models described in RW
2
+ # We use the bloom model as a starting point for these model.
3
+ # Please refer to the bloom models for usage instructions.
4
+
5
+ import math
6
+ import warnings
7
+ from typing import Optional, Tuple, Union
8
+
9
+ import torch
10
+ import torch.utils.checkpoint
11
+ from torch import nn
12
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, LayerNorm, MSELoss
13
+ from torch.nn import functional as F
14
+
15
+ from transformers.modeling_outputs import (
16
+ BaseModelOutputWithPastAndCrossAttentions,
17
+ CausalLMOutputWithCrossAttentions,
18
+ QuestionAnsweringModelOutput,
19
+ SequenceClassifierOutputWithPast,
20
+ TokenClassifierOutput,
21
+ )
22
+ from transformers.modeling_utils import PreTrainedModel
23
+ from transformers.utils import logging
24
+ from .configuration_RW import RWConfig
25
+
26
+ logger = logging.get_logger(__name__)
27
+
28
+ # NOTE(Hesslow): Unfortunately we did not fuse matmul and bias during training, this means that there's one additional quantization to bfloat16 between the operations.
29
+ # In order not to degrade the quality of our HF-port, we keep these characteristics in the final model.
30
+ class Linear(nn.Linear):
31
+ def forward(self, input: torch.Tensor) -> torch.Tensor:
32
+ ret = input @ self.weight.T
33
+ if self.bias is None:
34
+ return ret
35
+ else:
36
+ return ret + self.bias
37
+
38
+
39
+ from einops import rearrange
40
+
41
+ # rotary pos emb helpers (torch.jit.script does not seem to support staticmethod...)
42
+ def rotate_half(x):
43
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
44
+ return torch.cat((-x2, x1), dim=x1.ndim - 1) # dim=-1 triggers a bug in torch < 1.8.0
45
+
46
+
47
+ class RotaryEmbedding(torch.nn.Module):
48
+ """Implementation of RotaryEmbedding from GPT-NeoX.
49
+ This implementation is design to operate on queries and keys that are compatible with
50
+ [batch_size, n_heads_per_partition, seq_len, head_dim] (e.g. MinGPTAttention format).
51
+ """
52
+
53
+ def __init__(
54
+ self,
55
+ head_dim: int,
56
+ base=10000,
57
+ ):
58
+ super().__init__()
59
+ inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2).float() / head_dim))
60
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
61
+ self.head_dim = head_dim
62
+ self.seq_len_cached = None
63
+ self.batch_size_cached = None
64
+ self.cos_cached: torch.Tensor | None = None
65
+ self.sin_cached: torch.Tensor | None = None
66
+
67
+ def cos_sin(
68
+ self,
69
+ seq_len: int,
70
+ device="cuda",
71
+ dtype=torch.bfloat16,
72
+ ) -> torch.Tensor:
73
+ if seq_len != self.seq_len_cached:
74
+ self.seq_len_cached = seq_len
75
+ t = torch.arange(seq_len, device=device).type_as(self.inv_freq)
76
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
77
+ emb = torch.cat((freqs, freqs), dim=-1).to(device)
78
+
79
+ if dtype in [torch.float16, torch.bfloat16]:
80
+ emb = emb.float()
81
+
82
+ self.cos_cached = emb.cos()[None, :, :]
83
+ self.sin_cached = emb.sin()[None, :, :]
84
+
85
+ self.cos_cached = self.cos_cached.type(dtype)
86
+ self.sin_cached = self.sin_cached.type(dtype)
87
+
88
+ return self.cos_cached, self.sin_cached
89
+
90
+ def forward(self, q, k):
91
+ batch, seq_len, head_dim = q.shape
92
+ cos, sin = self.cos_sin(seq_len, q.device, q.dtype)
93
+ return (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
94
+
95
+
96
+ def _make_causal_mask(
97
+ input_ids_shape: torch.Size, device: torch.device, past_key_values_length: int
98
+ ) -> torch.BoolTensor:
99
+ batch_size, target_length = input_ids_shape
100
+ mask = torch.empty((target_length, target_length + past_key_values_length), dtype=torch.bool, device=device)
101
+ # ONNX doesn't support `torch.Tensor.triu` properly, thus we use this workaround
102
+ seq_ids = torch.arange(target_length, device=device)
103
+ mask[:, past_key_values_length:] = seq_ids[:, None] < seq_ids[None, :]
104
+
105
+ if past_key_values_length > 0:
106
+ mask[:, :past_key_values_length] = False
107
+
108
+ expanded_mask = mask[None, None, :, :].expand(batch_size, 1, target_length, target_length + past_key_values_length)
109
+ return expanded_mask
110
+
111
+
112
+ def _expand_mask(mask: torch.Tensor, tgt_length: int) -> torch.BoolTensor:
113
+ batch_size, src_length = mask.shape
114
+ tgt_length = tgt_length if tgt_length is not None else src_length
115
+
116
+ expanded_mask = ~(mask[:, None, None, :].to(torch.bool))
117
+ return expanded_mask.expand(batch_size, 1, tgt_length, src_length)
118
+
119
+
120
+ def build_alibi_tensor(attention_mask: torch.Tensor, num_heads: int, dtype: torch.dtype) -> torch.Tensor:
121
+ batch_size, seq_length = attention_mask.shape
122
+ closest_power_of_2 = 2 ** math.floor(math.log2(num_heads))
123
+ base = torch.tensor(
124
+ 2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), device=attention_mask.device, dtype=torch.float32
125
+ )
126
+ powers = torch.arange(1, 1 + closest_power_of_2, device=attention_mask.device, dtype=torch.int32)
127
+ slopes = torch.pow(base, powers)
128
+
129
+ if closest_power_of_2 != num_heads:
130
+ extra_base = torch.tensor(
131
+ 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), device=attention_mask.device, dtype=torch.float32
132
+ )
133
+ num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2)
134
+ extra_powers = torch.arange(1, 1 + 2 * num_remaining_heads, 2, device=attention_mask.device, dtype=torch.int32)
135
+ slopes = torch.cat([slopes, torch.pow(extra_base, extra_powers)], dim=0)
136
+
137
+ # Note: alibi will added to the attention bias that will be applied to the query, key product of attention
138
+ # => therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length)
139
+ # => here we set (batch_size=1, num_heads=num_heads, query_length=1, key_length=max_length)
140
+ # => the query_length dimension will then be broadcasted correctly
141
+ # This is more or less identical to T5's relative position bias:
142
+ # https://github.com/huggingface/transformers/blob/f681437203baa7671de3174b0fa583c349d9d5e1/src/transformers/models/t5/modeling_t5.py#L527
143
+ arange_tensor = ((attention_mask.cumsum(dim=-1) - 1) * attention_mask)[:, None, :]
144
+ alibi = slopes[..., None].bfloat16() * arange_tensor
145
+ return alibi.reshape(batch_size * num_heads, 1, seq_length).to(dtype)
146
+
147
+
148
+ def dropout_add(x: torch.Tensor, residual: torch.Tensor, prob: float, training: bool) -> torch.Tensor:
149
+ out = F.dropout(x, p=prob, training=training)
150
+ out = residual + out
151
+ return out
152
+
153
+
154
+ class Attention(nn.Module):
155
+ def __init__(self, config: RWConfig):
156
+ super().__init__()
157
+
158
+ self.hidden_size = config.hidden_size
159
+ self.num_heads = config.n_head
160
+ self.head_dim = self.hidden_size // self.num_heads
161
+ self.split_size = self.hidden_size
162
+ self.hidden_dropout = config.hidden_dropout
163
+
164
+ if self.head_dim * self.num_heads != self.hidden_size:
165
+ raise ValueError(
166
+ f"`hidden_size` must be divisible by num_heads (got `hidden_size`: {self.hidden_size} and `num_heads`:"
167
+ f" {self.num_heads})."
168
+ )
169
+
170
+ self.maybe_rotary = RotaryEmbedding(config.head_dim) if config.rotary else lambda q, k: (q, k)
171
+
172
+ # Layer-wise attention scaling
173
+ self.inv_norm_factor = 1.0 / math.sqrt(self.head_dim)
174
+ self.beta = self.inv_norm_factor
175
+
176
+ self.query_key_value = Linear(
177
+ self.hidden_size,
178
+ (config.n_head_kv * 2 + config.n_head) * self.head_dim,
179
+ bias=config.bias,
180
+ )
181
+ self.dense = Linear(self.hidden_size, self.hidden_size, bias=config.bias)
182
+ self.attention_dropout = nn.Dropout(config.attention_dropout)
183
+ self.num_kv = config.n_head_kv
184
+
185
+ def _split_heads(self, fused_qkv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
186
+ """
187
+ Split the last dimension into (num_heads, head_dim), results share same memory
188
+ storage as `fused_qkv`
189
+
190
+ Args:
191
+ fused_qkv (`torch.tensor`, *required*): [batch_size, seq_length, num_heads * 3 * head_dim]
192
+
193
+ Returns:
194
+ query: [batch_size, seq_length, num_heads, head_dim]
195
+ key: [batch_size, seq_length, num_heads, head_dim]
196
+ value: [batch_size, seq_length, num_heads, head_dim]
197
+ """
198
+ batch, seq_len, _ = fused_qkv.shape
199
+ qkv = fused_qkv.view(batch, seq_len, -1, self.num_heads // self.num_kv + 2, 64)
200
+ q = qkv[:, :, :, :-2]
201
+ k = qkv[:, :, :, [-2]]
202
+ v = qkv[:, :, :, [-1]]
203
+ k = torch.broadcast_to(k, q.shape)
204
+ v = torch.broadcast_to(v, q.shape)
205
+
206
+ q, k, v = [
207
+ rearrange(
208
+ x,
209
+ "batch seq_len group num_heads head_dim ->\
210
+ batch seq_len (group num_heads) head_dim",
211
+ head_dim=self.head_dim,
212
+ )
213
+ for x in [q, k, v]
214
+ ]
215
+ return q, k, v
216
+
217
+ def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:
218
+ """
219
+ Merge heads together over the last dimenstion
220
+
221
+ Args:
222
+ x: (`torch.tensor`, *required*): [batch_size * num_heads, seq_length, head_dim]
223
+
224
+ Returns:
225
+ torch.tensor: [batch_size, seq_length, num_heads * head_dim]
226
+ """
227
+ # What we want to achieve is:
228
+ # batch_size * num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads * head_dim
229
+ batch_size_and_num_heads, seq_length, _ = x.shape
230
+ batch_size = batch_size_and_num_heads // self.num_heads
231
+
232
+ # First view to decompose the batch size
233
+ # batch_size * num_heads, seq_length, head_dim -> batch_size, num_heads, seq_length, head_dim
234
+ x = x.view(batch_size, self.num_heads, seq_length, self.head_dim)
235
+
236
+ # batch_size, num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads, head_dim
237
+ x = x.permute(0, 2, 1, 3)
238
+
239
+ # batch_size, seq_length, num_heads, head_dim -> batch_size, seq_length, num_heads * head_dim
240
+ return x.reshape(batch_size, seq_length, self.num_heads * self.head_dim)
241
+
242
+ def forward(
243
+ self,
244
+ hidden_states: torch.Tensor,
245
+ alibi: torch.Tensor,
246
+ attention_mask: torch.Tensor,
247
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
248
+ head_mask: Optional[torch.Tensor] = None,
249
+ use_cache: bool = False,
250
+ output_attentions: bool = False,
251
+ ):
252
+ fused_qkv = self.query_key_value(hidden_states) # [batch_size, seq_length, 3 x hidden_size]
253
+
254
+ # 3 x [batch_size, seq_length, num_heads, head_dim]
255
+ (query_layer, key_layer, value_layer) = self._split_heads(fused_qkv)
256
+
257
+ batch_size, q_length, _, _ = query_layer.shape
258
+
259
+ query_layer = query_layer.transpose(1, 2).reshape(batch_size * self.num_heads, q_length, self.head_dim)
260
+ key_layer = key_layer.transpose(1, 2).reshape(
261
+ batch_size * self.num_heads,
262
+ q_length,
263
+ self.head_dim,
264
+ )
265
+ value_layer = value_layer.transpose(1, 2).reshape(batch_size * self.num_heads, q_length, self.head_dim)
266
+
267
+ query_layer, key_layer = self.maybe_rotary(query_layer, key_layer)
268
+
269
+ if layer_past is not None:
270
+ past_key, past_value = layer_past
271
+ # concatenate along seq_length dimension:
272
+ # - key: [batch_size * self.num_heads, head_dim, kv_length]
273
+ # - value: [batch_size * self.num_heads, kv_length, head_dim]
274
+ key_layer = torch.cat((past_key, key_layer), dim=1)
275
+ value_layer = torch.cat((past_value, value_layer), dim=1)
276
+
277
+ _, kv_length, _ = key_layer.shape
278
+
279
+ if use_cache is True:
280
+ present = (key_layer, value_layer)
281
+ else:
282
+ present = None
283
+
284
+ if alibi is None:
285
+ query_layer_ = query_layer.reshape(batch_size, self.num_heads, -1, self.head_dim)
286
+ key_layer_ = key_layer.reshape(batch_size, self.num_heads, -1, self.head_dim)
287
+ value_layer_ = value_layer.reshape(batch_size, self.num_heads, -1, self.head_dim)
288
+
289
+ attn_output = F.scaled_dot_product_attention(
290
+ query_layer_, key_layer_, value_layer_, None, 0.0, is_causal=True
291
+ )
292
+
293
+ x = attn_output.view(batch_size, self.num_heads, q_length, self.head_dim)
294
+ x = x.permute(0, 2, 1, 3)
295
+ attn_output = x.reshape(batch_size, q_length, self.num_heads * self.head_dim)
296
+
297
+ output_tensor = self.dense(attn_output)
298
+
299
+ outputs = (output_tensor, present)
300
+ assert not output_attentions # not supported.
301
+ return outputs
302
+ else:
303
+ attention_mask_float = (attention_mask * 1.0).masked_fill(attention_mask, -1e9).to(torch.bfloat16)
304
+ matmul_result = query_layer @ key_layer.transpose(-1, -2)
305
+
306
+ # change view to [batch_size, num_heads, q_length, kv_length]
307
+ attention_scores = matmul_result.view(batch_size, self.num_heads, q_length, kv_length)
308
+
309
+ # cast attention scores to fp32, compute scaled softmax and cast back to initial dtype - [batch_size, num_heads, q_length, kv_length]
310
+ input_dtype = attention_scores.dtype
311
+ # `float16` has a minimum value of -65504.0, whereas `bfloat16` and `float32` have a minimum value of `-3.4e+38`
312
+ if input_dtype == torch.float16 or input_dtype == torch.bfloat16:
313
+ attention_scores = attention_scores.to(torch.float32)
314
+ # attn_weights = torch.masked_fill(attention_scores, attention_mask, torch.finfo(attention_scores.dtype).min)
315
+ attention_probs = F.softmax(
316
+ (attention_scores + alibi.view(batch_size, self.num_heads, 1, -1)) * self.inv_norm_factor
317
+ + attention_mask_float,
318
+ dim=-1,
319
+ dtype=hidden_states.dtype,
320
+ )
321
+ # [batch_size, num_heads, q_length, kv_length]
322
+ attention_probs = self.attention_dropout(attention_probs)
323
+
324
+ if head_mask is not None:
325
+ attention_probs = attention_probs * head_mask
326
+
327
+ # change view [batch_size x num_heads, q_length, kv_length]
328
+ attention_probs_reshaped = attention_probs.view(batch_size * self.num_heads, q_length, kv_length)
329
+
330
+ # matmul: [batch_size * num_heads, q_length, head_dim]
331
+ context_layer = attention_probs_reshaped @ value_layer
332
+
333
+ # change view [batch_size, num_heads, q_length, head_dim]
334
+ context_layer = self._merge_heads(context_layer)
335
+
336
+ output_tensor = self.dense(context_layer)
337
+
338
+ outputs = (output_tensor, present)
339
+ if output_attentions:
340
+ outputs += (attention_probs,)
341
+
342
+ return outputs
343
+
344
+
345
+ class MLP(nn.Module):
346
+ def __init__(self, config: RWConfig):
347
+ super().__init__()
348
+ hidden_size = config.hidden_size
349
+
350
+ self.dense_h_to_4h = Linear(hidden_size, 4 * hidden_size, bias=config.bias)
351
+ self.act = nn.GELU()
352
+ self.dense_4h_to_h = Linear(4 * hidden_size, hidden_size, bias=config.bias)
353
+ self.hidden_dropout = config.hidden_dropout
354
+
355
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
356
+ x = self.act(self.dense_h_to_4h(x))
357
+ x = self.dense_4h_to_h(x)
358
+ return x
359
+
360
+
361
+ class DecoderLayer(nn.Module):
362
+ def __init__(self, config: RWConfig):
363
+ super().__init__()
364
+ hidden_size = config.hidden_size
365
+
366
+ self.ln_attn = LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
367
+ self.ln_mlp = LayerNorm(hidden_size, eps=config.layer_norm_epsilon)
368
+
369
+ self.num_heads = config.n_head
370
+ self.self_attention = Attention(config)
371
+
372
+ self.mlp = MLP(config)
373
+
374
+ self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm
375
+ self.hidden_dropout = config.hidden_dropout
376
+
377
+ self.config = config
378
+
379
+ def forward(
380
+ self,
381
+ hidden_states: torch.Tensor,
382
+ alibi: torch.Tensor,
383
+ attention_mask: torch.Tensor,
384
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
385
+ head_mask: Optional[torch.Tensor] = None,
386
+ use_cache: bool = False,
387
+ output_attentions: bool = False,
388
+ ):
389
+
390
+ ln_attn = self.ln_attn(hidden_states)
391
+ ln_mlp = self.ln_mlp(hidden_states)
392
+
393
+ residual = hidden_states
394
+
395
+ # Self attention.
396
+ attn_outputs = self.self_attention(
397
+ ln_attn,
398
+ layer_past=layer_past,
399
+ attention_mask=attention_mask,
400
+ alibi=alibi,
401
+ head_mask=head_mask,
402
+ use_cache=use_cache,
403
+ output_attentions=output_attentions,
404
+ )
405
+
406
+ attention_output = attn_outputs[0]
407
+
408
+ outputs = attn_outputs[1:]
409
+
410
+ # MLP.
411
+ mlp_output = self.mlp(ln_mlp)
412
+
413
+ output = dropout_add(
414
+ mlp_output + attention_output, residual, self.config.hidden_dropout, training=self.training
415
+ )
416
+
417
+ if use_cache:
418
+ outputs = (output,) + outputs
419
+ else:
420
+ outputs = (output,) + outputs[1:]
421
+
422
+ return outputs # hidden_states, present, attentions
423
+
424
+
425
+ class RWPreTrainedModel(PreTrainedModel):
426
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
427
+ """
428
+ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
429
+ models.
430
+ """
431
+
432
+ config_class = RWConfig
433
+ base_model_prefix = "transformer"
434
+ supports_gradient_checkpointing = True
435
+ _no_split_modules = ["DecoderLayer"]
436
+
437
+ def __init__(self, *inputs, **kwargs):
438
+ super().__init__(*inputs, **kwargs)
439
+
440
+ def _init_weights(self, module: nn.Module):
441
+ """Initialize the weights."""
442
+ if isinstance(module, nn.Linear) or isinstance(module, Linear):
443
+ # Slightly different from the TF version which uses truncated_normal for initialization
444
+ # cf https://github.com/pytorch/pytorch/pull/5617
445
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
446
+ if module.bias is not None:
447
+ module.bias.data.zero_()
448
+ elif isinstance(module, nn.Embedding):
449
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
450
+ if module.padding_idx is not None:
451
+ module.weight.data[module.padding_idx].zero_()
452
+ elif isinstance(module, LayerNorm):
453
+ module.bias.data.zero_()
454
+ module.weight.data.fill_(1.0)
455
+
456
+ def _set_gradient_checkpointing(self, module: nn.Module, value: bool = False):
457
+ if isinstance(module, RWModel):
458
+ module.gradient_checkpointing = value
459
+
460
+ @staticmethod
461
+ def _convert_to_standard_cache(
462
+ past_key_value: Tuple[Tuple[torch.Tensor, torch.Tensor]], batch_size: int
463
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor]]:
464
+ """
465
+ Standardizes the format of the cache so as to match most implementations, i.e. to tuple(tuple([batch_size,
466
+ num_heads, ...]))
467
+ """
468
+ batch_size_times_num_heads, head_dim, seq_length = past_key_value[0][0].shape
469
+ num_heads = batch_size_times_num_heads // batch_size
470
+ # key: [batch_size * num_heads, head_dim, seq_length] -> [batch_size, num_heads, head_dim, seq_length]
471
+ # value: [batch_size * num_heads, seq_length, head_dim] -> [batch_size, num_heads, seq_length, head_dim]
472
+ return tuple(
473
+ (
474
+ layer_past[0].view(batch_size, num_heads, head_dim, seq_length),
475
+ layer_past[1].view(batch_size, num_heads, seq_length, head_dim),
476
+ )
477
+ for layer_past in past_key_value
478
+ )
479
+
480
+ @staticmethod
481
+ def _convert_to_rw_cache(
482
+ past_key_value: Tuple[Tuple[torch.Tensor, torch.Tensor]]
483
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor]]:
484
+ batch_size, num_heads, head_dim, seq_length = past_key_value[0][0].shape
485
+ batch_size_times_num_heads = batch_size * num_heads
486
+ # key: [batch_size, num_heads, head_dim, seq_length] -> [batch_size * num_heads, head_dim, seq_length]
487
+ # value: [batch_size, num_heads, seq_length, head_dim] -> [batch_size * num_heads, seq_length, head_dim]
488
+ return tuple(
489
+ (
490
+ layer_past[0].view(batch_size_times_num_heads, head_dim, seq_length),
491
+ layer_past[1].view(batch_size_times_num_heads, seq_length, head_dim),
492
+ )
493
+ for layer_past in past_key_value
494
+ )
495
+
496
+
497
+ class RWModel(RWPreTrainedModel):
498
+ def __init__(self, config: RWConfig):
499
+ super().__init__(config)
500
+
501
+ self.embed_dim = config.hidden_size
502
+ self.num_heads = config.n_head
503
+ self.alibi = config.alibi
504
+
505
+ # Embedding + LN Embedding
506
+ self.word_embeddings = nn.Embedding(config.vocab_size, self.embed_dim)
507
+
508
+ # Transformer blocks
509
+ self.h = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)])
510
+
511
+ # Final Layer Norm
512
+ self.ln_f = LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
513
+
514
+ self.gradient_checkpointing = False
515
+
516
+ # Initialize weights and apply final processing
517
+ self.post_init()
518
+
519
+ def get_input_embeddings(self):
520
+ return self.word_embeddings
521
+
522
+ def _prepare_attn_mask(
523
+ self, attention_mask: torch.Tensor, input_shape: Tuple[int, int], past_key_values_length: int
524
+ ) -> torch.BoolTensor:
525
+ # create causal mask
526
+ # [batch_size, seq_length] -> [batch_size, 1, tgt_length, src_length]
527
+ combined_attention_mask = None
528
+ device = attention_mask.device
529
+ _, src_length = input_shape
530
+
531
+ if src_length > 1:
532
+ combined_attention_mask = _make_causal_mask(
533
+ input_shape, device=device, past_key_values_length=past_key_values_length
534
+ )
535
+
536
+ # [batch_size, seq_length] -> [batch_size, 1, tgt_length, src_length]
537
+ expanded_attn_mask = _expand_mask(attention_mask, tgt_length=src_length)
538
+ combined_attention_mask = (
539
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask | combined_attention_mask
540
+ )
541
+
542
+ return combined_attention_mask
543
+
544
+ def set_input_embeddings(self, new_embeddings: torch.Tensor):
545
+ self.word_embeddings = new_embeddings
546
+
547
+ def forward(
548
+ self,
549
+ input_ids: Optional[torch.LongTensor] = None,
550
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
551
+ attention_mask: Optional[torch.Tensor] = None,
552
+ head_mask: Optional[torch.LongTensor] = None,
553
+ inputs_embeds: Optional[torch.LongTensor] = None,
554
+ use_cache: Optional[bool] = None,
555
+ output_attentions: Optional[bool] = None,
556
+ output_hidden_states: Optional[bool] = None,
557
+ return_dict: Optional[bool] = None,
558
+ **deprecated_arguments,
559
+ ) -> Union[Tuple[torch.Tensor, ...], BaseModelOutputWithPastAndCrossAttentions]:
560
+ if deprecated_arguments.pop("position_ids", False) is not False:
561
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
562
+ warnings.warn(
563
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
564
+ " passing `position_ids`.",
565
+ FutureWarning,
566
+ )
567
+ if len(deprecated_arguments) > 0:
568
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
569
+
570
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
571
+ output_hidden_states = (
572
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
573
+ )
574
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
575
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
576
+
577
+ if input_ids is not None and inputs_embeds is not None:
578
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
579
+ elif input_ids is not None:
580
+ batch_size, seq_length = input_ids.shape
581
+ elif inputs_embeds is not None:
582
+ batch_size, seq_length, _ = inputs_embeds.shape
583
+ else:
584
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
585
+
586
+ if past_key_values is None:
587
+ past_key_values = tuple([None] * len(self.h))
588
+
589
+ # Prepare head mask if needed
590
+ # 1.0 in head_mask indicate we keep the head
591
+ # attention_probs has shape batch_size x num_heads x N x N
592
+ # head_mask has shape n_layer x batch x num_heads x N x N
593
+ head_mask = self.get_head_mask(head_mask, self.config.n_layer)
594
+
595
+ if inputs_embeds is None:
596
+ inputs_embeds = self.word_embeddings(input_ids)
597
+
598
+ hidden_states = inputs_embeds
599
+
600
+ presents = () if use_cache else None
601
+ all_self_attentions = () if output_attentions else None
602
+ all_hidden_states = () if output_hidden_states else None
603
+
604
+ # Compute alibi tensor: check build_alibi_tensor documentation
605
+ seq_length_with_past = seq_length
606
+ past_key_values_length = 0
607
+ if past_key_values[0] is not None:
608
+ past_key_values_length = past_key_values[0][0].shape[2]
609
+ seq_length_with_past = seq_length_with_past + past_key_values_length
610
+ if attention_mask is None:
611
+ attention_mask = torch.ones((batch_size, seq_length_with_past), device=hidden_states.device)
612
+ else:
613
+ attention_mask = attention_mask.to(hidden_states.device)
614
+
615
+ if self.alibi:
616
+ alibi = build_alibi_tensor(attention_mask, self.num_heads, dtype=hidden_states.dtype)
617
+ else:
618
+ alibi = None
619
+
620
+ causal_mask = self._prepare_attn_mask(
621
+ attention_mask,
622
+ input_shape=(batch_size, seq_length),
623
+ past_key_values_length=past_key_values_length,
624
+ )
625
+
626
+ for i, (block, layer_past) in enumerate(zip(self.h, past_key_values)):
627
+
628
+ if output_hidden_states:
629
+ all_hidden_states = all_hidden_states + (hidden_states,)
630
+
631
+ if self.gradient_checkpointing and self.training:
632
+
633
+ if use_cache:
634
+ logger.warning(
635
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
636
+ )
637
+ use_cache = False
638
+
639
+ def create_custom_forward(module):
640
+ def custom_forward(*inputs):
641
+ # None for past_key_value
642
+ return module(*inputs, use_cache=use_cache, output_attentions=output_attentions)
643
+
644
+ return custom_forward
645
+
646
+ outputs = torch.utils.checkpoint.checkpoint(
647
+ create_custom_forward(block),
648
+ hidden_states,
649
+ alibi,
650
+ causal_mask,
651
+ head_mask[i],
652
+ )
653
+ else:
654
+ outputs = block(
655
+ hidden_states,
656
+ layer_past=layer_past,
657
+ attention_mask=causal_mask,
658
+ head_mask=head_mask[i],
659
+ use_cache=use_cache,
660
+ output_attentions=output_attentions,
661
+ alibi=alibi,
662
+ )
663
+
664
+ hidden_states = outputs[0]
665
+ if use_cache is True:
666
+ presents = presents + (outputs[1],)
667
+
668
+ if output_attentions:
669
+ all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)
670
+
671
+ # Add last hidden state
672
+ hidden_states = self.ln_f(hidden_states)
673
+
674
+ if output_hidden_states:
675
+ all_hidden_states = all_hidden_states + (hidden_states,)
676
+
677
+ if not return_dict:
678
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
679
+
680
+ return BaseModelOutputWithPastAndCrossAttentions(
681
+ last_hidden_state=hidden_states,
682
+ past_key_values=presents,
683
+ hidden_states=all_hidden_states,
684
+ attentions=all_self_attentions,
685
+ )
686
+
687
+
688
+ class RWForCausalLM(RWPreTrainedModel):
689
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
690
+
691
+ def __init__(self, config: RWConfig):
692
+ super().__init__(config)
693
+ self.transformer = RWModel(config)
694
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
695
+
696
+ # Initialize weights and apply final processing
697
+ self.post_init()
698
+
699
+ def get_output_embeddings(self):
700
+ return self.lm_head
701
+
702
+ def set_output_embeddings(self, new_embeddings: torch.Tensor):
703
+ self.lm_head = new_embeddings
704
+
705
+ def prepare_inputs_for_generation(
706
+ self,
707
+ input_ids: torch.LongTensor,
708
+ past: Optional[torch.Tensor] = None,
709
+ attention_mask: Optional[torch.Tensor] = None,
710
+ **kwargs,
711
+ ) -> dict:
712
+ # only last token for input_ids if past is not None
713
+ if past:
714
+ input_ids = input_ids[:, -1].unsqueeze(-1)
715
+
716
+ # the cache may be in the stardard format (e.g. in contrastive search), convert to our's format if needed
717
+ if past[0][0].shape[0] == input_ids.shape[0]:
718
+ past = self._convert_to_rw_cache(past)
719
+
720
+ return {
721
+ "input_ids": input_ids,
722
+ "past_key_values": past,
723
+ "use_cache": kwargs.get("use_cache"),
724
+ "attention_mask": attention_mask,
725
+ }
726
+
727
+ def forward(
728
+ self,
729
+ input_ids: Optional[torch.LongTensor] = None,
730
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
731
+ attention_mask: Optional[torch.Tensor] = None,
732
+ head_mask: Optional[torch.Tensor] = None,
733
+ inputs_embeds: Optional[torch.Tensor] = None,
734
+ labels: Optional[torch.Tensor] = None,
735
+ use_cache: Optional[bool] = None,
736
+ output_attentions: Optional[bool] = None,
737
+ output_hidden_states: Optional[bool] = None,
738
+ return_dict: Optional[bool] = None,
739
+ **deprecated_arguments,
740
+ ) -> Union[Tuple[torch.Tensor], CausalLMOutputWithCrossAttentions]:
741
+ r"""
742
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
743
+ Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
744
+ `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
745
+ are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
746
+ """
747
+ if deprecated_arguments.pop("position_ids", False) is not False:
748
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
749
+ warnings.warn(
750
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
751
+ " passing `position_ids`.",
752
+ FutureWarning,
753
+ )
754
+ if len(deprecated_arguments) > 0:
755
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
756
+
757
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
758
+
759
+ transformer_outputs = self.transformer(
760
+ input_ids,
761
+ past_key_values=past_key_values,
762
+ attention_mask=attention_mask,
763
+ head_mask=head_mask,
764
+ inputs_embeds=inputs_embeds,
765
+ use_cache=use_cache,
766
+ output_attentions=output_attentions,
767
+ output_hidden_states=output_hidden_states,
768
+ return_dict=return_dict,
769
+ )
770
+ hidden_states = transformer_outputs[0]
771
+
772
+ lm_logits = self.lm_head(hidden_states)
773
+
774
+ loss = None
775
+ if labels is not None:
776
+ # Shift so that tokens < n predict n
777
+ shift_logits = lm_logits[..., :-1, :].contiguous()
778
+ shift_labels = labels[..., 1:].contiguous()
779
+ batch_size, seq_length, vocab_size = shift_logits.shape
780
+ # Flatten the tokens
781
+ loss_fct = CrossEntropyLoss()
782
+ loss = loss_fct(
783
+ shift_logits.view(batch_size * seq_length, vocab_size), shift_labels.view(batch_size * seq_length)
784
+ )
785
+
786
+ if not return_dict:
787
+ output = (lm_logits,) + transformer_outputs[1:]
788
+ return ((loss,) + output) if loss is not None else output
789
+
790
+ return CausalLMOutputWithCrossAttentions(
791
+ loss=loss,
792
+ logits=lm_logits,
793
+ past_key_values=transformer_outputs.past_key_values,
794
+ hidden_states=transformer_outputs.hidden_states,
795
+ attentions=transformer_outputs.attentions,
796
+ )
797
+
798
+ def _reorder_cache(
799
+ self, past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
800
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
801
+ """
802
+ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
803
+ [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
804
+ beam_idx at every generation step.
805
+
806
+ Output shares the same memory storage as `past`.
807
+ """
808
+ standardized_past = self._convert_to_standard_cache(past, batch_size=len(beam_idx))
809
+
810
+ # Get a copy of `beam_idx` on all the devices where we need those indices.
811
+ device_to_beam_idx = {
812
+ past_state.device: beam_idx.to(past_state.device) for layer_past in past for past_state in layer_past
813
+ }
814
+ reordered_past = tuple(
815
+ (
816
+ layer_past[0].index_select(0, device_to_beam_idx[layer_past[0].device]),
817
+ layer_past[1].index_select(0, device_to_beam_idx[layer_past[0].device]),
818
+ )
819
+ for layer_past in standardized_past
820
+ )
821
+ return self._convert_to_rw_cache(reordered_past)
822
+
823
+
824
+ class RWForSequenceClassification(RWPreTrainedModel):
825
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
826
+
827
+ def __init__(self, config: RWConfig):
828
+ super().__init__(config)
829
+ self.num_labels = config.num_labels
830
+ self.transformer = RWModel(config)
831
+ self.score = nn.Linear(config.hidden_size, config.num_labels, bias=False)
832
+
833
+ # Initialize weights and apply final processing
834
+ self.post_init()
835
+
836
+ def forward(
837
+ self,
838
+ input_ids: Optional[torch.LongTensor] = None,
839
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
840
+ attention_mask: Optional[torch.Tensor] = None,
841
+ head_mask: Optional[torch.Tensor] = None,
842
+ inputs_embeds: Optional[torch.Tensor] = None,
843
+ labels: Optional[torch.Tensor] = None,
844
+ use_cache: Optional[bool] = None,
845
+ output_attentions: Optional[bool] = None,
846
+ output_hidden_states: Optional[bool] = None,
847
+ return_dict: Optional[bool] = None,
848
+ **deprecated_arguments,
849
+ ) -> Union[Tuple[torch.Tensor], SequenceClassifierOutputWithPast]:
850
+ r"""
851
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
852
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
853
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
854
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
855
+ """
856
+ if deprecated_arguments.pop("position_ids", False) is not False:
857
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
858
+ warnings.warn(
859
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
860
+ " passing `position_ids`.",
861
+ FutureWarning,
862
+ )
863
+ if len(deprecated_arguments) > 0:
864
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
865
+
866
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
867
+
868
+ transformer_outputs = self.transformer(
869
+ input_ids,
870
+ past_key_values=past_key_values,
871
+ attention_mask=attention_mask,
872
+ head_mask=head_mask,
873
+ inputs_embeds=inputs_embeds,
874
+ use_cache=use_cache,
875
+ output_attentions=output_attentions,
876
+ output_hidden_states=output_hidden_states,
877
+ return_dict=return_dict,
878
+ )
879
+
880
+ hidden_states = transformer_outputs[0]
881
+ logits = self.score(hidden_states)
882
+
883
+ if input_ids is not None:
884
+ batch_size = input_ids.shape[0]
885
+ else:
886
+ batch_size = inputs_embeds.shape[0]
887
+
888
+ if self.config.pad_token_id is None and batch_size != 1:
889
+ raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
890
+ if self.config.pad_token_id is None:
891
+ sequence_lengths = -1
892
+ else:
893
+ if input_ids is not None:
894
+ sequence_lengths = torch.ne(input_ids, self.config.pad_token_id).sum(dim=-1) - 1
895
+ else:
896
+ sequence_lengths = -1
897
+ logger.warning(
898
+ f"{self.__class__.__name__} will not detect padding tokens in `inputs_embeds`. Results may be "
899
+ "unexpected if using padding tokens in conjunction with `inputs_embeds.`"
900
+ )
901
+
902
+ pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
903
+
904
+ loss = None
905
+ if labels is not None:
906
+ if self.config.problem_type is None:
907
+ if self.num_labels == 1:
908
+ self.config.problem_type = "regression"
909
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
910
+ self.config.problem_type = "single_label_classification"
911
+ else:
912
+ self.config.problem_type = "multi_label_classification"
913
+
914
+ if self.config.problem_type == "regression":
915
+ loss_fct = MSELoss()
916
+ if self.num_labels == 1:
917
+ loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
918
+ else:
919
+ loss = loss_fct(pooled_logits, labels)
920
+ elif self.config.problem_type == "single_label_classification":
921
+ loss_fct = CrossEntropyLoss()
922
+ loss = loss_fct(pooled_logits, labels)
923
+ elif self.config.problem_type == "multi_label_classification":
924
+ loss_fct = BCEWithLogitsLoss()
925
+ loss = loss_fct(pooled_logits, labels)
926
+ if not return_dict:
927
+ output = (pooled_logits,) + transformer_outputs[1:]
928
+ return ((loss,) + output) if loss is not None else output
929
+
930
+ return SequenceClassifierOutputWithPast(
931
+ loss=loss,
932
+ logits=pooled_logits,
933
+ past_key_values=transformer_outputs.past_key_values,
934
+ hidden_states=transformer_outputs.hidden_states,
935
+ attentions=transformer_outputs.attentions,
936
+ )
937
+
938
+
939
+ class RWForTokenClassification(RWPreTrainedModel):
940
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
941
+
942
+ def __init__(self, config: RWConfig):
943
+ super().__init__(config)
944
+ self.num_labels = config.num_labels
945
+
946
+ self.transformer = RWModel(config)
947
+ if hasattr(config, "classifier_dropout") and config.classifier_dropout is not None:
948
+ classifier_dropout = config.classifier_dropout
949
+ elif hasattr(config, "hidden_dropout") and config.hidden_dropout is not None:
950
+ classifier_dropout = config.hidden_dropout
951
+ else:
952
+ classifier_dropout = 0.1
953
+ self.dropout = nn.Dropout(classifier_dropout)
954
+ self.classifier = nn.Linear(config.hidden_size, config.num_labels)
955
+
956
+ # Initialize weights and apply final processing
957
+ self.post_init()
958
+
959
+ def forward(
960
+ self,
961
+ input_ids: Optional[torch.LongTensor] = None,
962
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
963
+ attention_mask: Optional[torch.Tensor] = None,
964
+ head_mask: Optional[torch.Tensor] = None,
965
+ inputs_embeds: Optional[torch.Tensor] = None,
966
+ labels: Optional[torch.Tensor] = None,
967
+ use_cache: Optional[bool] = None,
968
+ output_attentions: Optional[bool] = None,
969
+ output_hidden_states: Optional[bool] = None,
970
+ return_dict: Optional[bool] = None,
971
+ **deprecated_arguments,
972
+ ) -> Union[Tuple[torch.Tensor], TokenClassifierOutput]:
973
+ r"""
974
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
975
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
976
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
977
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
978
+ """
979
+ if deprecated_arguments.pop("position_ids", False) is not False:
980
+ # `position_ids` could have been `torch.Tensor` or `None` so defaulting pop to `False` allows to detect if users were passing explicitly `None`
981
+ warnings.warn(
982
+ "`position_ids` have no functionality in BLOOM and will be removed in v5.0.0. You can safely ignore"
983
+ " passing `position_ids`.",
984
+ FutureWarning,
985
+ )
986
+ if len(deprecated_arguments) > 0:
987
+ raise ValueError(f"Got unexpected arguments: {deprecated_arguments}")
988
+
989
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
990
+
991
+ transformer_outputs = self.transformer(
992
+ input_ids,
993
+ past_key_values=past_key_values,
994
+ attention_mask=attention_mask,
995
+ head_mask=head_mask,
996
+ inputs_embeds=inputs_embeds,
997
+ use_cache=use_cache,
998
+ output_attentions=output_attentions,
999
+ output_hidden_states=output_hidden_states,
1000
+ return_dict=return_dict,
1001
+ )
1002
+
1003
+ hidden_states = transformer_outputs[0]
1004
+ hidden_states = self.dropout(hidden_states)
1005
+ logits = self.classifier(hidden_states)
1006
+
1007
+ loss = None
1008
+ if labels is not None:
1009
+ batch_size, seq_length = labels.shape
1010
+ loss_fct = CrossEntropyLoss()
1011
+ loss = loss_fct(logits.view(batch_size * seq_length, self.num_labels), labels.view(batch_size * seq_length))
1012
+
1013
+ if not return_dict:
1014
+ output = (logits,) + transformer_outputs[2:]
1015
+ return ((loss,) + output) if loss is not None else output
1016
+
1017
+ return TokenClassifierOutput(
1018
+ loss=loss,
1019
+ logits=logits,
1020
+ hidden_states=transformer_outputs.hidden_states,
1021
+ attentions=transformer_outputs.attentions,
1022
+ )
1023
+
1024
+
1025
+ class RWForQuestionAnswering(RWPreTrainedModel):
1026
+ _keys_to_ignore_on_load_missing = [r"h.*.self_attention.scale_mask_softmax.causal_mask", r"lm_head.weight"]
1027
+
1028
+ def __init__(self, config):
1029
+ super().__init__(config)
1030
+ self.transformer = RWModel(config)
1031
+ self.qa_outputs = nn.Linear(config.hidden_size, 2)
1032
+
1033
+ # Initialize weights and apply final processing
1034
+ self.post_init()
1035
+
1036
+ def forward(
1037
+ self,
1038
+ input_ids: Optional[torch.LongTensor] = None,
1039
+ attention_mask: Optional[torch.FloatTensor] = None,
1040
+ position_ids: Optional[torch.LongTensor] = None,
1041
+ head_mask: Optional[torch.FloatTensor] = None,
1042
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1043
+ start_positions: Optional[torch.LongTensor] = None,
1044
+ end_positions: Optional[torch.LongTensor] = None,
1045
+ output_attentions: Optional[bool] = None,
1046
+ output_hidden_states: Optional[bool] = None,
1047
+ return_dict: Optional[bool] = None,
1048
+ ) -> Union[Tuple, QuestionAnsweringModelOutput]:
1049
+ r"""
1050
+ start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1051
+ Labels for position (index) of the start of the labelled span for computing the token classification loss.
1052
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
1053
+ are not taken into account for computing the loss.
1054
+ end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1055
+ Labels for position (index) of the end of the labelled span for computing the token classification loss.
1056
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
1057
+ are not taken into account for computing the loss.
1058
+ """
1059
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1060
+
1061
+ outputs = self.transformer(
1062
+ input_ids,
1063
+ attention_mask=attention_mask,
1064
+ position_ids=position_ids,
1065
+ head_mask=head_mask,
1066
+ inputs_embeds=inputs_embeds,
1067
+ output_attentions=output_attentions,
1068
+ output_hidden_states=output_hidden_states,
1069
+ return_dict=return_dict,
1070
+ )
1071
+
1072
+ sequence_output = outputs[0]
1073
+
1074
+ logits = self.qa_outputs(sequence_output)
1075
+ start_logits, end_logits = logits.split(1, dim=-1)
1076
+ start_logits = start_logits.squeeze(-1).contiguous()
1077
+ end_logits = end_logits.squeeze(-1).contiguous()
1078
+
1079
+ total_loss = None
1080
+ if start_positions is not None and end_positions is not None:
1081
+ # If we are on multi-GPU, split add a dimension
1082
+ if len(start_positions.size()) > 1:
1083
+ start_positions = start_positions.squeeze(-1)
1084
+ if len(end_positions.size()) > 1:
1085
+ end_positions = end_positions.squeeze(-1)
1086
+ # sometimes the start/end positions are outside our model inputs, we ignore these terms
1087
+ ignored_index = start_logits.size(1)
1088
+ start_positions = start_positions.clamp(0, ignored_index)
1089
+ end_positions = end_positions.clamp(0, ignored_index)
1090
+
1091
+ loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
1092
+ start_loss = loss_fct(start_logits, start_positions)
1093
+ end_loss = loss_fct(end_logits, end_positions)
1094
+ total_loss = (start_loss + end_loss) / 2
1095
+
1096
+ if not return_dict:
1097
+ output = (start_logits, end_logits) + outputs[2:]
1098
+ return ((total_loss,) + output) if total_loss is not None else output
1099
+
1100
+ return QuestionAnsweringModelOutput(
1101
+ loss=total_loss,
1102
+ start_logits=start_logits,
1103
+ end_logits=end_logits,
1104
+ hidden_states=outputs.hidden_states,
1105
+ attentions=outputs.attentions,
1106
+ )
tiiuae/falcon-40b/pytorch_model-00001-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:384bd92e6d1d0b6a133b1d81bad915a10d954d1bfcdf4fdc8b6d0d81b4f5e9e8
3
+ size 9504770141
tiiuae/falcon-40b/pytorch_model-00002-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:638cb048ee6ad66f85a176e980332b0d61cbddb2d5dc9b5ad1863d87e020350c
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00003-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2125ceee3992a093f9808efc9a720643a44e59b17baa77eddb0fbec6965ce5a
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00004-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2fc011e39d4a9c5958630b1bb946dec9ca54adbccc52804fed545490f995b20b
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00005-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae277c7c5aef0d0fea9e9e6b13113dfef0078a36c78805bebe14d32e9ab887d1
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00006-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d6376a884bc80a6ea95da83f963508e57d8508333c424bed7beda565e4aaa0f3
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00007-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3f07c8ec31edebe5ff80ed32d175475a044a0517d84c5eff6f6247ad0a4432d
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00008-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7af5021fe9afb9b4186a59a91e75aa60570c66fba43390b634387ee54bd73223
3
+ size 9513159151
tiiuae/falcon-40b/pytorch_model-00009-of-00009.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c5c794d3106945f514b5501904506485377c95ce2cef030373c7cbe7459ef610
3
+ size 7575220277
tiiuae/falcon-40b/pytorch_model.bin.index.json ADDED
@@ -0,0 +1,491 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 83671941120
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "pytorch_model-00009-of-00009.bin",
7
+ "transformer.h.0.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
8
+ "transformer.h.0.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
9
+ "transformer.h.0.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
10
+ "transformer.h.0.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
11
+ "transformer.h.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
12
+ "transformer.h.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
13
+ "transformer.h.0.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
14
+ "transformer.h.0.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
15
+ "transformer.h.1.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
16
+ "transformer.h.1.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
17
+ "transformer.h.1.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
18
+ "transformer.h.1.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
19
+ "transformer.h.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
20
+ "transformer.h.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
21
+ "transformer.h.1.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
22
+ "transformer.h.1.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
23
+ "transformer.h.10.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
24
+ "transformer.h.10.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
25
+ "transformer.h.10.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
26
+ "transformer.h.10.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
27
+ "transformer.h.10.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
28
+ "transformer.h.10.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
29
+ "transformer.h.10.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
30
+ "transformer.h.10.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
31
+ "transformer.h.11.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
32
+ "transformer.h.11.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
33
+ "transformer.h.11.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
34
+ "transformer.h.11.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
35
+ "transformer.h.11.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
36
+ "transformer.h.11.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
37
+ "transformer.h.11.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
38
+ "transformer.h.11.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
39
+ "transformer.h.12.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
40
+ "transformer.h.12.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
41
+ "transformer.h.12.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
42
+ "transformer.h.12.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
43
+ "transformer.h.12.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
44
+ "transformer.h.12.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
45
+ "transformer.h.12.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
46
+ "transformer.h.12.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
47
+ "transformer.h.13.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
48
+ "transformer.h.13.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
49
+ "transformer.h.13.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
50
+ "transformer.h.13.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
51
+ "transformer.h.13.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
52
+ "transformer.h.13.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
53
+ "transformer.h.13.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
54
+ "transformer.h.13.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
55
+ "transformer.h.14.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
56
+ "transformer.h.14.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
57
+ "transformer.h.14.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
58
+ "transformer.h.14.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
59
+ "transformer.h.14.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
60
+ "transformer.h.14.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
61
+ "transformer.h.14.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
62
+ "transformer.h.14.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
63
+ "transformer.h.15.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
64
+ "transformer.h.15.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
65
+ "transformer.h.15.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
66
+ "transformer.h.15.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
67
+ "transformer.h.15.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
68
+ "transformer.h.15.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
69
+ "transformer.h.15.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
70
+ "transformer.h.15.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
71
+ "transformer.h.16.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
72
+ "transformer.h.16.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
73
+ "transformer.h.16.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
74
+ "transformer.h.16.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
75
+ "transformer.h.16.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
76
+ "transformer.h.16.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
77
+ "transformer.h.16.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
78
+ "transformer.h.16.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
79
+ "transformer.h.17.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
80
+ "transformer.h.17.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
81
+ "transformer.h.17.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
82
+ "transformer.h.17.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
83
+ "transformer.h.17.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
84
+ "transformer.h.17.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
85
+ "transformer.h.17.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
86
+ "transformer.h.17.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
87
+ "transformer.h.18.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
88
+ "transformer.h.18.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
89
+ "transformer.h.18.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
90
+ "transformer.h.18.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
91
+ "transformer.h.18.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
92
+ "transformer.h.18.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
93
+ "transformer.h.18.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
94
+ "transformer.h.18.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
95
+ "transformer.h.19.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
96
+ "transformer.h.19.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
97
+ "transformer.h.19.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
98
+ "transformer.h.19.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
99
+ "transformer.h.19.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00009.bin",
100
+ "transformer.h.19.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00009.bin",
101
+ "transformer.h.19.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
102
+ "transformer.h.19.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
103
+ "transformer.h.2.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
104
+ "transformer.h.2.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
105
+ "transformer.h.2.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
106
+ "transformer.h.2.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
107
+ "transformer.h.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
108
+ "transformer.h.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
109
+ "transformer.h.2.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
110
+ "transformer.h.2.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
111
+ "transformer.h.20.ln_attn.bias": "pytorch_model-00003-of-00009.bin",
112
+ "transformer.h.20.ln_attn.weight": "pytorch_model-00003-of-00009.bin",
113
+ "transformer.h.20.ln_mlp.bias": "pytorch_model-00003-of-00009.bin",
114
+ "transformer.h.20.ln_mlp.weight": "pytorch_model-00003-of-00009.bin",
115
+ "transformer.h.20.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
116
+ "transformer.h.20.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
117
+ "transformer.h.20.self_attention.dense.weight": "pytorch_model-00003-of-00009.bin",
118
+ "transformer.h.20.self_attention.query_key_value.weight": "pytorch_model-00003-of-00009.bin",
119
+ "transformer.h.21.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
120
+ "transformer.h.21.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
121
+ "transformer.h.21.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
122
+ "transformer.h.21.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
123
+ "transformer.h.21.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
124
+ "transformer.h.21.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
125
+ "transformer.h.21.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
126
+ "transformer.h.21.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
127
+ "transformer.h.22.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
128
+ "transformer.h.22.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
129
+ "transformer.h.22.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
130
+ "transformer.h.22.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
131
+ "transformer.h.22.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
132
+ "transformer.h.22.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
133
+ "transformer.h.22.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
134
+ "transformer.h.22.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
135
+ "transformer.h.23.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
136
+ "transformer.h.23.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
137
+ "transformer.h.23.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
138
+ "transformer.h.23.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
139
+ "transformer.h.23.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
140
+ "transformer.h.23.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
141
+ "transformer.h.23.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
142
+ "transformer.h.23.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
143
+ "transformer.h.24.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
144
+ "transformer.h.24.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
145
+ "transformer.h.24.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
146
+ "transformer.h.24.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
147
+ "transformer.h.24.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
148
+ "transformer.h.24.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
149
+ "transformer.h.24.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
150
+ "transformer.h.24.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
151
+ "transformer.h.25.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
152
+ "transformer.h.25.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
153
+ "transformer.h.25.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
154
+ "transformer.h.25.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
155
+ "transformer.h.25.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
156
+ "transformer.h.25.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
157
+ "transformer.h.25.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
158
+ "transformer.h.25.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
159
+ "transformer.h.26.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
160
+ "transformer.h.26.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
161
+ "transformer.h.26.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
162
+ "transformer.h.26.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
163
+ "transformer.h.26.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00009.bin",
164
+ "transformer.h.26.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00009.bin",
165
+ "transformer.h.26.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
166
+ "transformer.h.26.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
167
+ "transformer.h.27.ln_attn.bias": "pytorch_model-00004-of-00009.bin",
168
+ "transformer.h.27.ln_attn.weight": "pytorch_model-00004-of-00009.bin",
169
+ "transformer.h.27.ln_mlp.bias": "pytorch_model-00004-of-00009.bin",
170
+ "transformer.h.27.ln_mlp.weight": "pytorch_model-00004-of-00009.bin",
171
+ "transformer.h.27.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
172
+ "transformer.h.27.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
173
+ "transformer.h.27.self_attention.dense.weight": "pytorch_model-00004-of-00009.bin",
174
+ "transformer.h.27.self_attention.query_key_value.weight": "pytorch_model-00004-of-00009.bin",
175
+ "transformer.h.28.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
176
+ "transformer.h.28.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
177
+ "transformer.h.28.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
178
+ "transformer.h.28.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
179
+ "transformer.h.28.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
180
+ "transformer.h.28.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
181
+ "transformer.h.28.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
182
+ "transformer.h.28.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
183
+ "transformer.h.29.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
184
+ "transformer.h.29.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
185
+ "transformer.h.29.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
186
+ "transformer.h.29.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
187
+ "transformer.h.29.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
188
+ "transformer.h.29.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
189
+ "transformer.h.29.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
190
+ "transformer.h.29.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
191
+ "transformer.h.3.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
192
+ "transformer.h.3.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
193
+ "transformer.h.3.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
194
+ "transformer.h.3.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
195
+ "transformer.h.3.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
196
+ "transformer.h.3.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
197
+ "transformer.h.3.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
198
+ "transformer.h.3.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
199
+ "transformer.h.30.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
200
+ "transformer.h.30.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
201
+ "transformer.h.30.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
202
+ "transformer.h.30.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
203
+ "transformer.h.30.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
204
+ "transformer.h.30.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
205
+ "transformer.h.30.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
206
+ "transformer.h.30.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
207
+ "transformer.h.31.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
208
+ "transformer.h.31.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
209
+ "transformer.h.31.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
210
+ "transformer.h.31.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
211
+ "transformer.h.31.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
212
+ "transformer.h.31.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
213
+ "transformer.h.31.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
214
+ "transformer.h.31.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
215
+ "transformer.h.32.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
216
+ "transformer.h.32.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
217
+ "transformer.h.32.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
218
+ "transformer.h.32.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
219
+ "transformer.h.32.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
220
+ "transformer.h.32.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
221
+ "transformer.h.32.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
222
+ "transformer.h.32.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
223
+ "transformer.h.33.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
224
+ "transformer.h.33.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
225
+ "transformer.h.33.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
226
+ "transformer.h.33.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
227
+ "transformer.h.33.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00009.bin",
228
+ "transformer.h.33.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00009.bin",
229
+ "transformer.h.33.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
230
+ "transformer.h.33.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
231
+ "transformer.h.34.ln_attn.bias": "pytorch_model-00005-of-00009.bin",
232
+ "transformer.h.34.ln_attn.weight": "pytorch_model-00005-of-00009.bin",
233
+ "transformer.h.34.ln_mlp.bias": "pytorch_model-00005-of-00009.bin",
234
+ "transformer.h.34.ln_mlp.weight": "pytorch_model-00005-of-00009.bin",
235
+ "transformer.h.34.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
236
+ "transformer.h.34.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
237
+ "transformer.h.34.self_attention.dense.weight": "pytorch_model-00005-of-00009.bin",
238
+ "transformer.h.34.self_attention.query_key_value.weight": "pytorch_model-00005-of-00009.bin",
239
+ "transformer.h.35.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
240
+ "transformer.h.35.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
241
+ "transformer.h.35.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
242
+ "transformer.h.35.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
243
+ "transformer.h.35.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
244
+ "transformer.h.35.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
245
+ "transformer.h.35.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
246
+ "transformer.h.35.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
247
+ "transformer.h.36.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
248
+ "transformer.h.36.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
249
+ "transformer.h.36.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
250
+ "transformer.h.36.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
251
+ "transformer.h.36.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
252
+ "transformer.h.36.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
253
+ "transformer.h.36.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
254
+ "transformer.h.36.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
255
+ "transformer.h.37.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
256
+ "transformer.h.37.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
257
+ "transformer.h.37.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
258
+ "transformer.h.37.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
259
+ "transformer.h.37.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
260
+ "transformer.h.37.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
261
+ "transformer.h.37.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
262
+ "transformer.h.37.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
263
+ "transformer.h.38.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
264
+ "transformer.h.38.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
265
+ "transformer.h.38.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
266
+ "transformer.h.38.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
267
+ "transformer.h.38.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
268
+ "transformer.h.38.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
269
+ "transformer.h.38.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
270
+ "transformer.h.38.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
271
+ "transformer.h.39.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
272
+ "transformer.h.39.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
273
+ "transformer.h.39.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
274
+ "transformer.h.39.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
275
+ "transformer.h.39.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
276
+ "transformer.h.39.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
277
+ "transformer.h.39.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
278
+ "transformer.h.39.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
279
+ "transformer.h.4.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
280
+ "transformer.h.4.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
281
+ "transformer.h.4.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
282
+ "transformer.h.4.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
283
+ "transformer.h.4.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
284
+ "transformer.h.4.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
285
+ "transformer.h.4.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
286
+ "transformer.h.4.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
287
+ "transformer.h.40.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
288
+ "transformer.h.40.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
289
+ "transformer.h.40.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
290
+ "transformer.h.40.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
291
+ "transformer.h.40.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00009.bin",
292
+ "transformer.h.40.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00009.bin",
293
+ "transformer.h.40.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
294
+ "transformer.h.40.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
295
+ "transformer.h.41.ln_attn.bias": "pytorch_model-00006-of-00009.bin",
296
+ "transformer.h.41.ln_attn.weight": "pytorch_model-00006-of-00009.bin",
297
+ "transformer.h.41.ln_mlp.bias": "pytorch_model-00006-of-00009.bin",
298
+ "transformer.h.41.ln_mlp.weight": "pytorch_model-00006-of-00009.bin",
299
+ "transformer.h.41.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
300
+ "transformer.h.41.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
301
+ "transformer.h.41.self_attention.dense.weight": "pytorch_model-00006-of-00009.bin",
302
+ "transformer.h.41.self_attention.query_key_value.weight": "pytorch_model-00006-of-00009.bin",
303
+ "transformer.h.42.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
304
+ "transformer.h.42.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
305
+ "transformer.h.42.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
306
+ "transformer.h.42.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
307
+ "transformer.h.42.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
308
+ "transformer.h.42.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
309
+ "transformer.h.42.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
310
+ "transformer.h.42.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
311
+ "transformer.h.43.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
312
+ "transformer.h.43.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
313
+ "transformer.h.43.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
314
+ "transformer.h.43.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
315
+ "transformer.h.43.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
316
+ "transformer.h.43.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
317
+ "transformer.h.43.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
318
+ "transformer.h.43.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
319
+ "transformer.h.44.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
320
+ "transformer.h.44.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
321
+ "transformer.h.44.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
322
+ "transformer.h.44.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
323
+ "transformer.h.44.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
324
+ "transformer.h.44.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
325
+ "transformer.h.44.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
326
+ "transformer.h.44.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
327
+ "transformer.h.45.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
328
+ "transformer.h.45.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
329
+ "transformer.h.45.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
330
+ "transformer.h.45.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
331
+ "transformer.h.45.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
332
+ "transformer.h.45.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
333
+ "transformer.h.45.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
334
+ "transformer.h.45.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
335
+ "transformer.h.46.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
336
+ "transformer.h.46.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
337
+ "transformer.h.46.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
338
+ "transformer.h.46.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
339
+ "transformer.h.46.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
340
+ "transformer.h.46.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
341
+ "transformer.h.46.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
342
+ "transformer.h.46.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
343
+ "transformer.h.47.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
344
+ "transformer.h.47.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
345
+ "transformer.h.47.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
346
+ "transformer.h.47.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
347
+ "transformer.h.47.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00009.bin",
348
+ "transformer.h.47.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00009.bin",
349
+ "transformer.h.47.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
350
+ "transformer.h.47.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
351
+ "transformer.h.48.ln_attn.bias": "pytorch_model-00007-of-00009.bin",
352
+ "transformer.h.48.ln_attn.weight": "pytorch_model-00007-of-00009.bin",
353
+ "transformer.h.48.ln_mlp.bias": "pytorch_model-00007-of-00009.bin",
354
+ "transformer.h.48.ln_mlp.weight": "pytorch_model-00007-of-00009.bin",
355
+ "transformer.h.48.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
356
+ "transformer.h.48.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
357
+ "transformer.h.48.self_attention.dense.weight": "pytorch_model-00007-of-00009.bin",
358
+ "transformer.h.48.self_attention.query_key_value.weight": "pytorch_model-00007-of-00009.bin",
359
+ "transformer.h.49.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
360
+ "transformer.h.49.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
361
+ "transformer.h.49.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
362
+ "transformer.h.49.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
363
+ "transformer.h.49.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
364
+ "transformer.h.49.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
365
+ "transformer.h.49.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
366
+ "transformer.h.49.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
367
+ "transformer.h.5.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
368
+ "transformer.h.5.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
369
+ "transformer.h.5.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
370
+ "transformer.h.5.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
371
+ "transformer.h.5.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00009.bin",
372
+ "transformer.h.5.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00009.bin",
373
+ "transformer.h.5.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
374
+ "transformer.h.5.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
375
+ "transformer.h.50.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
376
+ "transformer.h.50.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
377
+ "transformer.h.50.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
378
+ "transformer.h.50.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
379
+ "transformer.h.50.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
380
+ "transformer.h.50.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
381
+ "transformer.h.50.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
382
+ "transformer.h.50.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
383
+ "transformer.h.51.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
384
+ "transformer.h.51.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
385
+ "transformer.h.51.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
386
+ "transformer.h.51.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
387
+ "transformer.h.51.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
388
+ "transformer.h.51.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
389
+ "transformer.h.51.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
390
+ "transformer.h.51.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
391
+ "transformer.h.52.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
392
+ "transformer.h.52.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
393
+ "transformer.h.52.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
394
+ "transformer.h.52.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
395
+ "transformer.h.52.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
396
+ "transformer.h.52.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
397
+ "transformer.h.52.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
398
+ "transformer.h.52.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
399
+ "transformer.h.53.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
400
+ "transformer.h.53.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
401
+ "transformer.h.53.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
402
+ "transformer.h.53.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
403
+ "transformer.h.53.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
404
+ "transformer.h.53.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
405
+ "transformer.h.53.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
406
+ "transformer.h.53.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
407
+ "transformer.h.54.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
408
+ "transformer.h.54.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
409
+ "transformer.h.54.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
410
+ "transformer.h.54.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
411
+ "transformer.h.54.mlp.dense_4h_to_h.weight": "pytorch_model-00008-of-00009.bin",
412
+ "transformer.h.54.mlp.dense_h_to_4h.weight": "pytorch_model-00008-of-00009.bin",
413
+ "transformer.h.54.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
414
+ "transformer.h.54.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
415
+ "transformer.h.55.ln_attn.bias": "pytorch_model-00008-of-00009.bin",
416
+ "transformer.h.55.ln_attn.weight": "pytorch_model-00008-of-00009.bin",
417
+ "transformer.h.55.ln_mlp.bias": "pytorch_model-00008-of-00009.bin",
418
+ "transformer.h.55.ln_mlp.weight": "pytorch_model-00008-of-00009.bin",
419
+ "transformer.h.55.mlp.dense_4h_to_h.weight": "pytorch_model-00009-of-00009.bin",
420
+ "transformer.h.55.mlp.dense_h_to_4h.weight": "pytorch_model-00009-of-00009.bin",
421
+ "transformer.h.55.self_attention.dense.weight": "pytorch_model-00008-of-00009.bin",
422
+ "transformer.h.55.self_attention.query_key_value.weight": "pytorch_model-00008-of-00009.bin",
423
+ "transformer.h.56.ln_attn.bias": "pytorch_model-00009-of-00009.bin",
424
+ "transformer.h.56.ln_attn.weight": "pytorch_model-00009-of-00009.bin",
425
+ "transformer.h.56.ln_mlp.bias": "pytorch_model-00009-of-00009.bin",
426
+ "transformer.h.56.ln_mlp.weight": "pytorch_model-00009-of-00009.bin",
427
+ "transformer.h.56.mlp.dense_4h_to_h.weight": "pytorch_model-00009-of-00009.bin",
428
+ "transformer.h.56.mlp.dense_h_to_4h.weight": "pytorch_model-00009-of-00009.bin",
429
+ "transformer.h.56.self_attention.dense.weight": "pytorch_model-00009-of-00009.bin",
430
+ "transformer.h.56.self_attention.query_key_value.weight": "pytorch_model-00009-of-00009.bin",
431
+ "transformer.h.57.ln_attn.bias": "pytorch_model-00009-of-00009.bin",
432
+ "transformer.h.57.ln_attn.weight": "pytorch_model-00009-of-00009.bin",
433
+ "transformer.h.57.ln_mlp.bias": "pytorch_model-00009-of-00009.bin",
434
+ "transformer.h.57.ln_mlp.weight": "pytorch_model-00009-of-00009.bin",
435
+ "transformer.h.57.mlp.dense_4h_to_h.weight": "pytorch_model-00009-of-00009.bin",
436
+ "transformer.h.57.mlp.dense_h_to_4h.weight": "pytorch_model-00009-of-00009.bin",
437
+ "transformer.h.57.self_attention.dense.weight": "pytorch_model-00009-of-00009.bin",
438
+ "transformer.h.57.self_attention.query_key_value.weight": "pytorch_model-00009-of-00009.bin",
439
+ "transformer.h.58.ln_attn.bias": "pytorch_model-00009-of-00009.bin",
440
+ "transformer.h.58.ln_attn.weight": "pytorch_model-00009-of-00009.bin",
441
+ "transformer.h.58.ln_mlp.bias": "pytorch_model-00009-of-00009.bin",
442
+ "transformer.h.58.ln_mlp.weight": "pytorch_model-00009-of-00009.bin",
443
+ "transformer.h.58.mlp.dense_4h_to_h.weight": "pytorch_model-00009-of-00009.bin",
444
+ "transformer.h.58.mlp.dense_h_to_4h.weight": "pytorch_model-00009-of-00009.bin",
445
+ "transformer.h.58.self_attention.dense.weight": "pytorch_model-00009-of-00009.bin",
446
+ "transformer.h.58.self_attention.query_key_value.weight": "pytorch_model-00009-of-00009.bin",
447
+ "transformer.h.59.ln_attn.bias": "pytorch_model-00009-of-00009.bin",
448
+ "transformer.h.59.ln_attn.weight": "pytorch_model-00009-of-00009.bin",
449
+ "transformer.h.59.ln_mlp.bias": "pytorch_model-00009-of-00009.bin",
450
+ "transformer.h.59.ln_mlp.weight": "pytorch_model-00009-of-00009.bin",
451
+ "transformer.h.59.mlp.dense_4h_to_h.weight": "pytorch_model-00009-of-00009.bin",
452
+ "transformer.h.59.mlp.dense_h_to_4h.weight": "pytorch_model-00009-of-00009.bin",
453
+ "transformer.h.59.self_attention.dense.weight": "pytorch_model-00009-of-00009.bin",
454
+ "transformer.h.59.self_attention.query_key_value.weight": "pytorch_model-00009-of-00009.bin",
455
+ "transformer.h.6.ln_attn.bias": "pytorch_model-00001-of-00009.bin",
456
+ "transformer.h.6.ln_attn.weight": "pytorch_model-00001-of-00009.bin",
457
+ "transformer.h.6.ln_mlp.bias": "pytorch_model-00001-of-00009.bin",
458
+ "transformer.h.6.ln_mlp.weight": "pytorch_model-00001-of-00009.bin",
459
+ "transformer.h.6.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
460
+ "transformer.h.6.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
461
+ "transformer.h.6.self_attention.dense.weight": "pytorch_model-00001-of-00009.bin",
462
+ "transformer.h.6.self_attention.query_key_value.weight": "pytorch_model-00001-of-00009.bin",
463
+ "transformer.h.7.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
464
+ "transformer.h.7.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
465
+ "transformer.h.7.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
466
+ "transformer.h.7.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
467
+ "transformer.h.7.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
468
+ "transformer.h.7.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
469
+ "transformer.h.7.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
470
+ "transformer.h.7.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
471
+ "transformer.h.8.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
472
+ "transformer.h.8.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
473
+ "transformer.h.8.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
474
+ "transformer.h.8.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
475
+ "transformer.h.8.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
476
+ "transformer.h.8.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
477
+ "transformer.h.8.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
478
+ "transformer.h.8.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
479
+ "transformer.h.9.ln_attn.bias": "pytorch_model-00002-of-00009.bin",
480
+ "transformer.h.9.ln_attn.weight": "pytorch_model-00002-of-00009.bin",
481
+ "transformer.h.9.ln_mlp.bias": "pytorch_model-00002-of-00009.bin",
482
+ "transformer.h.9.ln_mlp.weight": "pytorch_model-00002-of-00009.bin",
483
+ "transformer.h.9.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00009.bin",
484
+ "transformer.h.9.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00009.bin",
485
+ "transformer.h.9.self_attention.dense.weight": "pytorch_model-00002-of-00009.bin",
486
+ "transformer.h.9.self_attention.query_key_value.weight": "pytorch_model-00002-of-00009.bin",
487
+ "transformer.ln_f.bias": "pytorch_model-00009-of-00009.bin",
488
+ "transformer.ln_f.weight": "pytorch_model-00009-of-00009.bin",
489
+ "transformer.word_embeddings.weight": "pytorch_model-00001-of-00009.bin"
490
+ }
491
+ }
tiiuae/falcon-40b/special_tokens_map.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ ">>TITLE<<",
4
+ ">>ABSTRACT<<",
5
+ ">>INTRODUCTION<<",
6
+ ">>SUMMARY<<",
7
+ ">>COMMENT<<",
8
+ ">>ANSWER<<",
9
+ ">>QUESTION<<",
10
+ ">>DOMAIN<<",
11
+ ">>PREFIX<<",
12
+ ">>SUFFIX<<",
13
+ ">>MIDDLE<<"
14
+ ],
15
+ "eos_token": "<|endoftext|>"
16
+ }
tiiuae/falcon-40b/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tiiuae/falcon-40b/tokenizer_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "eos_token": "<|endoftext|>",
4
+ "model_max_length": 2048,
5
+ "special_tokens_map_file": null,
6
+ "tokenizer_class": "PreTrainedTokenizerFast"
7
+ }