sharpenb commited on
Commit
e91f711
1 Parent(s): 686a892

Upload folder using huggingface_hub (#1)

Browse files

- ef008a9061e2165d77123d0afe3e629b4958bd50bb5509f0886bb45e4848583f (b679a0c0908c7747867171876e01c90008beeb10)
- c92389534b9436593e249614948a25f621625fbad349ec9b60b1a61d9a214fc5 (c13a6c1e41ad8af22df0ccbafc939879ed07e175)
- 5b05e167270b35b199c821afe747c0e20a35cbae71b0b91492b8d901c0601f6e (7e95919de02a707b34f1bd48b47013d59e9fc1ab)

README.md ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pruna-engine
3
+ thumbnail: "https://assets-global.website-files.com/646b351987a8d8ce158d1940/64ec9e96b4334c0e1ac41504_Logo%20with%20white%20text.svg"
4
+ metrics:
5
+ - memory_disk
6
+ - memory_inference
7
+ - inference_latency
8
+ - inference_throughput
9
+ - inference_CO2_emissions
10
+ - inference_energy_consumption
11
+ ---
12
+ <!-- header start -->
13
+ <!-- 200823 -->
14
+ <div style="width: auto; margin-left: auto; margin-right: auto">
15
+ <a href="https://www.pruna.ai/" target="_blank" rel="noopener noreferrer">
16
+ <img src="https://i.imgur.com/eDAlcgk.png" alt="PrunaAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
17
+ </a>
18
+ </div>
19
+ <!-- header end -->
20
+
21
+ [![Twitter](https://img.shields.io/twitter/follow/PrunaAI?style=social)](https://twitter.com/PrunaAI)
22
+ [![GitHub](https://img.shields.io/github/followers/PrunaAI?label=Follow%20%40PrunaAI&style=social)](https://github.com/PrunaAI)
23
+ [![LinkedIn](https://img.shields.io/badge/LinkedIn-Connect-blue)](https://www.linkedin.com/company/93832878/admin/feed/posts/?feedType=following)
24
+ [![Discord](https://img.shields.io/badge/Discord-Join%20Us-blue?style=social&logo=discord)](https://discord.gg/CP4VSgck)
25
+
26
+ # Simply make AI models cheaper, smaller, faster, and greener!
27
+
28
+ - Give a thumbs up if you like this model!
29
+ - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
30
+ - Request access to easily compress your *own* AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
31
+ - Read the documentations to know more [here](https://pruna-ai-pruna.readthedocs-hosted.com/en/latest/)
32
+ - Join Pruna AI community on Discord [here](https://discord.gg/CP4VSgck) to share feedback/suggestions or get help.
33
+
34
+ ## Results
35
+
36
+ ![image info](./plots.png)
37
+
38
+ **Frequently Asked Questions**
39
+ - ***How does the compression work?*** The model is compressed with llm-int8.
40
+ - ***How does the model quality change?*** The quality of the model output might vary compared to the base model.
41
+ - ***How is the model efficiency evaluated?*** These results were obtained on NVIDIA A100-PCIE-40GB with configuration described in `model/smash_config.json` and are obtained after a hardware warmup. The smashed model is directly compared to the original base model. Efficiency results may vary in other settings (e.g. other hardware, image size, batch size, ...). We recommend to directly run them in the use-case conditions to know if the smashed model can benefit you.
42
+ - ***What is the model format?*** We use safetensors.
43
+ - ***What is the naming convention for Pruna Huggingface models?*** We take the original model name and append "turbo", "tiny", or "green" if the smashed model has a measured inference speed, inference memory, or inference energy consumption which is less than 90% of the original base model.
44
+ - ***How to compress my own models?*** You can request premium access to more compression methods and tech support for your specific use-cases [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
45
+ - ***What are "first" metrics?*** Results mentioning "first" are obtained after the first run of the model. The first run might take more memory or be slower than the subsequent runs due cuda overheads.
46
+ - ***What are "Sync" and "Async" metrics?*** "Sync" metrics are obtained by syncing all GPU processes and stop measurement when all of them are executed. "Async" metrics are obtained without syncing all GPU processes and stop when the model output can be used by the CPU. We provide both metrics since both could be relevant depending on the use-case. We recommend to test the efficiency gains directly in your use-cases.
47
+
48
+ ## Setup
49
+
50
+ You can run the smashed model with these steps:
51
+
52
+ 0. Check requirements from the original repo NousResearch/Yarn-Mistral-7b-128k installed. In particular, check python, cuda, and transformers versions.
53
+ 1. Make sure that you have installed quantization related packages.
54
+ ```bash
55
+ pip install transformers accelerate bitsandbytes>0.37.0
56
+ ```
57
+ 2. Load & run the model.
58
+ ```python
59
+ from transformers import AutoModelForCausalLM, AutoTokenizer
60
+
61
+ model = AutoModelForCausalLM.from_pretrained("PrunaAI/NousResearch-Yarn-Mistral-7b-128k-bnb-8bit-smashed",
62
+ trust_remote_code=True)
63
+ tokenizer = AutoTokenizer.from_pretrained("NousResearch/Yarn-Mistral-7b-128k")
64
+
65
+ input_ids = tokenizer("What is the color of prunes?,", return_tensors='pt').to(model.device)["input_ids"]
66
+
67
+ outputs = model.generate(input_ids, max_new_tokens=216)
68
+ ```
69
+
70
+ ## Configurations
71
+
72
+ The configuration info are in `smash_config.json`.
73
+
74
+ ## Credits & License
75
+
76
+ The license of the smashed model follows the license of the original model. Please check the license of the original model NousResearch/Yarn-Mistral-7b-128k before using this model which provided the base model. The license of the `pruna-engine` is [here](https://pypi.org/project/pruna-engine/) on Pypi.
77
+
78
+ ## Want to compress other models?
79
+
80
+ - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
81
+ - Request access to easily compress your own AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
config.json ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/tmp/tmpf81p3y9u",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_mistral.MistralConfig",
8
+ "AutoModelForCausalLM": "modeling_mistral_yarn.MistralForCausalLM"
9
+ },
10
+ "bos_token_id": 1,
11
+ "eos_token_id": 2,
12
+ "hidden_act": "silu",
13
+ "hidden_size": 4096,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 14336,
16
+ "max_position_embeddings": 32768,
17
+ "max_sequence_length": 131072,
18
+ "model_type": "mistral",
19
+ "num_attention_heads": 32,
20
+ "num_hidden_layers": 32,
21
+ "num_key_value_heads": 8,
22
+ "quantization_config": {
23
+ "bnb_4bit_compute_dtype": "bfloat16",
24
+ "bnb_4bit_quant_type": "fp4",
25
+ "bnb_4bit_use_double_quant": true,
26
+ "llm_int8_enable_fp32_cpu_offload": false,
27
+ "llm_int8_has_fp16_weight": false,
28
+ "llm_int8_skip_modules": [
29
+ "lm_head"
30
+ ],
31
+ "llm_int8_threshold": 6.0,
32
+ "load_in_4bit": false,
33
+ "load_in_8bit": true,
34
+ "quant_method": "bitsandbytes"
35
+ },
36
+ "rms_norm_eps": 1e-05,
37
+ "rope_scaling": {
38
+ "factor": 16.0,
39
+ "finetuned": true,
40
+ "original_max_position_embeddings": 8192,
41
+ "type": "yarn"
42
+ },
43
+ "rope_theta": 10000.0,
44
+ "sliding_window": 131072,
45
+ "tie_word_embeddings": false,
46
+ "torch_dtype": "float16",
47
+ "transformers_version": "4.37.1",
48
+ "use_cache": true,
49
+ "vocab_size": 32000
50
+ }
configuration_mistral.py ADDED
@@ -0,0 +1,181 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 Mistral AI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ Mistral model configuration"""
16
+
17
+ from transformers.configuration_utils import PretrainedConfig
18
+ from transformers.utils import logging
19
+
20
+
21
+ logger = logging.get_logger(__name__)
22
+
23
+ MISTRAL_PRETRAINED_CONFIG_ARCHIVE_MAP = {
24
+ "mistralai/Mistral-7B-v0.1": "https://huggingface.co/mistralai/Mistral-7B-v0.1/resolve/main/config.json",
25
+ "mistralai/Mistral-7B-Instruct-v0.1": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/resolve/main/config.json",
26
+ }
27
+
28
+
29
+ class MistralConfig(PretrainedConfig):
30
+ r"""
31
+ This is the configuration class to store the configuration of a [`MistralModel`]. It is used to instantiate an
32
+ Mistral model according to the specified arguments, defining the model architecture. Instantiating a configuration
33
+ with the defaults will yield a similar configuration to that of the Mistral-7B-v0.1 or Mistral-7B-Instruct-v0.1.
34
+
35
+ [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
36
+ [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
37
+
38
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
39
+ documentation from [`PretrainedConfig`] for more information.
40
+
41
+
42
+ Args:
43
+ vocab_size (`int`, *optional*, defaults to 32000):
44
+ Vocabulary size of the Mistral model. Defines the number of different tokens that can be represented by the
45
+ `inputs_ids` passed when calling [`MistralModel`]
46
+ hidden_size (`int`, *optional*, defaults to 4096):
47
+ Dimension of the hidden representations.
48
+ intermediate_size (`int`, *optional*, defaults to 14336):
49
+ Dimension of the MLP representations.
50
+ num_hidden_layers (`int`, *optional*, defaults to 32):
51
+ Number of hidden layers in the Transformer encoder.
52
+ num_attention_heads (`int`, *optional*, defaults to 32):
53
+ Number of attention heads for each attention layer in the Transformer encoder.
54
+ num_key_value_heads (`int`, *optional*, defaults to 8):
55
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
56
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
57
+ `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
58
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
59
+ by meanpooling all the original heads within that group. For more details checkout [this
60
+ paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `8`.
61
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
62
+ The non-linear activation function (function or string) in the decoder.
63
+ max_position_embeddings (`int`, *optional*, defaults to `4096*32`):
64
+ The maximum sequence length that this model might ever be used with. Mistral's sliding window attention
65
+ allows sequence of up to 4096*32 tokens.
66
+ initializer_range (`float`, *optional*, defaults to 0.02):
67
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
68
+ rms_norm_eps (`float`, *optional*, defaults to 1e-06):
69
+ The epsilon used by the rms normalization layers.
70
+ use_cache (`bool`, *optional*, defaults to `True`):
71
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
72
+ relevant if `config.is_decoder=True`.
73
+ pad_token_id (`int`, *optional*):
74
+ The id of the padding token.
75
+ bos_token_id (`int`, *optional*, defaults to 1):
76
+ The id of the "beginning-of-sequence" token.
77
+ eos_token_id (`int`, *optional*, defaults to 2):
78
+ The id of the "end-of-sequence" token.
79
+ tie_word_embeddings (`bool`, *optional*, defaults to `False`):
80
+ Whether the model's input and output word embeddings should be tied.
81
+ rope_scaling (`Dict`, *optional*):
82
+ Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports three scaling
83
+ strategies: linear and dynamic. Their scaling factor must be an float greater than 1. The expected format
84
+ is `{"type": strategy name, "factor": scaling factor}`.
85
+ rope_theta (`float`, *optional*, defaults to 10000.0):
86
+ The base period of the RoPE embeddings.
87
+ sliding_window (`int`, *optional*, defaults to 4096):
88
+ Sliding window attention window size. If not specified, will default to `4096`.
89
+
90
+
91
+ ```python
92
+ >>> from transformers import MistralModel, MistralConfig
93
+
94
+ >>> # Initializing a Mistral 7B style configuration
95
+ >>> configuration = MistralConfig()
96
+
97
+ >>> # Initializing a model from the Mistral 7B style configuration
98
+ >>> model = MistralModel(configuration)
99
+
100
+ >>> # Accessing the model configuration
101
+ >>> configuration = model.config
102
+ ```"""
103
+
104
+ model_type = "mistral"
105
+ keys_to_ignore_at_inference = ["past_key_values"]
106
+
107
+ def __init__(
108
+ self,
109
+ vocab_size=32000,
110
+ hidden_size=4096,
111
+ intermediate_size=14336,
112
+ num_hidden_layers=32,
113
+ num_attention_heads=32,
114
+ num_key_value_heads=8,
115
+ hidden_act="silu",
116
+ max_position_embeddings=4096 * 32,
117
+ initializer_range=0.02,
118
+ rms_norm_eps=1e-6,
119
+ use_cache=True,
120
+ pad_token_id=None,
121
+ bos_token_id=1,
122
+ eos_token_id=2,
123
+ tie_word_embeddings=False,
124
+ rope_scaling=None,
125
+ rope_theta=10000.0,
126
+ sliding_window=4096,
127
+ **kwargs,
128
+ ):
129
+ self.vocab_size = vocab_size
130
+ self.max_position_embeddings = max_position_embeddings
131
+ self.hidden_size = hidden_size
132
+ self.intermediate_size = intermediate_size
133
+ self.num_hidden_layers = num_hidden_layers
134
+ self.num_attention_heads = num_attention_heads
135
+ self.sliding_window = sliding_window
136
+
137
+ # for backward compatibility
138
+ if num_key_value_heads is None:
139
+ num_key_value_heads = num_attention_heads
140
+
141
+ self.num_key_value_heads = num_key_value_heads
142
+ self.hidden_act = hidden_act
143
+ self.initializer_range = initializer_range
144
+ self.rms_norm_eps = rms_norm_eps
145
+ self.use_cache = use_cache
146
+ self.rope_scaling = rope_scaling
147
+ self._rope_scaling_validation()
148
+ self.rope_theta = rope_theta
149
+
150
+ super().__init__(
151
+ pad_token_id=pad_token_id,
152
+ bos_token_id=bos_token_id,
153
+ eos_token_id=eos_token_id,
154
+ tie_word_embeddings=tie_word_embeddings,
155
+ **kwargs,
156
+ )
157
+
158
+ def _rope_scaling_validation(self):
159
+ """
160
+ Validate the `rope_scaling` configuration.
161
+ """
162
+ if self.rope_scaling is None:
163
+ return
164
+
165
+ if not isinstance(self.rope_scaling, dict):
166
+ raise ValueError(
167
+ "`rope_scaling` must be a dictionary, "
168
+ f"got {self.rope_scaling}"
169
+ )
170
+ rope_scaling_type = self.rope_scaling.get("type", None)
171
+ rope_scaling_factor = self.rope_scaling.get("factor", None)
172
+ if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic", "yarn", "dynamic-yarn"]:
173
+ raise ValueError(
174
+ f"`rope_scaling`'s name field must be one of ['linear', 'dynamic', 'yarn', 'dynamic-yarn'], got {rope_scaling_type}"
175
+ )
176
+ if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
177
+ raise ValueError(f"`rope_scaling`'s factor field must be an float > 1, got {rope_scaling_factor}")
178
+ if rope_scaling_type == "yarn" or rope_scaling_type == "dynamic-yarn":
179
+ original_max_position_embeddings = self.rope_scaling.get("original_max_position_embeddings", None)
180
+ if original_max_position_embeddings is None or not isinstance(original_max_position_embeddings, int):
181
+ raise ValueError(f"`rope_scaling.original_max_position_embeddings` must be set to an int when using yarn, and dynamic-yarn")
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.37.1"
6
+ }
model-00001-of-00002.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa85e0c7985680f338e12dcb341a573aeda42766d3a69e65082509405932b4b6
3
+ size 4947079848
model-00002-of-00002.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5cf601920bc57be9dc77f1713e35d7a269947d800e42378c208e3e78d3e2d1d1
3
+ size 2562623232
model.safetensors.index.json ADDED
@@ -0,0 +1,522 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 7509647360
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00002-of-00002.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
8
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
9
+ "model.layers.0.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
10
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
11
+ "model.layers.0.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
12
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
13
+ "model.layers.0.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
14
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
15
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
16
+ "model.layers.0.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
17
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
18
+ "model.layers.0.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
19
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
20
+ "model.layers.0.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
21
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
22
+ "model.layers.0.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
23
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
24
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
25
+ "model.layers.1.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
26
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
27
+ "model.layers.1.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
28
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
29
+ "model.layers.1.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
30
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
31
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
32
+ "model.layers.1.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
33
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
34
+ "model.layers.1.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
35
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
36
+ "model.layers.1.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
37
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
38
+ "model.layers.1.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
39
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
40
+ "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
41
+ "model.layers.10.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
42
+ "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
43
+ "model.layers.10.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
44
+ "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
45
+ "model.layers.10.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
46
+ "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
47
+ "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
48
+ "model.layers.10.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
49
+ "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
50
+ "model.layers.10.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
51
+ "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
52
+ "model.layers.10.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
53
+ "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
54
+ "model.layers.10.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
55
+ "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
56
+ "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
57
+ "model.layers.11.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
58
+ "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
59
+ "model.layers.11.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
60
+ "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
61
+ "model.layers.11.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
62
+ "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
63
+ "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
64
+ "model.layers.11.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
65
+ "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
66
+ "model.layers.11.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
67
+ "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
68
+ "model.layers.11.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
69
+ "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
70
+ "model.layers.11.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
71
+ "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
72
+ "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
73
+ "model.layers.12.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
74
+ "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
75
+ "model.layers.12.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
76
+ "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
77
+ "model.layers.12.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
78
+ "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
79
+ "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
80
+ "model.layers.12.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
81
+ "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
82
+ "model.layers.12.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
83
+ "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
84
+ "model.layers.12.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
85
+ "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
86
+ "model.layers.12.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
87
+ "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
88
+ "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
89
+ "model.layers.13.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
90
+ "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
91
+ "model.layers.13.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
92
+ "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
93
+ "model.layers.13.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
94
+ "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
95
+ "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
96
+ "model.layers.13.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
97
+ "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
98
+ "model.layers.13.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
99
+ "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
100
+ "model.layers.13.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
101
+ "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
102
+ "model.layers.13.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
103
+ "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
104
+ "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
105
+ "model.layers.14.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
106
+ "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
107
+ "model.layers.14.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
108
+ "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
109
+ "model.layers.14.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
110
+ "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
111
+ "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
112
+ "model.layers.14.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
113
+ "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
114
+ "model.layers.14.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
115
+ "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
116
+ "model.layers.14.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
117
+ "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
118
+ "model.layers.14.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
119
+ "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
120
+ "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
121
+ "model.layers.15.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
122
+ "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
123
+ "model.layers.15.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
124
+ "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
125
+ "model.layers.15.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
126
+ "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
127
+ "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
128
+ "model.layers.15.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
129
+ "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
130
+ "model.layers.15.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
131
+ "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
132
+ "model.layers.15.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
133
+ "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
134
+ "model.layers.15.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
135
+ "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
136
+ "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
137
+ "model.layers.16.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
138
+ "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
139
+ "model.layers.16.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
140
+ "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
141
+ "model.layers.16.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
142
+ "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
143
+ "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
144
+ "model.layers.16.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
145
+ "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
146
+ "model.layers.16.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
147
+ "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
148
+ "model.layers.16.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
149
+ "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
150
+ "model.layers.16.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
151
+ "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
152
+ "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
153
+ "model.layers.17.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
154
+ "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
155
+ "model.layers.17.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
156
+ "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
157
+ "model.layers.17.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
158
+ "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
159
+ "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
160
+ "model.layers.17.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
161
+ "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
162
+ "model.layers.17.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
163
+ "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
164
+ "model.layers.17.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
165
+ "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
166
+ "model.layers.17.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
167
+ "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
168
+ "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
169
+ "model.layers.18.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
170
+ "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
171
+ "model.layers.18.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
172
+ "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
173
+ "model.layers.18.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
174
+ "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
175
+ "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
176
+ "model.layers.18.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
177
+ "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
178
+ "model.layers.18.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
179
+ "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
180
+ "model.layers.18.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
181
+ "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
182
+ "model.layers.18.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
183
+ "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
184
+ "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
185
+ "model.layers.19.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
186
+ "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
187
+ "model.layers.19.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
188
+ "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
189
+ "model.layers.19.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
190
+ "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
191
+ "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
192
+ "model.layers.19.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
193
+ "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
194
+ "model.layers.19.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
195
+ "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
196
+ "model.layers.19.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
197
+ "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
198
+ "model.layers.19.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
199
+ "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
200
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
201
+ "model.layers.2.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
202
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
203
+ "model.layers.2.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
204
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
205
+ "model.layers.2.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
206
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
207
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
208
+ "model.layers.2.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
209
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
210
+ "model.layers.2.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
211
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
212
+ "model.layers.2.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
213
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
214
+ "model.layers.2.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
215
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
216
+ "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
217
+ "model.layers.20.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
218
+ "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
219
+ "model.layers.20.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
220
+ "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
221
+ "model.layers.20.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
222
+ "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
223
+ "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
224
+ "model.layers.20.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
225
+ "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
226
+ "model.layers.20.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
227
+ "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
228
+ "model.layers.20.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
229
+ "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
230
+ "model.layers.20.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
231
+ "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
232
+ "model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
233
+ "model.layers.21.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
234
+ "model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
235
+ "model.layers.21.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
236
+ "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
237
+ "model.layers.21.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
238
+ "model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
239
+ "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
240
+ "model.layers.21.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
241
+ "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
242
+ "model.layers.21.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
243
+ "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
244
+ "model.layers.21.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
245
+ "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
246
+ "model.layers.21.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
247
+ "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
248
+ "model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
249
+ "model.layers.22.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
250
+ "model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
251
+ "model.layers.22.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
252
+ "model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
253
+ "model.layers.22.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
254
+ "model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
255
+ "model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
256
+ "model.layers.22.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
257
+ "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
258
+ "model.layers.22.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
259
+ "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
260
+ "model.layers.22.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
261
+ "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
262
+ "model.layers.22.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
263
+ "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
264
+ "model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
265
+ "model.layers.23.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
266
+ "model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
267
+ "model.layers.23.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
268
+ "model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
269
+ "model.layers.23.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
270
+ "model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
271
+ "model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
272
+ "model.layers.23.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
273
+ "model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
274
+ "model.layers.23.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
275
+ "model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
276
+ "model.layers.23.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
277
+ "model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
278
+ "model.layers.23.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
279
+ "model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
280
+ "model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
281
+ "model.layers.24.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
282
+ "model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
283
+ "model.layers.24.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
284
+ "model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
285
+ "model.layers.24.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
286
+ "model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
287
+ "model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
288
+ "model.layers.24.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
289
+ "model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
290
+ "model.layers.24.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
291
+ "model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
292
+ "model.layers.24.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
293
+ "model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
294
+ "model.layers.24.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
295
+ "model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
296
+ "model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
297
+ "model.layers.25.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
298
+ "model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
299
+ "model.layers.25.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
300
+ "model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
301
+ "model.layers.25.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
302
+ "model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
303
+ "model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
304
+ "model.layers.25.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
305
+ "model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
306
+ "model.layers.25.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
307
+ "model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
308
+ "model.layers.25.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
309
+ "model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
310
+ "model.layers.25.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
311
+ "model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
312
+ "model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
313
+ "model.layers.26.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
314
+ "model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
315
+ "model.layers.26.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
316
+ "model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
317
+ "model.layers.26.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
318
+ "model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
319
+ "model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
320
+ "model.layers.26.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
321
+ "model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
322
+ "model.layers.26.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
323
+ "model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
324
+ "model.layers.26.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
325
+ "model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
326
+ "model.layers.26.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
327
+ "model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
328
+ "model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
329
+ "model.layers.27.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
330
+ "model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
331
+ "model.layers.27.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
332
+ "model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
333
+ "model.layers.27.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
334
+ "model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
335
+ "model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
336
+ "model.layers.27.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
337
+ "model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
338
+ "model.layers.27.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
339
+ "model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
340
+ "model.layers.27.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
341
+ "model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
342
+ "model.layers.27.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
343
+ "model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
344
+ "model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
345
+ "model.layers.28.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
346
+ "model.layers.28.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
347
+ "model.layers.28.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
348
+ "model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
349
+ "model.layers.28.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
350
+ "model.layers.28.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
351
+ "model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
352
+ "model.layers.28.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
353
+ "model.layers.28.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
354
+ "model.layers.28.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
355
+ "model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
356
+ "model.layers.28.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
357
+ "model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
358
+ "model.layers.28.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
359
+ "model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
360
+ "model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
361
+ "model.layers.29.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
362
+ "model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
363
+ "model.layers.29.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
364
+ "model.layers.29.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
365
+ "model.layers.29.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
366
+ "model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
367
+ "model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
368
+ "model.layers.29.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
369
+ "model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
370
+ "model.layers.29.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
371
+ "model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
372
+ "model.layers.29.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
373
+ "model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
374
+ "model.layers.29.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
375
+ "model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
376
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
377
+ "model.layers.3.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
378
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
379
+ "model.layers.3.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
380
+ "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
381
+ "model.layers.3.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
382
+ "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
383
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
384
+ "model.layers.3.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
385
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
386
+ "model.layers.3.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
387
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
388
+ "model.layers.3.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
389
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
390
+ "model.layers.3.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
391
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
392
+ "model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
393
+ "model.layers.30.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
394
+ "model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
395
+ "model.layers.30.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
396
+ "model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
397
+ "model.layers.30.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
398
+ "model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
399
+ "model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
400
+ "model.layers.30.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
401
+ "model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
402
+ "model.layers.30.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
403
+ "model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
404
+ "model.layers.30.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
405
+ "model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
406
+ "model.layers.30.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
407
+ "model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
408
+ "model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
409
+ "model.layers.31.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
410
+ "model.layers.31.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
411
+ "model.layers.31.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
412
+ "model.layers.31.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
413
+ "model.layers.31.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
414
+ "model.layers.31.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
415
+ "model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
416
+ "model.layers.31.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
417
+ "model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
418
+ "model.layers.31.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
419
+ "model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
420
+ "model.layers.31.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
421
+ "model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
422
+ "model.layers.31.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
423
+ "model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
424
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
425
+ "model.layers.4.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
426
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
427
+ "model.layers.4.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
428
+ "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
429
+ "model.layers.4.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
430
+ "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
431
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
432
+ "model.layers.4.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
433
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
434
+ "model.layers.4.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
435
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
436
+ "model.layers.4.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
437
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
438
+ "model.layers.4.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
439
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
440
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
441
+ "model.layers.5.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
442
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
443
+ "model.layers.5.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
444
+ "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
445
+ "model.layers.5.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
446
+ "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
447
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
448
+ "model.layers.5.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
449
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
450
+ "model.layers.5.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
451
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
452
+ "model.layers.5.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
453
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
454
+ "model.layers.5.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
455
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
456
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
457
+ "model.layers.6.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
458
+ "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
459
+ "model.layers.6.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
460
+ "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
461
+ "model.layers.6.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
462
+ "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
463
+ "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
464
+ "model.layers.6.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
465
+ "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
466
+ "model.layers.6.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
467
+ "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
468
+ "model.layers.6.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
469
+ "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
470
+ "model.layers.6.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
471
+ "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
472
+ "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
473
+ "model.layers.7.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
474
+ "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
475
+ "model.layers.7.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
476
+ "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
477
+ "model.layers.7.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
478
+ "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
479
+ "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
480
+ "model.layers.7.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
481
+ "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
482
+ "model.layers.7.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
483
+ "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
484
+ "model.layers.7.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
485
+ "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
486
+ "model.layers.7.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
487
+ "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
488
+ "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
489
+ "model.layers.8.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
490
+ "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
491
+ "model.layers.8.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
492
+ "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
493
+ "model.layers.8.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
494
+ "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
495
+ "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
496
+ "model.layers.8.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
497
+ "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
498
+ "model.layers.8.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
499
+ "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
500
+ "model.layers.8.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
501
+ "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
502
+ "model.layers.8.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
503
+ "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
504
+ "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
505
+ "model.layers.9.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
506
+ "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
507
+ "model.layers.9.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
508
+ "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
509
+ "model.layers.9.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
510
+ "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
511
+ "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
512
+ "model.layers.9.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
513
+ "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
514
+ "model.layers.9.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
515
+ "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
516
+ "model.layers.9.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
517
+ "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
518
+ "model.layers.9.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
519
+ "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
520
+ "model.norm.weight": "model-00002-of-00002.safetensors"
521
+ }
522
+ }
modeling_mistral_yarn.py ADDED
@@ -0,0 +1,1489 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 Mistral AI and the HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
5
+ # and OPT implementations in this library. It has been modified from its
6
+ # original forms to accommodate minor architectural differences compared
7
+ # to GPT-NeoX and OPT used by the Meta AI team that trained the model.
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ """ PyTorch Mistral model."""
21
+ import inspect
22
+ import math
23
+ from typing import List, Optional, Tuple, Union
24
+
25
+ import torch
26
+ import torch.nn.functional as F
27
+ import torch.utils.checkpoint
28
+ from torch import nn
29
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
30
+
31
+ from transformers.activations import ACT2FN
32
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast, SequenceClassifierOutputWithPast
33
+ from transformers.modeling_utils import PreTrainedModel
34
+ from transformers.utils import (
35
+ add_start_docstrings,
36
+ add_start_docstrings_to_model_forward,
37
+ is_flash_attn_2_available,
38
+ logging,
39
+ replace_return_docstrings,
40
+ )
41
+ from .configuration_mistral import MistralConfig
42
+
43
+
44
+ if is_flash_attn_2_available():
45
+ from flash_attn import flash_attn_func, flash_attn_varlen_func
46
+ from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa
47
+
48
+ _flash_supports_window_size = "window_size" in list(inspect.signature(flash_attn_func).parameters)
49
+
50
+
51
+ logger = logging.get_logger(__name__)
52
+
53
+ _CONFIG_FOR_DOC = "MistralConfig"
54
+
55
+
56
+ # Copied from transformers.models.llama.modeling_llama._get_unpad_data
57
+ def _get_unpad_data(padding_mask):
58
+ seqlens_in_batch = padding_mask.sum(dim=-1, dtype=torch.int32)
59
+ indices = torch.nonzero(padding_mask.flatten(), as_tuple=False).flatten()
60
+ max_seqlen_in_batch = seqlens_in_batch.max().item()
61
+ cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
62
+ return (
63
+ indices,
64
+ cu_seqlens,
65
+ max_seqlen_in_batch,
66
+ )
67
+
68
+
69
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
70
+ def _make_causal_mask(
71
+ input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
72
+ ):
73
+ """
74
+ Make causal mask used for bi-directional self-attention.
75
+ """
76
+ bsz, tgt_len = input_ids_shape
77
+ mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)
78
+ mask_cond = torch.arange(mask.size(-1), device=device)
79
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
80
+ mask = mask.to(dtype)
81
+
82
+ if past_key_values_length > 0:
83
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
84
+ return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
85
+
86
+ def _make_sliding_window_causal_mask(
87
+ input_ids_shape: torch.Size,
88
+ dtype: torch.dtype,
89
+ device: torch.device,
90
+ past_key_values_length: int = 0,
91
+ sliding_window: int = 4096,
92
+ ):
93
+ """
94
+ Make causal mask used for sliding window attention
95
+ """
96
+ bsz, tgt_len = input_ids_shape
97
+
98
+ tensor = torch.full(
99
+ (tgt_len, tgt_len),
100
+ fill_value=1,
101
+ device=device,
102
+ )
103
+ mask = torch.tril(tensor, diagonal=0)
104
+ # make the mask banded to account for sliding window
105
+ mask = torch.triu(mask, diagonal=-sliding_window)
106
+ mask = torch.log(mask).to(dtype)
107
+
108
+ if past_key_values_length > 0:
109
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
110
+ return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
111
+
112
+
113
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
114
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
115
+ """
116
+ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
117
+ """
118
+ bsz, src_len = mask.size()
119
+ tgt_len = tgt_len if tgt_len is not None else src_len
120
+
121
+ expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
122
+
123
+ inverted_mask = 1.0 - expanded_mask
124
+
125
+ return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
126
+
127
+ # Inverse dim formula to find dim based on number of rotations
128
+ def _yarn_find_correction_dim(num_rotations, dim, base=10000, max_position_embeddings=2048):
129
+ return (dim * math.log(max_position_embeddings/(num_rotations * 2 * math.pi)))/(2 * math.log(base))
130
+
131
+ # Find dim range bounds based on rotations
132
+ def _yarn_find_correction_range(low_rot, high_rot, dim, base=10000, max_position_embeddings=2048):
133
+ low = math.floor(_yarn_find_correction_dim(
134
+ low_rot, dim, base, max_position_embeddings))
135
+ high = math.ceil(_yarn_find_correction_dim(
136
+ high_rot, dim, base, max_position_embeddings))
137
+ return max(low, 0), min(high, dim-1) # Clamp values just in case
138
+
139
+ def _yarn_linear_ramp_mask(min, max, dim):
140
+ if min == max:
141
+ max += 0.001 # Prevent singularity
142
+
143
+ linear_func = (torch.arange(dim, dtype=torch.float32) - min) / (max - min)
144
+ ramp_func = torch.clamp(linear_func, 0, 1)
145
+ return ramp_func
146
+
147
+ def _yarn_get_mscale(scale=1):
148
+ if scale <= 1:
149
+ return 1.0
150
+ return 0.07 * math.log(scale) + 1.0
151
+
152
+ # Copied from transformers.models.llama.modeling_llama.LlamaRMSNorm with Llama->Mistral
153
+ class MistralRMSNorm(nn.Module):
154
+ def __init__(self, hidden_size, eps=1e-6):
155
+ """
156
+ MistralRMSNorm is equivalent to T5LayerNorm
157
+ """
158
+ super().__init__()
159
+ self.weight = nn.Parameter(torch.ones(hidden_size))
160
+ self.variance_epsilon = eps
161
+
162
+ def forward(self, hidden_states):
163
+ input_dtype = hidden_states.dtype
164
+ hidden_states = hidden_states.to(torch.float32)
165
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
166
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
167
+ return self.weight * hidden_states.to(input_dtype)
168
+
169
+
170
+ # Copied from transformers.models.llama.modeling_llama.LlamaRotaryEmbedding with Llama->Mistral
171
+ class MistralRotaryEmbedding(nn.Module):
172
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
173
+ super().__init__()
174
+
175
+ self.dim = dim
176
+ self.max_position_embeddings = max_position_embeddings
177
+ self.base = base
178
+ inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
179
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
180
+
181
+ # Build here to make `torch.jit.trace` work.
182
+ self._set_cos_sin_cache(
183
+ seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
184
+ )
185
+
186
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
187
+ self.max_seq_len_cached = seq_len
188
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
189
+
190
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
191
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
192
+ emb = torch.cat((freqs, freqs), dim=-1)
193
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
194
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
195
+
196
+ def forward(self, x, seq_len=None):
197
+ # x: [bs, num_attention_heads, seq_len, head_size]
198
+ if seq_len > self.max_seq_len_cached:
199
+ self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
200
+
201
+ return (
202
+ self.cos_cached[:seq_len].to(dtype=x.dtype),
203
+ self.sin_cached[:seq_len].to(dtype=x.dtype),
204
+ )
205
+
206
+ class MistralLinearScalingRotaryEmbedding(MistralRotaryEmbedding):
207
+ """MistralRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev"""
208
+
209
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
210
+ self.scaling_factor = scaling_factor
211
+ super().__init__(dim, max_position_embeddings, base, device)
212
+
213
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
214
+ self.max_seq_len_cached = seq_len
215
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
216
+ t = t / self.scaling_factor
217
+
218
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
219
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
220
+ emb = torch.cat((freqs, freqs), dim=-1)
221
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
222
+ self.register_buffer("sin_cached", emb.cos().to(dtype), persistent=False)
223
+
224
+
225
+ class MistralDynamicNTKScalingRotaryEmbedding(MistralRotaryEmbedding):
226
+ """MistralRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla"""
227
+
228
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
229
+ self.scaling_factor = scaling_factor
230
+ super().__init__(dim, max_position_embeddings, base, device)
231
+
232
+ def _set_cos_sin_cache(self, seq_len, device, dtype):
233
+ self.max_seq_len_cached = seq_len
234
+
235
+ if seq_len > self.max_position_embeddings:
236
+ base = self.base * (
237
+ (self.scaling_factor * seq_len / self.max_position_embeddings) - (self.scaling_factor - 1)
238
+ ) ** (self.dim / (self.dim - 2))
239
+ inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
240
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
241
+
242
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
243
+
244
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
245
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
246
+ emb = torch.cat((freqs, freqs), dim=-1)
247
+ self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
248
+ self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
249
+
250
+
251
+ class MistralYaRNScaledRotaryEmbedding(torch.nn.Module):
252
+ """MistralRotaryEmbedding extended with YaRN. See: https://arxiv.org/abs/2309.00071"""
253
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, scale=1, original_max_position_embeddings=2048,
254
+ extrapolation_factor=1, attn_factor=1, beta_fast=128, beta_slow=2, finetuned=False, device=None):
255
+ super().__init__()
256
+
257
+ self.dim = dim
258
+ self.max_position_embeddings = max_position_embeddings
259
+ self.base = base
260
+ self.scale = scale
261
+ self.original_max_position_embeddings = original_max_position_embeddings
262
+ self.extrapolation_factor = extrapolation_factor
263
+ self.attn_factor = attn_factor
264
+ self.beta_fast = beta_fast
265
+ self.beta_slow = beta_slow
266
+
267
+ self.yarn(device)
268
+
269
+ # Build here to make `torch.jit.trace` work.
270
+ self.max_seq_len_cached = max_position_embeddings
271
+ t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
272
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
273
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
274
+ emb = torch.cat((freqs, freqs), dim=-1)
275
+ dtype = torch.get_default_dtype()
276
+
277
+ self.register_buffer("cos_cached", (emb.cos() * self.mscale).to(dtype), persistent=False)
278
+ self.register_buffer("sin_cached", (emb.sin() * self.mscale).to(dtype), persistent=False)
279
+
280
+ def forward(self, x, seq_len=None):
281
+ # x: [bs, num_attention_heads, seq_len, head_size]
282
+ # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case.
283
+ if seq_len > self.max_seq_len_cached:
284
+ self.max_seq_len_cached = seq_len
285
+
286
+ t = torch.arange(self.max_seq_len_cached, device=x.device, dtype=self.inv_freq.dtype)
287
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
288
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
289
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
290
+
291
+ self.register_buffer("cos_cached", (emb.cos() * self.mscale).to(x.dtype), persistent=False)
292
+ self.register_buffer("sin_cached", (emb.sin() * self.mscale).to(x.dtype), persistent=False)
293
+ return (
294
+ self.cos_cached[:seq_len].to(dtype=x.dtype),
295
+ self.sin_cached[:seq_len].to(dtype=x.dtype),
296
+ )
297
+
298
+ def yarn(self, device):
299
+ pos_freqs = self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim)
300
+ inv_freq_extrapolation = 1.0 / pos_freqs
301
+ inv_freq_interpolation = 1.0 / (self.scale * pos_freqs)
302
+
303
+ low, high = _yarn_find_correction_range(self.beta_fast, self.beta_slow, self.dim, self.base, self.original_max_position_embeddings)
304
+ inv_freq_mask = (1 - _yarn_linear_ramp_mask(low, high, self.dim // 2).float().to(device)) * self.extrapolation_factor # Get n-d rotational scaling corrected for extrapolation
305
+ inv_freq = inv_freq_interpolation * (1 - inv_freq_mask) + inv_freq_extrapolation * inv_freq_mask
306
+
307
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
308
+ self.mscale = float(_yarn_get_mscale(self.scale) * self.attn_factor) # Get n-d magnitude scaling corrected for interpolation
309
+
310
+
311
+ class MistralDynamicYaRNScaledRotaryEmbedding(torch.nn.Module):
312
+ """MistralRotaryEmbedding extended with Dynamic YaRN. See: https://arxiv.org/abs/2309.00071"""
313
+ def __init__(self, dim, max_position_embeddings=2048, base=10000, original_max_position_embeddings=2048,
314
+ extrapolation_factor=1, attn_factor=1, beta_fast=128, beta_slow=2, finetuned=False, device=None):
315
+ super().__init__()
316
+
317
+ self.dim = dim
318
+ self.max_position_embeddings = max_position_embeddings
319
+ self.base = base
320
+ self.original_max_position_embeddings = original_max_position_embeddings
321
+ self.extrapolation_factor = extrapolation_factor
322
+ self.attn_factor = attn_factor
323
+ self.beta_fast = beta_fast
324
+ self.beta_slow = beta_slow
325
+
326
+ if finetuned:
327
+ self.yarn(self.max_position_embeddings / self.original_max_position_embeddings, device)
328
+ else:
329
+ inv_freq = 1.0 / \
330
+ (base ** (torch.arange(0, dim, 2).float().to(device) / dim))
331
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
332
+ self.mscale = 1
333
+
334
+ # Build here to make `torch.jit.trace` work.
335
+ self.max_seq_len_cached = max_position_embeddings
336
+ t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=torch.float32)
337
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
338
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
339
+ emb = torch.cat((freqs, freqs), dim=-1)
340
+ dtype = torch.get_default_dtype()
341
+
342
+ self.register_buffer("cos_cached", (emb.cos() * self.mscale).to(dtype), persistent=False)
343
+ self.register_buffer("sin_cached", (emb.sin() * self.mscale).to(dtype), persistent=False)
344
+
345
+ def forward(self, x, seq_len=None):
346
+ # x: [bs, num_attention_heads, seq_len, head_size]
347
+ # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case.
348
+ if seq_len > self.max_seq_len_cached:
349
+ self.max_seq_len_cached = seq_len
350
+
351
+ self.yarn(seq_len / self.max_position_embeddings, x.device)
352
+
353
+ t = torch.arange(self.max_seq_len_cached, device=x.device, dtype=self.inv_freq.dtype)
354
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
355
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
356
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
357
+
358
+ self.register_buffer("cos_cached", (emb.cos() * self.mscale).to(x.dtype), persistent=False)
359
+ self.register_buffer("sin_cached", (emb.sin() * self.mscale).to(x.dtype), persistent=False)
360
+ return (
361
+ self.cos_cached[:seq_len].to(dtype=x.dtype),
362
+ self.sin_cached[:seq_len].to(dtype=x.dtype),
363
+ )
364
+
365
+ def yarn(self, scale, device):
366
+ pos_freqs = self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim)
367
+ inv_freq_extrapolation = 1.0 / pos_freqs
368
+ inv_freq_interpolation = 1.0 / (scale * pos_freqs)
369
+
370
+ low, high = _yarn_find_correction_range(self.beta_fast, self.beta_slow, self.dim, self.base, self.original_max_position_embeddings)
371
+ inv_freq_mask = (1 - _yarn_linear_ramp_mask(low, high, self.dim // 2).float().to(device)) * self.extrapolation_factor # Get n-d rotational scaling corrected for extrapolation
372
+ inv_freq = inv_freq_interpolation * (1 - inv_freq_mask) + inv_freq_extrapolation * inv_freq_mask
373
+
374
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
375
+ self.mscale = float(_yarn_get_mscale(scale) * self.attn_factor) # Get n-d magnitude scaling corrected for interpolation
376
+
377
+ # Copied from transformers.models.llama.modeling_llama.rotate_half
378
+ def rotate_half(x):
379
+ """Rotates half the hidden dims of the input."""
380
+ x1 = x[..., : x.shape[-1] // 2]
381
+ x2 = x[..., x.shape[-1] // 2 :]
382
+ return torch.cat((-x2, x1), dim=-1)
383
+
384
+
385
+ # Copied from transformers.models.llama.modeling_llama.apply_rotary_pos_emb
386
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
387
+ cos = cos[position_ids].unsqueeze(1) # [seq_len, dim] -> [batch_size, 1, seq_len, head_dim]
388
+ sin = sin[position_ids].unsqueeze(1)
389
+ q_embed = (q * cos) + (rotate_half(q) * sin)
390
+ k_embed = (k * cos) + (rotate_half(k) * sin)
391
+ return q_embed, k_embed
392
+
393
+
394
+ class MistralMLP(nn.Module):
395
+ def __init__(self, config):
396
+ super().__init__()
397
+ self.config = config
398
+ self.hidden_size = config.hidden_size
399
+ self.intermediate_size = config.intermediate_size
400
+ self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
401
+ self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
402
+ self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
403
+ self.act_fn = ACT2FN[config.hidden_act]
404
+
405
+ def forward(self, x):
406
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
407
+
408
+
409
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
410
+ """
411
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
412
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
413
+ """
414
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
415
+ if n_rep == 1:
416
+ return hidden_states
417
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
418
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
419
+
420
+
421
+ class MistralAttention(nn.Module):
422
+ """
423
+ Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer
424
+ and "Generating Long Sequences with Sparse Transformers".
425
+ """
426
+
427
+ def __init__(self, config: MistralConfig):
428
+ super().__init__()
429
+ self.config = config
430
+ self.hidden_size = config.hidden_size
431
+ self.num_heads = config.num_attention_heads
432
+ self.head_dim = self.hidden_size // self.num_heads
433
+ self.num_key_value_heads = config.num_key_value_heads
434
+ self.num_key_value_groups = self.num_heads // self.num_key_value_heads
435
+ self.max_position_embeddings = config.max_position_embeddings
436
+ self.rope_theta = config.rope_theta
437
+
438
+ if (self.head_dim * self.num_heads) != self.hidden_size:
439
+ raise ValueError(
440
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
441
+ f" and `num_heads`: {self.num_heads})."
442
+ )
443
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
444
+ self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
445
+ self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
446
+ self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)
447
+
448
+ self._init_rope()
449
+
450
+ def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
451
+ return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
452
+
453
+ def _init_rope(self):
454
+ if self.config.rope_scaling is None:
455
+ self.rotary_emb = MistralRotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings, base=self.rope_theta)
456
+ else:
457
+ scaling_type = self.config.rope_scaling["type"]
458
+ scaling_factor = self.config.rope_scaling["factor"]
459
+ if scaling_type == "linear":
460
+ self.rotary_emb = MistralLinearScalingRotaryEmbedding(
461
+ self.head_dim, max_position_embeddings=self.max_position_embeddings,
462
+ scaling_factor=scaling_factor, base=self.rope_theta,
463
+ )
464
+ elif scaling_type == "dynamic":
465
+ self.rotary_emb = MistralDynamicNTKScalingRotaryEmbedding(
466
+ self.head_dim, max_position_embeddings=self.max_position_embeddings, scaling_factor=scaling_factor,
467
+ base=self.rope_theta,
468
+ )
469
+ elif scaling_type == "yarn":
470
+ original_max_position_embeddings = self.config.rope_scaling["original_max_position_embeddings"]
471
+ self.rotary_emb = MistralYaRNScaledRotaryEmbedding(
472
+ self.head_dim, max_position_embeddings=self.max_position_embeddings, scale=scaling_factor,
473
+ original_max_position_embeddings=original_max_position_embeddings, base=self.rope_theta,
474
+ )
475
+ elif scaling_type == "dynamic-yarn":
476
+ original_max_position_embeddings = self.config.rope_scaling["original_max_position_embeddings"]
477
+ self.rotary_emb = MistralDynamicYaRNScaledRotaryEmbedding(
478
+ self.head_dim, max_position_embeddings=self.max_position_embeddings,
479
+ original_max_position_embeddings=original_max_position_embeddings, base=self.rope_theta,
480
+ )
481
+ else:
482
+ raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
483
+
484
+ def forward(
485
+ self,
486
+ hidden_states: torch.Tensor,
487
+ attention_mask: Optional[torch.Tensor] = None,
488
+ position_ids: Optional[torch.LongTensor] = None,
489
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
490
+ output_attentions: bool = False,
491
+ use_cache: bool = False,
492
+ padding_mask: Optional[torch.Tensor] = None,
493
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
494
+ bsz, q_len, _ = hidden_states.size()
495
+
496
+ query_states = self.q_proj(hidden_states)
497
+ key_states = self.k_proj(hidden_states)
498
+ value_states = self.v_proj(hidden_states)
499
+
500
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
501
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
502
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
503
+
504
+ kv_seq_len = key_states.shape[-2]
505
+ if past_key_value is not None:
506
+ kv_seq_len += past_key_value[0].shape[-2]
507
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
508
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
509
+
510
+ if past_key_value is not None:
511
+ # reuse k, v, self_attention
512
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
513
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
514
+
515
+ past_key_value = (key_states, value_states) if use_cache else None
516
+
517
+ # repeat k/v heads if n_kv_heads < n_heads
518
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
519
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
520
+
521
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
522
+
523
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
524
+ raise ValueError(
525
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
526
+ f" {attn_weights.size()}"
527
+ )
528
+
529
+ if attention_mask is not None:
530
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
531
+ raise ValueError(
532
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
533
+ )
534
+
535
+ attn_weights = attn_weights + attention_mask
536
+
537
+ # upcast attention to fp32
538
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
539
+ attn_output = torch.matmul(attn_weights, value_states)
540
+
541
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
542
+ raise ValueError(
543
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
544
+ f" {attn_output.size()}"
545
+ )
546
+
547
+ attn_output = attn_output.transpose(1, 2).contiguous()
548
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
549
+
550
+ attn_output = self.o_proj(attn_output)
551
+
552
+ if not output_attentions:
553
+ attn_weights = None
554
+
555
+ return attn_output, attn_weights, past_key_value
556
+
557
+
558
+ class MistralFlashAttention2(MistralAttention):
559
+ """
560
+ Mistral flash attention module. This module inherits from `MistralAttention` as the weights of the module stays
561
+ untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
562
+ flash attention and deal with padding tokens in case the input contains any of them.
563
+ """
564
+
565
+ def forward(
566
+ self,
567
+ hidden_states: torch.Tensor,
568
+ attention_mask: Optional[torch.Tensor] = None,
569
+ position_ids: Optional[torch.LongTensor] = None,
570
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
571
+ output_attentions: bool = False,
572
+ use_cache: bool = False,
573
+ padding_mask: Optional[torch.LongTensor] = None,
574
+ ):
575
+ bsz, q_len, _ = hidden_states.size()
576
+
577
+ query_states = self.q_proj(hidden_states)
578
+ key_states = self.k_proj(hidden_states)
579
+ value_states = self.v_proj(hidden_states)
580
+
581
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
582
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
583
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
584
+
585
+ kv_seq_len = key_states.shape[-2]
586
+ if past_key_value is not None:
587
+ kv_seq_len += past_key_value[0].shape[-2]
588
+
589
+ # Because the input can be padded, the absolute sequence length depends on the max position id.
590
+ rotary_seq_len = max(kv_seq_len, position_ids[:, -1].max().item()) + 1
591
+ cos, sin = self.rotary_emb(value_states, seq_len=rotary_seq_len)
592
+
593
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
594
+
595
+ use_sliding_windows = (
596
+ _flash_supports_window_size
597
+ and hasattr(self.config, "sliding_window")
598
+ and kv_seq_len > self.config.sliding_window
599
+ )
600
+
601
+ if not _flash_supports_window_size:
602
+ logger.warning_once(
603
+ "The current flash attention version does not support sliding window attention, for a more memory efficient implementation"
604
+ " make sure to upgrade flash-attn library."
605
+ )
606
+
607
+ if past_key_value is not None:
608
+ # Activate slicing cache only if the config has a value `sliding_windows` attribute
609
+ if hasattr(self.config, "sliding_window") and kv_seq_len > self.config.sliding_window:
610
+ slicing_tokens = kv_seq_len - self.config.sliding_window
611
+
612
+ past_key = past_key_value[0]
613
+ past_value = past_key_value[1]
614
+
615
+ past_key = past_key[:, :, slicing_tokens:, :].contiguous()
616
+ past_value = past_value[:, :, slicing_tokens:, :].contiguous()
617
+
618
+ if past_key.shape[-2] != self.config.sliding_window - 1:
619
+ raise ValueError(
620
+ f"past key much have a shape of (`batch_size, num_heads, self.config.sliding_window-1, head_dim`), got"
621
+ f" {past_key.shape}"
622
+ )
623
+
624
+ past_key_value = (past_key, past_value)
625
+
626
+ if padding_mask is not None:
627
+ padding_mask = padding_mask[:, slicing_tokens:]
628
+ padding_mask = torch.cat([padding_mask, torch.ones_like(padding_mask[:, -1:])], dim=-1)
629
+
630
+ key_states = torch.cat([past_key_value[0], key_states], dim=2)
631
+ value_states = torch.cat([past_key_value[1], value_states], dim=2)
632
+
633
+ past_key_value = (key_states, value_states) if use_cache else None
634
+
635
+ # repeat k/v heads if n_kv_heads < n_heads
636
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
637
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
638
+
639
+ # TODO: Mistral does not have dropout in the config??
640
+ # It is recommended to use dropout with FA according to the docs
641
+ # when training.
642
+ dropout_rate = 0.0 # if not self.training else self.attn_dropout
643
+
644
+ # In PEFT, usually we cast the layer norms in float32 for training stability reasons
645
+ # therefore the input hidden states gets silently casted in float32. Hence, we need
646
+ # cast them back in float16 just to be sure everything works as expected.
647
+ input_dtype = query_states.dtype
648
+ if input_dtype == torch.float32:
649
+ logger.warning_once(
650
+ "The input hidden states seems to be silently casted in float32, this might be related to"
651
+ " the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
652
+ " float16."
653
+ )
654
+
655
+ query_states = query_states.to(torch.float16)
656
+ key_states = key_states.to(torch.float16)
657
+ value_states = value_states.to(torch.float16)
658
+
659
+ # Reashape to the expected shape for Flash Attention
660
+ query_states = query_states.transpose(1, 2)
661
+ key_states = key_states.transpose(1, 2)
662
+ value_states = value_states.transpose(1, 2)
663
+
664
+ attn_output = self._flash_attention_forward(
665
+ query_states,
666
+ key_states,
667
+ value_states,
668
+ padding_mask,
669
+ q_len,
670
+ dropout=dropout_rate,
671
+ use_sliding_windows=use_sliding_windows,
672
+ )
673
+
674
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
675
+ attn_output = self.o_proj(attn_output)
676
+
677
+ if not output_attentions:
678
+ attn_weights = None
679
+
680
+ return attn_output, attn_weights, past_key_value
681
+
682
+ def _flash_attention_forward(
683
+ self,
684
+ query_states,
685
+ key_states,
686
+ value_states,
687
+ padding_mask,
688
+ query_length,
689
+ dropout=0.0,
690
+ softmax_scale=None,
691
+ use_sliding_windows=False,
692
+ ):
693
+ """
694
+ Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
695
+ first unpad the input, then computes the attention scores and pad the final attention scores.
696
+
697
+ Args:
698
+ query_states (`torch.Tensor`):
699
+ Input query states to be passed to Flash Attention API
700
+ key_states (`torch.Tensor`):
701
+ Input key states to be passed to Flash Attention API
702
+ value_states (`torch.Tensor`):
703
+ Input value states to be passed to Flash Attention API
704
+ padding_mask (`torch.Tensor`):
705
+ The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
706
+ position of padding tokens and 1 for the position of non-padding tokens.
707
+ dropout (`int`, *optional*):
708
+ Attention dropout
709
+ softmax_scale (`float`, *optional*):
710
+ The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
711
+ use_sliding_windows (`bool`, *optional*):
712
+ Whether to activate sliding window attention.
713
+ """
714
+ # Contains at least one padding token in the sequence
715
+ if padding_mask is not None:
716
+ batch_size = query_states.shape[0]
717
+ query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._upad_input(
718
+ query_states, key_states, value_states, padding_mask, query_length
719
+ )
720
+
721
+ cu_seqlens_q, cu_seqlens_k = cu_seq_lens
722
+ max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
723
+
724
+ if not use_sliding_windows:
725
+ attn_output_unpad = flash_attn_varlen_func(
726
+ query_states,
727
+ key_states,
728
+ value_states,
729
+ cu_seqlens_q=cu_seqlens_q,
730
+ cu_seqlens_k=cu_seqlens_k,
731
+ max_seqlen_q=max_seqlen_in_batch_q,
732
+ max_seqlen_k=max_seqlen_in_batch_k,
733
+ dropout_p=dropout,
734
+ softmax_scale=softmax_scale,
735
+ causal=True,
736
+ )
737
+ else:
738
+ attn_output_unpad = flash_attn_varlen_func(
739
+ query_states,
740
+ key_states,
741
+ value_states,
742
+ cu_seqlens_q=cu_seqlens_q,
743
+ cu_seqlens_k=cu_seqlens_k,
744
+ max_seqlen_q=max_seqlen_in_batch_q,
745
+ max_seqlen_k=max_seqlen_in_batch_k,
746
+ dropout_p=dropout,
747
+ softmax_scale=softmax_scale,
748
+ causal=True,
749
+ window_size=(self.config.sliding_window, self.config.sliding_window),
750
+ )
751
+
752
+ attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)
753
+ else:
754
+ if not use_sliding_windows:
755
+ attn_output = flash_attn_func(
756
+ query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=True
757
+ )
758
+ else:
759
+ attn_output = flash_attn_func(
760
+ query_states,
761
+ key_states,
762
+ value_states,
763
+ dropout,
764
+ softmax_scale=softmax_scale,
765
+ causal=True,
766
+ window_size=(self.config.sliding_window, self.config.sliding_window),
767
+ )
768
+
769
+ return attn_output
770
+
771
+ def _upad_input(self, query_layer, key_layer, value_layer, padding_mask, query_length):
772
+ batch_size, kv_seq_len, num_heads, head_dim = key_layer.shape
773
+
774
+ # On the first iteration we need to properly re-create the padding mask
775
+ # by slicing it on the proper place
776
+ if kv_seq_len != padding_mask.shape[-1]:
777
+ padding_mask_num_tokens = padding_mask.shape[-1]
778
+ padding_mask = padding_mask[:, padding_mask_num_tokens - kv_seq_len :]
779
+
780
+ indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(padding_mask)
781
+
782
+ key_layer = index_first_axis(key_layer.reshape(batch_size * kv_seq_len, num_heads, head_dim), indices_k)
783
+ value_layer = index_first_axis(value_layer.reshape(batch_size * kv_seq_len, num_heads, head_dim), indices_k)
784
+
785
+ if query_length == kv_seq_len:
786
+ query_layer = index_first_axis(
787
+ query_layer.reshape(batch_size * kv_seq_len, num_heads, head_dim), indices_k
788
+ )
789
+ cu_seqlens_q = cu_seqlens_k
790
+ max_seqlen_in_batch_q = max_seqlen_in_batch_k
791
+ indices_q = indices_k
792
+ elif query_length == 1:
793
+ max_seqlen_in_batch_q = 1
794
+ cu_seqlens_q = torch.arange(
795
+ batch_size + 1, dtype=torch.int32, device=query_layer.device
796
+ ) # There is a memcpy here, that is very bad.
797
+ indices_q = cu_seqlens_q[:-1]
798
+ query_layer = query_layer.squeeze(1)
799
+ else:
800
+ # The -q_len: slice assumes left padding.
801
+ padding_mask = padding_mask[:, -query_length:]
802
+ query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, padding_mask)
803
+
804
+ return (
805
+ query_layer,
806
+ key_layer,
807
+ value_layer,
808
+ indices_q,
809
+ (cu_seqlens_q, cu_seqlens_k),
810
+ (max_seqlen_in_batch_q, max_seqlen_in_batch_k),
811
+ )
812
+
813
+
814
+ class MistralDecoderLayer(nn.Module):
815
+ def __init__(self, config: MistralConfig):
816
+ super().__init__()
817
+ self.hidden_size = config.hidden_size
818
+ self.self_attn = (
819
+ MistralAttention(config=config)
820
+ if not getattr(config, "_flash_attn_2_enabled", False)
821
+ else MistralFlashAttention2(config)
822
+ )
823
+ self.mlp = MistralMLP(config)
824
+ self.input_layernorm = MistralRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
825
+ self.post_attention_layernorm = MistralRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
826
+
827
+ def forward(
828
+ self,
829
+ hidden_states: torch.Tensor,
830
+ attention_mask: Optional[torch.Tensor] = None,
831
+ position_ids: Optional[torch.LongTensor] = None,
832
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
833
+ output_attentions: Optional[bool] = False,
834
+ use_cache: Optional[bool] = False,
835
+ padding_mask: Optional[torch.Tensor] = None,
836
+ ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
837
+ """
838
+ Args:
839
+ hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
840
+ attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
841
+ `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
842
+ output_attentions (`bool`, *optional*):
843
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under
844
+ returned tensors for more detail.
845
+ use_cache (`bool`, *optional*):
846
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
847
+ (see `past_key_values`).
848
+ past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
849
+ """
850
+
851
+ residual = hidden_states
852
+
853
+ hidden_states = self.input_layernorm(hidden_states)
854
+
855
+ # Self Attention
856
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
857
+ hidden_states=hidden_states,
858
+ attention_mask=attention_mask,
859
+ position_ids=position_ids,
860
+ past_key_value=past_key_value,
861
+ output_attentions=output_attentions,
862
+ use_cache=use_cache,
863
+ padding_mask=padding_mask,
864
+ )
865
+ hidden_states = residual + hidden_states
866
+
867
+ # Fully Connected
868
+ residual = hidden_states
869
+ hidden_states = self.post_attention_layernorm(hidden_states)
870
+ hidden_states = self.mlp(hidden_states)
871
+ hidden_states = residual + hidden_states
872
+
873
+ outputs = (hidden_states,)
874
+
875
+ if output_attentions:
876
+ outputs += (self_attn_weights,)
877
+
878
+ if use_cache:
879
+ outputs += (present_key_value,)
880
+
881
+ return outputs
882
+
883
+
884
+ MISTRAL_START_DOCSTRING = r"""
885
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
886
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
887
+ etc.)
888
+
889
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
890
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
891
+ and behavior.
892
+
893
+ Parameters:
894
+ config ([`MistralConfig`]):
895
+ Model configuration class with all the parameters of the model. Initializing with a config file does not
896
+ load the weights associated with the model, only the configuration. Check out the
897
+ [`~PreTrainedModel.from_pretrained`] method to load the model weights.
898
+ """
899
+
900
+
901
+ @add_start_docstrings(
902
+ "The bare Mistral Model outputting raw hidden-states without any specific head on top.",
903
+ MISTRAL_START_DOCSTRING,
904
+ )
905
+ class MistralPreTrainedModel(PreTrainedModel):
906
+ config_class = MistralConfig
907
+ base_model_prefix = "model"
908
+ supports_gradient_checkpointing = True
909
+ _no_split_modules = ["MistralDecoderLayer"]
910
+ _skip_keys_device_placement = "past_key_values"
911
+ _supports_flash_attn_2 = True
912
+
913
+ def _init_weights(self, module):
914
+ std = self.config.initializer_range
915
+ if isinstance(module, nn.Linear):
916
+ module.weight.data.normal_(mean=0.0, std=std)
917
+ if module.bias is not None:
918
+ module.bias.data.zero_()
919
+ elif isinstance(module, nn.Embedding):
920
+ module.weight.data.normal_(mean=0.0, std=std)
921
+ if module.padding_idx is not None:
922
+ module.weight.data[module.padding_idx].zero_()
923
+
924
+ def _set_gradient_checkpointing(self, module, value=False):
925
+ if isinstance(module, MistralModel):
926
+ module.gradient_checkpointing = value
927
+
928
+
929
+ MISTRAL_INPUTS_DOCSTRING = r"""
930
+ Args:
931
+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
932
+ Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
933
+ it.
934
+
935
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
936
+ [`PreTrainedTokenizer.__call__`] for details.
937
+
938
+ [What are input IDs?](../glossary#input-ids)
939
+ attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
940
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
941
+
942
+ - 1 for tokens that are **not masked**,
943
+ - 0 for tokens that are **masked**.
944
+
945
+ [What are attention masks?](../glossary#attention-mask)
946
+
947
+ Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
948
+ [`PreTrainedTokenizer.__call__`] for details.
949
+
950
+ If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see
951
+ `past_key_values`).
952
+
953
+ If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
954
+ and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
955
+ information on the default strategy.
956
+
957
+ - 1 indicates the head is **not masked**,
958
+ - 0 indicates the head is **masked**.
959
+ position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
960
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
961
+ config.n_positions - 1]`.
962
+
963
+ [What are position IDs?](../glossary#position-ids)
964
+ past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
965
+ Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape
966
+ `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape
967
+ `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.
968
+
969
+ Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
970
+ blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
971
+
972
+ If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
973
+ don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
974
+ `decoder_input_ids` of shape `(batch_size, sequence_length)`.
975
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
976
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
977
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
978
+ model's internal embedding lookup matrix.
979
+ use_cache (`bool`, *optional*):
980
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
981
+ `past_key_values`).
982
+ output_attentions (`bool`, *optional*):
983
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
984
+ tensors for more detail.
985
+ output_hidden_states (`bool`, *optional*):
986
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
987
+ more detail.
988
+ return_dict (`bool`, *optional*):
989
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
990
+ """
991
+
992
+
993
+ @add_start_docstrings(
994
+ "The bare Mistral Model outputting raw hidden-states without any specific head on top.",
995
+ MISTRAL_START_DOCSTRING,
996
+ )
997
+ class MistralModel(MistralPreTrainedModel):
998
+ """
999
+ Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`]
1000
+
1001
+ Args:
1002
+ config: MistralConfig
1003
+ """
1004
+
1005
+ def __init__(self, config: MistralConfig):
1006
+ super().__init__(config)
1007
+ self.padding_idx = config.pad_token_id
1008
+ self.vocab_size = config.vocab_size
1009
+
1010
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
1011
+ self.layers = nn.ModuleList([MistralDecoderLayer(config) for _ in range(config.num_hidden_layers)])
1012
+ self.norm = MistralRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
1013
+
1014
+ self.gradient_checkpointing = False
1015
+ # Initialize weights and apply final processing
1016
+ self.post_init()
1017
+
1018
+ def get_input_embeddings(self):
1019
+ return self.embed_tokens
1020
+
1021
+ def set_input_embeddings(self, value):
1022
+ self.embed_tokens = value
1023
+
1024
+ def _prepare_decoder_attention_mask(
1025
+ self, attention_mask, input_shape, inputs_embeds, past_key_values_length, sliding_window
1026
+ ):
1027
+ # create causal mask
1028
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
1029
+ combined_attention_mask = None
1030
+ if input_shape[-1] > 1:
1031
+ if sliding_window is not None:
1032
+ combined_attention_mask = _make_sliding_window_causal_mask(
1033
+ input_shape,
1034
+ inputs_embeds.dtype,
1035
+ device=inputs_embeds.device,
1036
+ past_key_values_length=past_key_values_length,
1037
+ sliding_window=sliding_window,
1038
+ )
1039
+ else:
1040
+ combined_attention_mask = _make_causal_mask(
1041
+ input_shape,
1042
+ inputs_embeds.dtype,
1043
+ device=inputs_embeds.device,
1044
+ past_key_values_length=past_key_values_length,
1045
+ )
1046
+
1047
+ if attention_mask is not None:
1048
+ # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
1049
+ expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
1050
+ inputs_embeds.device
1051
+ )
1052
+ combined_attention_mask = (
1053
+ expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
1054
+ )
1055
+
1056
+ return combined_attention_mask
1057
+
1058
+ @add_start_docstrings_to_model_forward(MISTRAL_INPUTS_DOCSTRING)
1059
+ def forward(
1060
+ self,
1061
+ input_ids: torch.LongTensor = None,
1062
+ attention_mask: Optional[torch.Tensor] = None,
1063
+ position_ids: Optional[torch.LongTensor] = None,
1064
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1065
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1066
+ use_cache: Optional[bool] = None,
1067
+ output_attentions: Optional[bool] = None,
1068
+ output_hidden_states: Optional[bool] = None,
1069
+ return_dict: Optional[bool] = None,
1070
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
1071
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1072
+ output_hidden_states = (
1073
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1074
+ )
1075
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
1076
+
1077
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1078
+
1079
+ # retrieve input_ids and inputs_embeds
1080
+ if input_ids is not None and inputs_embeds is not None:
1081
+ raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
1082
+ elif input_ids is not None:
1083
+ batch_size, seq_length = input_ids.shape
1084
+ elif inputs_embeds is not None:
1085
+ batch_size, seq_length, _ = inputs_embeds.shape
1086
+ else:
1087
+ raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
1088
+
1089
+ seq_length_with_past = seq_length
1090
+ past_key_values_length = 0
1091
+
1092
+ if past_key_values is not None:
1093
+ past_key_values_length = past_key_values[0][0].shape[2]
1094
+ seq_length_with_past = seq_length_with_past + past_key_values_length
1095
+
1096
+ if position_ids is None:
1097
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
1098
+ position_ids = torch.arange(
1099
+ past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
1100
+ )
1101
+ position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
1102
+ else:
1103
+ position_ids = position_ids.view(-1, seq_length).long()
1104
+
1105
+ if inputs_embeds is None:
1106
+ inputs_embeds = self.embed_tokens(input_ids)
1107
+
1108
+ padding_mask = None
1109
+
1110
+ # embed positions
1111
+ if attention_mask is None:
1112
+ attention_mask = torch.ones(
1113
+ (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device
1114
+ )
1115
+ elif 0 in attention_mask:
1116
+ padding_mask = attention_mask
1117
+
1118
+ if (
1119
+ padding_mask is not None
1120
+ and hasattr(self.config, "_flash_attn_2_enabled")
1121
+ and self.config._flash_attn_2_enabled
1122
+ ):
1123
+ is_padding_right = padding_mask[:, -1].sum().item() != batch_size
1124
+ if is_padding_right:
1125
+ raise ValueError(
1126
+ "You are attempting to perform batched generation with padding_side='right'"
1127
+ " this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to "
1128
+ " call `tokenizer.padding_side = 'left'` before tokenizing the input. "
1129
+ )
1130
+
1131
+ attention_mask = self._prepare_decoder_attention_mask(
1132
+ attention_mask,
1133
+ (batch_size, seq_length),
1134
+ inputs_embeds,
1135
+ past_key_values_length,
1136
+ sliding_window=self.config.sliding_window if hasattr(self.config, "sliding_window") else None,
1137
+ )
1138
+
1139
+ hidden_states = inputs_embeds
1140
+
1141
+ if self.gradient_checkpointing and self.training:
1142
+ if use_cache:
1143
+ logger.warning_once(
1144
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
1145
+ )
1146
+ use_cache = False
1147
+
1148
+ # decoder layers
1149
+ all_hidden_states = () if output_hidden_states else None
1150
+ all_self_attns = () if output_attentions else None
1151
+ next_decoder_cache = () if use_cache else None
1152
+
1153
+ for idx, decoder_layer in enumerate(self.layers):
1154
+ if output_hidden_states:
1155
+ all_hidden_states += (hidden_states,)
1156
+
1157
+ past_key_value = past_key_values[idx] if past_key_values is not None else None
1158
+
1159
+ if self.gradient_checkpointing and self.training:
1160
+
1161
+ def create_custom_forward(module):
1162
+ def custom_forward(*inputs):
1163
+ # None for past_key_value
1164
+ return module(*inputs, past_key_value, output_attentions, padding_mask=padding_mask)
1165
+
1166
+ return custom_forward
1167
+
1168
+ layer_outputs = torch.utils.checkpoint.checkpoint(
1169
+ create_custom_forward(decoder_layer),
1170
+ hidden_states,
1171
+ attention_mask,
1172
+ position_ids,
1173
+ )
1174
+ else:
1175
+ layer_outputs = decoder_layer(
1176
+ hidden_states,
1177
+ attention_mask=attention_mask,
1178
+ position_ids=position_ids,
1179
+ past_key_value=past_key_value,
1180
+ output_attentions=output_attentions,
1181
+ use_cache=use_cache,
1182
+ padding_mask=padding_mask,
1183
+ )
1184
+
1185
+ hidden_states = layer_outputs[0]
1186
+
1187
+ if use_cache:
1188
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
1189
+
1190
+ if output_attentions:
1191
+ all_self_attns += (layer_outputs[1],)
1192
+
1193
+ hidden_states = self.norm(hidden_states)
1194
+
1195
+ # add hidden states from the last decoder layer
1196
+ if output_hidden_states:
1197
+ all_hidden_states += (hidden_states,)
1198
+
1199
+ next_cache = next_decoder_cache if use_cache else None
1200
+ if not return_dict:
1201
+ return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
1202
+ return BaseModelOutputWithPast(
1203
+ last_hidden_state=hidden_states,
1204
+ past_key_values=next_cache,
1205
+ hidden_states=all_hidden_states,
1206
+ attentions=all_self_attns,
1207
+ )
1208
+
1209
+
1210
+ class MistralForCausalLM(MistralPreTrainedModel):
1211
+ _tied_weights_keys = ["lm_head.weight"]
1212
+
1213
+ def __init__(self, config):
1214
+ super().__init__(config)
1215
+ self.model = MistralModel(config)
1216
+ self.vocab_size = config.vocab_size
1217
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
1218
+
1219
+ # Initialize weights and apply final processing
1220
+ self.post_init()
1221
+
1222
+ def get_input_embeddings(self):
1223
+ return self.model.embed_tokens
1224
+
1225
+ def set_input_embeddings(self, value):
1226
+ self.model.embed_tokens = value
1227
+
1228
+ def get_output_embeddings(self):
1229
+ return self.lm_head
1230
+
1231
+ def set_output_embeddings(self, new_embeddings):
1232
+ self.lm_head = new_embeddings
1233
+
1234
+ def set_decoder(self, decoder):
1235
+ self.model = decoder
1236
+
1237
+ def get_decoder(self):
1238
+ return self.model
1239
+
1240
+ @add_start_docstrings_to_model_forward(MISTRAL_INPUTS_DOCSTRING)
1241
+ @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
1242
+ def forward(
1243
+ self,
1244
+ input_ids: torch.LongTensor = None,
1245
+ attention_mask: Optional[torch.Tensor] = None,
1246
+ position_ids: Optional[torch.LongTensor] = None,
1247
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1248
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1249
+ labels: Optional[torch.LongTensor] = None,
1250
+ use_cache: Optional[bool] = None,
1251
+ output_attentions: Optional[bool] = None,
1252
+ output_hidden_states: Optional[bool] = None,
1253
+ return_dict: Optional[bool] = None,
1254
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
1255
+ r"""
1256
+ Args:
1257
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1258
+ Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
1259
+ config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
1260
+ (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
1261
+
1262
+ Returns:
1263
+
1264
+ Example:
1265
+
1266
+ ```python
1267
+ >>> from transformers import AutoTokenizer, MistralForCausalLM
1268
+
1269
+ >>> model = MistralForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
1270
+ >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
1271
+
1272
+ >>> prompt = "Hey, are you conscious? Can you talk to me?"
1273
+ >>> inputs = tokenizer(prompt, return_tensors="pt")
1274
+
1275
+ >>> # Generate
1276
+ >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
1277
+ >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
1278
+ "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
1279
+ ```"""
1280
+
1281
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1282
+ output_hidden_states = (
1283
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1284
+ )
1285
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1286
+
1287
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
1288
+ outputs = self.model(
1289
+ input_ids=input_ids,
1290
+ attention_mask=attention_mask,
1291
+ position_ids=position_ids,
1292
+ past_key_values=past_key_values,
1293
+ inputs_embeds=inputs_embeds,
1294
+ use_cache=use_cache,
1295
+ output_attentions=output_attentions,
1296
+ output_hidden_states=output_hidden_states,
1297
+ return_dict=return_dict,
1298
+ )
1299
+
1300
+ hidden_states = outputs[0]
1301
+ logits = self.lm_head(hidden_states)
1302
+ logits = logits.float()
1303
+
1304
+ loss = None
1305
+ if labels is not None:
1306
+ # Shift so that tokens < n predict n
1307
+ shift_logits = logits[..., :-1, :].contiguous()
1308
+ shift_labels = labels[..., 1:].contiguous()
1309
+ # Flatten the tokens
1310
+ loss_fct = CrossEntropyLoss()
1311
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
1312
+ shift_labels = shift_labels.view(-1)
1313
+ # Enable model parallelism
1314
+ shift_labels = shift_labels.to(shift_logits.device)
1315
+ loss = loss_fct(shift_logits, shift_labels)
1316
+
1317
+ if not return_dict:
1318
+ output = (logits,) + outputs[1:]
1319
+ return (loss,) + output if loss is not None else output
1320
+
1321
+ return CausalLMOutputWithPast(
1322
+ loss=loss,
1323
+ logits=logits,
1324
+ past_key_values=outputs.past_key_values,
1325
+ hidden_states=outputs.hidden_states,
1326
+ attentions=outputs.attentions,
1327
+ )
1328
+
1329
+ def prepare_inputs_for_generation(
1330
+ self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
1331
+ ):
1332
+ if past_key_values:
1333
+ input_ids = input_ids[:, -1:]
1334
+
1335
+ position_ids = kwargs.get("position_ids", None)
1336
+ if attention_mask is not None and position_ids is None:
1337
+ # create position_ids on the fly for batch generation
1338
+ position_ids = attention_mask.long().cumsum(-1) - 1
1339
+ position_ids.masked_fill_(attention_mask == 0, 1)
1340
+ if past_key_values:
1341
+ position_ids = position_ids[:, -1].unsqueeze(-1)
1342
+
1343
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
1344
+ if inputs_embeds is not None and past_key_values is None:
1345
+ model_inputs = {"inputs_embeds": inputs_embeds}
1346
+ else:
1347
+ model_inputs = {"input_ids": input_ids}
1348
+
1349
+ model_inputs.update(
1350
+ {
1351
+ "position_ids": position_ids,
1352
+ "past_key_values": past_key_values,
1353
+ "use_cache": kwargs.get("use_cache"),
1354
+ "attention_mask": attention_mask,
1355
+ }
1356
+ )
1357
+ return model_inputs
1358
+
1359
+ @staticmethod
1360
+ def _reorder_cache(past_key_values, beam_idx):
1361
+ reordered_past = ()
1362
+ for layer_past in past_key_values:
1363
+ reordered_past += (
1364
+ tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
1365
+ )
1366
+ return reordered_past
1367
+
1368
+
1369
+ @add_start_docstrings(
1370
+ """
1371
+ The Mistral Model transformer with a sequence classification head on top (linear layer).
1372
+
1373
+ [`MistralForSequenceClassification`] uses the last token in order to do the classification, as other causal models
1374
+ (e.g. GPT-2) do.
1375
+
1376
+ Since it does classification on the last token, it requires to know the position of the last token. If a
1377
+ `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If
1378
+ no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
1379
+ padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in
1380
+ each row of the batch).
1381
+ """,
1382
+ MISTRAL_START_DOCSTRING,
1383
+ )
1384
+ # Copied from transformers.models.llama.modeling_llama.LlamaForSequenceClassification with Llama->Mistral, LLAMA->MISTRAL
1385
+ class MistralForSequenceClassification(MistralPreTrainedModel):
1386
+ def __init__(self, config):
1387
+ super().__init__(config)
1388
+ self.num_labels = config.num_labels
1389
+ self.model = MistralModel(config)
1390
+ self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)
1391
+
1392
+ # Initialize weights and apply final processing
1393
+ self.post_init()
1394
+
1395
+ def get_input_embeddings(self):
1396
+ return self.model.embed_tokens
1397
+
1398
+ def set_input_embeddings(self, value):
1399
+ self.model.embed_tokens = value
1400
+
1401
+ @add_start_docstrings_to_model_forward(MISTRAL_INPUTS_DOCSTRING)
1402
+ def forward(
1403
+ self,
1404
+ input_ids: torch.LongTensor = None,
1405
+ attention_mask: Optional[torch.Tensor] = None,
1406
+ position_ids: Optional[torch.LongTensor] = None,
1407
+ past_key_values: Optional[List[torch.FloatTensor]] = None,
1408
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1409
+ labels: Optional[torch.LongTensor] = None,
1410
+ use_cache: Optional[bool] = None,
1411
+ output_attentions: Optional[bool] = None,
1412
+ output_hidden_states: Optional[bool] = None,
1413
+ return_dict: Optional[bool] = None,
1414
+ ) -> Union[Tuple, SequenceClassifierOutputWithPast]:
1415
+ r"""
1416
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1417
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
1418
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
1419
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
1420
+ """
1421
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1422
+
1423
+ transformer_outputs = self.model(
1424
+ input_ids,
1425
+ attention_mask=attention_mask,
1426
+ position_ids=position_ids,
1427
+ past_key_values=past_key_values,
1428
+ inputs_embeds=inputs_embeds,
1429
+ use_cache=use_cache,
1430
+ output_attentions=output_attentions,
1431
+ output_hidden_states=output_hidden_states,
1432
+ return_dict=return_dict,
1433
+ )
1434
+ hidden_states = transformer_outputs[0]
1435
+ logits = self.score(hidden_states)
1436
+
1437
+ if input_ids is not None:
1438
+ batch_size = input_ids.shape[0]
1439
+ else:
1440
+ batch_size = inputs_embeds.shape[0]
1441
+
1442
+ if self.config.pad_token_id is None and batch_size != 1:
1443
+ raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
1444
+ if self.config.pad_token_id is None:
1445
+ sequence_lengths = -1
1446
+ else:
1447
+ if input_ids is not None:
1448
+ sequence_lengths = (torch.eq(input_ids, self.config.pad_token_id).long().argmax(-1) - 1).to(
1449
+ logits.device
1450
+ )
1451
+ else:
1452
+ sequence_lengths = -1
1453
+
1454
+ pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
1455
+
1456
+ loss = None
1457
+ if labels is not None:
1458
+ labels = labels.to(logits.device)
1459
+ if self.config.problem_type is None:
1460
+ if self.num_labels == 1:
1461
+ self.config.problem_type = "regression"
1462
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
1463
+ self.config.problem_type = "single_label_classification"
1464
+ else:
1465
+ self.config.problem_type = "multi_label_classification"
1466
+
1467
+ if self.config.problem_type == "regression":
1468
+ loss_fct = MSELoss()
1469
+ if self.num_labels == 1:
1470
+ loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
1471
+ else:
1472
+ loss = loss_fct(pooled_logits, labels)
1473
+ elif self.config.problem_type == "single_label_classification":
1474
+ loss_fct = CrossEntropyLoss()
1475
+ loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1))
1476
+ elif self.config.problem_type == "multi_label_classification":
1477
+ loss_fct = BCEWithLogitsLoss()
1478
+ loss = loss_fct(pooled_logits, labels)
1479
+ if not return_dict:
1480
+ output = (pooled_logits,) + transformer_outputs[1:]
1481
+ return ((loss,) + output) if loss is not None else output
1482
+
1483
+ return SequenceClassifierOutputWithPast(
1484
+ loss=loss,
1485
+ logits=pooled_logits,
1486
+ past_key_values=transformer_outputs.past_key_values,
1487
+ hidden_states=transformer_outputs.hidden_states,
1488
+ attentions=transformer_outputs.attentions,
1489
+ )
plots.png ADDED
smash_config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "api_key": null,
3
+ "verify_url": "http://johnrachwan.pythonanywhere.com",
4
+ "smash_config": {
5
+ "pruners": "None",
6
+ "factorizers": "None",
7
+ "quantizers": "['llm-int8']",
8
+ "compilers": "None",
9
+ "task": "text_text_generation",
10
+ "device": "cuda",
11
+ "cache_dir": "/ceph/hdd/staff/charpent/.cache/modelsnsslxiwz",
12
+ "batch_size": 1,
13
+ "model_name": "NousResearch/Yarn-Mistral-7b-128k",
14
+ "pruning_ratio": 0.0,
15
+ "n_quantization_bits": 8,
16
+ "output_deviation": 0.005,
17
+ "max_batch_size": 1,
18
+ "qtype_weight": "torch.qint8",
19
+ "qtype_activation": "torch.quint8",
20
+ "qobserver": "<class 'torch.ao.quantization.observer.MinMaxObserver'>",
21
+ "qscheme": "torch.per_tensor_symmetric",
22
+ "qconfig": "x86",
23
+ "group_size": 128,
24
+ "damp_percent": 0.1,
25
+ "save_load_fn": "bitsandbytes"
26
+ }
27
+ }