Finetuning MPT 7B

Mohanad_Abukmeil: <p><strong>Hello,</strong></p>
<p><strong>I finetuned MPT 7B model based using a large dataset for text generation. As a result of fine-tuning, I received two Pytprch models as follows:</strong></p>
<p>model1 = pytorch_model-00001-of-00002.bin<br>
model2 = ptorh_model-00002-of-00002.bin</p>
<p><strong>I used the following code to load the models:</strong></p>
<p>combined_state_dict = {**model1, **model2}</p>
<h1><a name="load-config-1" class="anchor" href="#load-config-1"></a>Load config</h1>
<p>config = AutoConfig.from_pretrained(‘path’, trust_remote_code=True,torch_dtype=torch.bfloat16)</p>
<h1><a name="initialize-model-2" class="anchor" href="#initialize-model-2"></a>Initialize model</h1>
<p>model = AutoModelForCausalLM.from_config(config,trust_remote_code=True).to(device)</p>
<h1><a name="load-combined-state-dict-3" class="anchor" href="#load-combined-state-dict-3"></a>Load combined state dict</h1>
<p>model.load_state_dict(combined_state_dict)</p>
<p><strong>Q1: Why I received two models instead of one?</strong></p>
<p><strong>Q2: Anyone can tell me if the above code is valid for the inference or should combine these models in a different way?</strong></p>

ptrblck: <ol>
<li>
<p>It’s unclear since you didn’t provide any information about how you are serializing the model. <code>.bin</code> could also indicate that a 3rd party package was used to store everything in raw binary.</p>
</li>
<li>
<p>I guess you are using a higher-level API from HuggingFace so their discussion board might be the better place to ask about their API.</p>
</li>
</ol>

Mohanad_Abukmeil: <p>Hi,<br>
I only fine-tuned the MPT 7B models utilizing the pre-trained weights, without making any changes to the config file. Additionally, I designated a directory path for storing the final fine-tuned PyTorch models. Upon examining the directory, I discovered that there are two sub-models.bin files, a result of the fine-tuning process. There’s also a JSON file named ‘pytorch_model.bin.index.json’, which contains the index of the layers to be used for inference. Some of these layers are found in the first model, while others are in the second model. I utilized the weight_map to load the appropriate layer from the correct model. The following code line was instrumental in resolving the issue:</p>
<p>base_path = “/path/”</p>
<h1><a name="load-the-index-information-1" class="anchor" href="#load-the-index-information-1"></a>Load the index information</h1>
<p>with open(base_path + ‘pytorch_model.bin.index.json’, ‘r’) as f:<br>
index_info = json.load(f)</p>
<h1><a name="initialize-an-empty-state-dict-to-hold-the-weights-2" class="anchor" href="#initialize-an-empty-state-dict-to-hold-the-weights-2"></a>Initialize an empty state dict to hold the weights</h1>
<p>state_dict = {}<br>
loaded_dicts = {}</p>
<p>for weight_name, file_name in index_info[‘weight_map’].items():<br>
# Load the state dict for this file (if it hasn’t been loaded already)<br>
if file_name not in loaded_dicts:<br>
loaded_dicts[file_name] = torch.load(base_path + file_name)</p>
<pre><code># Get the weights for this layer
layer_weights = loaded_dicts[file_name][weight_name]

# Add these weights to our combined state dict
state_dict[weight_name] = layer_weights
</code></pre>
<h1><a name="load-config-3" class="anchor" href="#load-config-3"></a>Load config</h1>
<p>config_path = base_path + ‘config.json’  # Replace with actual config path if different<br>
config = AutoConfig.from_pretrained(config_path,trust_remote_code=True)</p>
<h1><a name="initialize-model-4" class="anchor" href="#initialize-model-4"></a>Initialize model</h1>
<p>model = AutoModelForCausalLM.from_config(config,trust_remote_code=True,torch_dtype=torch.bfloat16)</p>
<h1><a name="load-combined-state-dict-into-the-model-5" class="anchor" href="#load-combined-state-dict-into-the-model-5"></a>Load combined state dict into the model</h1>
<p>model.load_state_dict(state_dict)</p>