How to copy my model parameters to another model

username: MRRP
cooked: <p>Hi<br>
I have a federated learning scenario in which i want to send my cloud model parameters to different clients. i tried different ways. i did it with</p>
<blockquote>
<p>model_dict[name_of_models[i]].conv1.weight.data = main_model.conv1.weight.detach().clone())</p>
</blockquote>
<p>and it was working but as i saw here its better to not use .data in my code.<br>
so i change it to</p>
<blockquote>
<p>model_dict[name_of_models[i]].conv1.weight = nn.Parameters(model.conv1.weight.detach().clone())</p>
</blockquote>
<p>but when i do this my clients model stop updating. i think that’s because i changed the parameters i referenced in their optimizer.<br>
now i’m doing it with</p>
<blockquote>
<p>with torch.no_grad():<br>
for i in range(number_of_clients):<br>
state_dict = model_dict[name_of_models[i]].state_dict()</p>
<pre><code>        state_dict['conv1.weight'] = main_model.conv1.weight.detach().clone()
        state_dict['conv2.weight'] = main_model.conv2.weight.detach().clone()
        
        state_dict['conv1.bias'] = main_model.conv1.bias.detach().clone()
        state_dict['conv2.bias'] = main_model.conv2.bias.detach().clone()
        
        state_dict['fc1.weight'] = main_model.fc1.weight.detach().clone()
        state_dict['fc2.weight'] = main_model.fc2.weight.detach().clone()
        state_dict['fc3.weight'] = main_model.fc3.weight.detach().clone()
    
        state_dict['fc1.bias'] = main_model.fc1.bias.detach().clone()
        state_dict['fc2.bias'] = main_model.fc2.bias.detach().clone()
        state_dict['fc3.bias'] = main_model.fc3.bias.detach().clone()
        
        model_dict[name_of_models[i]] = model_dict[name_of_models[i]].load_state_dict(state_dict)
</code></pre>
</blockquote>
<p>but when i do this i get the error ‘_IncompatibleKeys’ object has no attribute ‘train’ while training my model.<br>
i will appreciate if anyone give me an advice on how to do this properly.<br>
thanks</p>

username: wayi
cooked: <p>Since you mentioned federated learning, shouldn’t the data transfer be in a distributed environment? How about the following code snippet?</p>
<pre><code class="lang-auto">    # Flattening all the parameters of the cloud model into a contiguous buffer to prepare for data transfer.
    flat_params = torch.cat([p.data.view(-1) for p in model.parameters()])

    # broadcast the tensors or call process group send/recv?
    ...
    
    # Copy the parameters to the client model layer by layer.
    offset = 0
    for p in module.parameters():
        p.data = flat_params[offset : offset + p.numel()].view_as(p)
        offset += p.numel()
</code></pre>

username: MRRP
cooked: <p>Thanks for replying<br>
First of all i thought its not recommended to use .data in our code.<br>
Second, sorry i didn’t understand what that offset is supposed to do. could you explain more?<br>
in my scenario i have 10 clients or node and each one of them have their own model with the same architecture. my code works when i update client models layer by layer with the code below</p>
<blockquote>
<p>model_dict[name_of_models[i]].conv1.weight.data = main_model.conv1.weight.detach().clone())</p>
</blockquote>
<p>as i heard that .data attribute can cause silent error in my code, i’m looking for some alternate way to do it.</p>

username: wayi
cooked: <aside class="quote no-group" data-username="MRRP" data-post="3" data-topic="126871">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/848f3c/48.png" class="avatar"> MRRP:</div>
<blockquote>
<p>Second, sorry i didn’t understand what that offset is supposed to do. could you explain more?</p>
</blockquote>
</aside>
<p>It’s used for unpacking a flattened tensor into a set of tensors layer by layer.</p>
<p><code>as i heard that .data attribute can cause silent error in my code, i’m looking for some alternate way to do it.</code></p>
<p>Hmm, not sure why it can have silent error in a distributed environment.</p>

username: mrshenli
cooked: <p>To set Tensor storage, the recommended way is to use <code>Tensor.set_()</code> ​ function: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.set_.html?highlight=set_#torch.Tensor.set_" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.set_ — PyTorch 1.9.0 documentation</a></p>
<p>Learned from <a class="mention" href="/u/alband">@albanD</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: seungjun
cooked: <p>You can refer to the following example.</p>
<p>You can change the model weights to a Tensor by <code>parameters_to_vector</code> function, communicate between ranks, revert that to weights by <code>vector_to_parameters</code> funciton.</p>
<p>I’m using a collective communication function to synchronize weight parameters on all GPUs but you can change that to point-to-point communication functions such as <code>torch.distributed.send</code> and <code>torch.distributed.recv</code>.</p>
<p>docs: <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">torch.distributed</a></p>
<pre><code class="lang-python">import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel

from torch.nn.utils import parameters_to_vector, vector_to_parameters

model = ...

# synchronize model parameters across nodes
vector = parameters_to_vector(model.parameters())

dist.broadcast(vector, 0)   # broadcast parameters to other processes
if dist.get_rank() != 0:
    vector_to_parameters(vector, model.parameters())
</code></pre>