Are there any recommended methods to clone a model?

mrdrozdov: <p>I’m interested to clone a model for various reasons (makes it easy to stack untied versions of the same model for instance). Any recommended methods for doing so?</p>

apaszke: <p>How about <code>copy.deepcopy(model)</code></p>

mrdrozdov: <p>Thanks! Seems like a reasonable approach.</p>

mrdrozdov: <p>Ended up going with something like:</p>
<pre><code>model_clone = model_cls(**kwargs)
model_clone.load_state_dict(copy.deepcopy(original_model.state_dict()))
</code></pre>
<p>Works well for me!</p>

apaszke: <p>No need to deepcopy the state dict. Its contents won’t be assigned, but copied into the clone.</p>

rk.epfl: <p>I have a situation where I am copying weights back and forth between two instances of the same model. Unfortunately, copy.deepcopy is not working for me. I am having to do:</p>
<pre><code class="lang-auto">mp = list(model.parameters())
mcp = list(model_copy.parameters())
n = len(mp)
for i in range(0, n):
    mp[i].data[:] = mcp[i].data[:]
</code></pre>
<p>While this is fine, I wonder why deepcopy function is not working.</p>

damaru: <p>Did you figure it out? I am having the same problem</p>

rk.epfl: <p>I did not find a different way when I had this problem. But more recently, I used the “copy_” operation to copy weights layer by layer from one model to another:</p>
<pre><code>mydict = mymodel.state_dict()
layer_names = list(mydict)

# Now, to copy values for a particular layer using the name or index of it:

mydict[layer_names[index_of_layer]].copy_(some_data_with_matching_shape)
</code></pre>
<p>If there is a better way, I would be happy to learn.</p>

Royi: <aside class="quote no-group" data-username="apaszke" data-post="2" data-topic="483">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>deepcopy</p>
</blockquote>
</aside>
<p>What happens if I do this:</p>
<pre><code class="lang-auto">hNetModel = Model()
    for trainBatch, trainLabels in hTrainLoader:
        &lt;Train the Model by a Function&gt;
        modelEvaluationMetric = hNetModel(Validation)
        if(modelEvaluationMetric &lt; bestModelMetric):
            hBestModel = hNetModel
</code></pre>
<p>Namely I run the model trhough the optimization and if its performance are the best so far I use <code>hBestModel = hNetModel</code>.<br>
At the end I save the dictionary of <code>hBestModel</code>.<br>
Does it makes sense or is it just another reference to the same net always?</p>

albanD: <p>It’s just a reference to the same net, so it will be changed when you keep optimizing.<br>
You’ll need to use deepcopy as suggested.</p>

Royi: <p>Even if the training happens in a different function?<br>
Something like:</p>
<pre><code class="lang-auto">hNet in NetList
hNet = TrainNet(hNetModel)
modelEvaluationMetric = hNetModel(Validation)
        if(modelEvaluationMetric &lt; bestModelMetric):
            hBestModel = hNet
</code></pre>
<p>I thought at least when something gets back from a function it is a different copy of it (Yea, I’m not so experienced with Python).</p>

Brando_Miranda: <p>just to make your answer clear you mean:</p>
<pre><code>new_mdl = copy.deepcopy(model)
</code></pre>
<p>right?</p>

Brando_Miranda: <p>why is deep copy not working for you? in what way is it not working compared to what u expected?</p>

Brando_Miranda: <p>Does something inspired from:</p>
<aside class="quote" data-post="2" data-topic="5841">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/copy-weights-only-from-a-networks-parameters/5841/2">Copy weights only from a network's parameters</a> 
  </div>
  <blockquote>
    I think you can do something like 
params1 = model1.named_parameters()
params2 = model2.named_parameters()

dict_params2 = dict(params2)

for name1, param1 in params1:
    if name1 in dict_params2:
        dict_params2[name1].data.copy_(param1.data)
  </blockquote>
</aside>

<p>or</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/48560227/how-to-take-the-average-of-the-weights-of-two-networks">
  <header class="source">

      <a href="https://stackoverflow.com/questions/48560227/how-to-take-the-average-of-the-weights-of-two-networks" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3990607/patapouf-ai" target="_blank" rel="noopener nofollow ugc">
    <img alt="patapouf_ai" src="https://i.stack.imgur.com/0ZRtB.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/48560227/how-to-take-the-average-of-the-weights-of-two-networks" target="_blank" rel="noopener nofollow ugc">How to take the average of the weights of two networks?</a>
</h4>

<div class="tags">
  <strong>python, neural-network, deep-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3990607/patapouf-ai" target="_blank" rel="noopener nofollow ugc">
    patapouf_ai
  </a>
  on <a href="https://stackoverflow.com/questions/48560227/how-to-take-the-average-of-the-weights-of-two-networks" target="_blank" rel="noopener nofollow ugc">10:15AM - 01 Feb 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>not work for you?</p>

X-czh: <p>Hi, copy.deepcopy(model) works fine for me in previous PyTorch versions, but as I’m migrating to version 0.4.0, it seems to break. It seems to have something to do with torch.device. How should I do cloning properly in version 0.4.0?</p>
<p>The traceback is as follows:<br>
(I run<br>
device = torch.device(‘cuda’)<br>
generator = Generator(args.vocab_size, g_embed_dim, g_hidden_dim, device).to(device)<br>
previously, and when I replace device with string ‘cuda’, it works then)</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 304, in <br>
rollout = Rollout(generator, args.update_rate)<br>
File “/home/x-czh/SeqGAN-PyTorch/rollout.py”, line 14, in <strong>init</strong><br>
self.own_model = copy.deepcopy(model)<br>
File “/usr/lib/python3.5/copy.py”, line 182, in deepcopy<br>
y = _reconstruct(x, rv, 1, memo)<br>
File “/usr/lib/python3.5/copy.py”, line 297, in _reconstruct<br>
state = deepcopy(state, memo)<br>
File “/usr/lib/python3.5/copy.py”, line 155, in deepcopy<br>
y = copier(x, memo)<br>
File “/usr/lib/python3.5/copy.py”, line 243, in _deepcopy_dict<br>
y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
File “/usr/lib/python3.5/copy.py”, line 182, in deepcopy<br>
y = _reconstruct(x, rv, 1, memo)<br>
File “/usr/lib/python3.5/copy.py”, line 292, in _reconstruct<br>
y = callable(*args)<br>
File “/usr/lib/python3.5/copyreg.py”, line 88, in <strong>newobj</strong><br>
return cls.<strong>new</strong>(cls, *args)<br>
TypeError: Device() received an invalid combination of arguments - got (), but expected one of:</p>
<ul>
<li>(torch.device device)</li>
<li>(str type, int index)</li>
</ul>

kuzand: <p>Deepcopy is not working for me.</p>
<p>I have a function train(model) which returns the trained model, model_trained = train(model_untrained). However as result both are trained at the end, but I want the model_untrained to be unchanged. So I tried to deep-copy the model_untrained inside the function before the training loop, but It is not working – the model is not trained correctly. Any idea why is it happening?</p>

ptrblck: <p>Are you trying to train the copied or the original model?<br>
In the first case I assume the optimizer doesn’t have the references to the appropriate parameters, thus probably no model is trained.<br>
Could you check it?</p>

kuzand: <p>Yes I am training the copied model. You are right about the optimizer, I was passing the original model parameters to it. Thanks for spotting it!</p>

shivamsaboo17: <p>I was trying to copy a model where the forward function is using <span class="mention">@torch.jit.script_method</span> so that I can load it later in C++.<br>
But when I am using deepcopy it gives the error:<br>
<code>can't pickle BaseModel objects</code><br>
where BaseModel is classname of my model. The same code is working correctly without using jit decorator. This could be something trivial but I am unable to find a workaround.</p>

52153d3b7bd2d9066500: <pre><code class="lang-auto">import pickle
copyed_model = pickle.loads(pickle.dumps(model))
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>