The best method for module cloning with parameter sharing?

supakjk: <p>It seems that <code>copy.deepcopy(module)</code> is recommended for cloning a module without parameter sharing.</p>
<p>Then, what would be the best approach for cloning with parameter sharing?</p>
<p>I mean, weights and grads will be automatically shared if we just forward the module with different inputs and dealing with the outputs. However, if there are other non-module variables that I want to share, I am not sure what would be the most elegant way.</p>
<p>Thanks!</p>

apaszke: <p>I’d say it really depends on the use case. I can’t think of any general recipe right now.</p>

f10w: <p>Any ideas on this please?<br>
Cloning a module is typically done using <code>copy.deepcopy</code>. For example, the official PyTorch <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/transformer.py#L487" rel="noopener nofollow ugc">implementation of the Transformer</a> uses the following:</p>
<pre><code class="lang-python">def _get_clones(module, N):
    return ModuleList([copy.deepcopy(module) for i in range(N)])
</code></pre>
<p>However, when <code>module</code> contains <strong>shared submodules</strong> or <strong>shared weights</strong> then it is expected to keep them shared instead of creating new objects. Unfortunately the above solution doesn’t work for this case.</p>
<p>As an example of particular use cases, consider torchvision’s official <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py" rel="noopener nofollow ugc">implementation</a> of ResNet. In this implementation, all the convolution layers share the same bias variable. Thus if we clone some convolution layers using <code>deepcopy</code> then new bias terms are created but they are no longer shared, which doesn’t make sense.</p>
<p>I hope somebody could some up with a solution. Thank you very much in advance for your help!</p>

ptrblck: <aside class="quote no-group" data-username="f10w" data-post="3" data-topic="1173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/f10w/48/1352_2.png" class="avatar"> f10w:</div>
<blockquote>
<p>In this implementation, all the convolution layers share the same bias variable. Thus if we clone some convolution layers using <code>deepcopy</code> then new bias terms are created but they are no longer shared, which doesn’t make sense.</p>
</blockquote>
</aside>
<p>Could you point me to the code where the bias is shared, please?<br>
I might have missed it but it’s the first time I hear that resnets share parameters.</p>

f10w: <p>Sorry my statement was confusing (but still technically correct I think). Actually the biases are not used in the convolution layers and thus they all point to the same object.</p>
<p>To detect shared parameters, I use the following function from fairseq:</p>
<pre><code class="lang-python">def _catalog_shared_params(module, memo=None, prefix=""):
    """Taken from https://github.com/pytorch/fairseq/blob/main/fairseq/trainer.py
    """
    if memo is None:
        first_call = True
        memo = {}
    else:
        first_call = False
    for name, param in module._parameters.items():
        param_prefix = prefix + ("." if prefix else "") + name
        if param not in memo:
            memo[param] = []
        memo[param].append(param_prefix)
    for name, m in module._modules.items():
        if m is None:
            continue
        submodule_prefix = prefix + ("." if prefix else "") + name
        _catalog_shared_params(m, memo, submodule_prefix)
    if first_call:
        return [x for x in memo.values() if len(x) &gt; 1]
</code></pre>
<p>We can obtain the list of shared parameters of, e.g., <code>resnet50</code>, as follows:</p>
<pre><code class="lang-python">model = torchvision.models.__dict__['resnet50']()
shared_params = _catalog_shared_params(model)
print(f'shared_params:\n{shared_params}')
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">shared_params:
[['conv1.bias', 'layer1.0.conv1.bias', 'layer1.0.conv2.bias', 'layer1.0.conv3.bias', 'layer1.0.downsample.0.bias', 'layer1.1.conv1.bias', 'layer1.1.conv2.bias', 'layer1.1.conv3.bias', 'layer1.2.conv1.bias', 'layer1.2.conv2.bias', 'layer1.2.conv3.bias', 'layer2.0.conv1.bias', 'layer2.0.conv2.bias', 'layer2.0.conv3.bias', 'layer2.0.downsample.0.bias', 'layer2.1.conv1.bias', 'layer2.1.conv2.bias', 'layer2.1.conv3.bias', 'layer2.2.conv1.bias', 'layer2.2.conv2.bias', 'layer2.2.conv3.bias', 'layer2.3.conv1.bias', 'layer2.3.conv2.bias', 'layer2.3.conv3.bias', 'layer3.0.conv1.bias', 'layer3.0.conv2.bias', 'layer3.0.conv3.bias', 'layer3.0.downsample.0.bias', 'layer3.1.conv1.bias', 'layer3.1.conv2.bias', 'layer3.1.conv3.bias', 'layer3.2.conv1.bias', 'layer3.2.conv2.bias', 'layer3.2.conv3.bias', 'layer3.3.conv1.bias', 'layer3.3.conv2.bias', 'layer3.3.conv3.bias', 'layer3.4.conv1.bias', 'layer3.4.conv2.bias', 'layer3.4.conv3.bias', 'layer3.5.conv1.bias', 'layer3.5.conv2.bias', 'layer3.5.conv3.bias', 'layer4.0.conv1.bias', 'layer4.0.conv2.bias', 'layer4.0.conv3.bias', 'layer4.0.downsample.0.bias', 'layer4.1.conv1.bias', 'layer4.1.conv2.bias', 'layer4.1.conv3.bias', 'layer4.2.conv1.bias', 'layer4.2.conv2.bias', 'layer4.2.conv3.bias']]
</code></pre>

f10w: <p>I’ve just checked again and it appears that after cloning the new biases are also shared, which is expected and thus the above is actually not a good example of use cases. Let me come back to this question in a future when I have another example. Thanks!</p>

Mostafa_Elhoushi: <p>Python’s <a href="https://github.com/python/cpython/blob/10de3600a908f96d1c43dac85ef867991d54708e/Lib/copy.py#L119" rel="noopener nofollow ugc">deepcopy function</a> has a <code>memo</code> argument for components that you don’t want to copy. So I used that to avoid cloning weights when cloning:</p>
<pre><code class="lang-auto">import copy

model_memo = {}
for _, W in model.named_parameters():
  model_memo[id(W)] = W

model_copy = copy.deepcopy(model, memo=model_memo)
</code></pre>