How to create model with sharing weight?

xiaozhun07: <p>I want to create a model with sharing weights, for example: given two input A, B, the first 3 NN layers share the same weights, and the next 2 NN layers are for A, B respectively.</p>
<p>How to create such model, and perform optimally?</p>

apaszke: <p><strong>EDIT: we do support sharing Parameters between modules, but it’s recommended to decompose your model into many pieces that don’t share parameters if possible.</strong></p>
<p>We don’t support using the same Parameters in many modules. Just reuse the base for two inputs:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        self.base = ...
        self.head_A = ...
        self.head_B = ...

    def forward(self, input1, input2):
        return self.head_A(self.base(input1)), self.head_B(self.base(input2))
        
</code></pre>

vladimir: <p>in your example, what will happen to gradients of self.base? will they be calculated taking into account both input1 and input2?</p>

apaszke: <p>Yes, you can use the same module multiple times during forward.</p>

jekbradbury: <p>There are lots of cases where you can’t just reuse a <code>Module</code> but you still want to share parameters (e.g. in language modeling you sometimes want your word embeddings and output linear layers to share weight matrices). I thought reusing <code>Parameter</code>s was ok? It’s used in the PTBLM example <a href="https://github.com/pytorch/examples/blob/master/word_language_model/model.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/word_language_model/model.py</a> and it’s something people will keep doing (and expect to work) unless it throws an error.</p>

apaszke: <p>Yeah they are supported, sorry for this. But it’s still considered better practice to not do it. I’ve updated the answer.</p>

11189: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/seokinj/coPassGAN/blob/master/models.py#L27" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/seokinj/coPassGAN/blob/master/models.py#L27" target="_blank" rel="nofollow noopener">seokinj/coPassGAN/blob/master/models.py#L27</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="17" style="counter-reset: li-counter 16 ;">
<li>        nn.ReLU(True),</li>
<li>        nn.Conv1d(DIM, DIM, 5, padding=2),#nn.Linear(DIM, DIM),</li>
<li>        nn.ReLU(True),</li>
<li>        nn.Conv1d(DIM, DIM, 5, padding=2),#nn.Linear(DIM, DIM),</li>
<li>    )</li>
<li>
</li>
<li>def forward(self, input):</li>
<li>    output = self.res_block(input)</li>
<li>    return input + (0.3*output)</li>
<li>
</li>
<li class="selected">G_block_share1 = ResBlock()</li>
<li>G_block_share2 = ResBlock()</li>
<li>
</li>
<li>D_block_share4 = ResBlock()</li>
<li>D_block_share5 = ResBlock()</li>
<li>
</li>
<li>class Generator_A(nn.Module):</li>
<li>
</li>
<li>def __init__(self, charmap):</li>
<li>    super(Generator_A, self).__init__()</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In this code, he make share modules(‘G_block_share/D_block_share’) out of class, and then use these share modules in different two classes(‘Generator A&amp;B or Discriminator A&amp;B)…</p>
<p>This code is right way to share weights between two generator/discriminators?</p>

N_Hunter: <aside class="quote no-group" data-username="apaszke" data-post="7" data-topic="398" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>Yeah they are supported, sorry for this. But it’s still considered better practice to not do it. I’ve updated the answer.</p>
</blockquote>
</aside>
<p>Could you please tell us why it is better to not do it? Thanks</p>

ZHANGHeng19931123: <p>Dear Apaszke, thank you for your updates! But I am still a little confused about your answer. Like in your example, you have 3 modules (base, headA, headB),  but how could you decompose them into pieces that don’t share parameters? Looking forward to your answer, please! Thank you for your attention.</p>

Baichuan: <p>I think it is wrong. Just define the G_block_share=ResBlock() is right.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#pytorch-control-flow-weight-sharing" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#pytorch-control-flow-weight-sharing</a></p>

Baichuan: <pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        self.base1 = ...
        self.base2=...
        self.head_A = ...
        self.head_B = ...

    def forward(self, input1, input2):
        return self.head_A(self.base1(input1)), self.head_B(self.base2(input2))
        
</code></pre>

Hazel: <p>But in this case, how would base1 and base2 share the same weights? It seems like base1 + head_A and base2 + head_B are totally separate models.</p>

dariodematties: <p>Hi <a class="mention" href="/u/apaszke">@apaszke</a>, in regards to this mechanism for sharing weights, which is the standard way of masking?<br>
I mean, in a batch of inputs, not all the members of the batch have the same number of inputs,<br>
for instance in a batch of 16 sequences you may well have some sequences with 10 elements while others with 2 elements<br>
so you can fill the sorter ones with 0s in order to match the maximum sequence length, but you do not want your weights to be adjusted in base to such padding inputs.<br>
Therefore, which is the standard technique in pytorch for masking this weight sharing?</p>
<p>Thanks!</p>

Alex_Li: <p>I think sometimes you have to use weight sharing, like in the case where you want one layer to be the transpose of another. For this case, one can do this:</p>
<pre><code class="lang-auto">shared_w =  torch.rand((n_y, n_z))*.2 - .1 # initialize somehow
self.yzdecoding = nn.Linear(n_y, n_z) # Create shared layers
self.zydecoding = nn.Linear(n_z, n_y)
self.yzdecoding.weight = nn.Parameter(shared_w.T) # Share weights
self.zydecoding.weight = nn.Parameter(shared_w)
</code></pre>
<p>Note that the (n_y, n_z) Linear layer has weights of shape (n_z, n_y), which may not be intuitive at first.</p>

yogeshluthra: <p>Thanks for question. I have a similar use case. Did you find an answer?</p>