Sharing weights between the units of the same layer and between layers

DeepLearner17: <p>Hello,</p>
<p>I have an MLP with a particular structure. I would like to share weights between the units of the same layer and between layers as depicted in the following figure :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24534fe0885934967980f7a98337ffa997e32816.png" data-download-href="https://discuss.pytorch.org/uploads/default/24534fe0885934967980f7a98337ffa997e32816" title="MLPl"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24534fe0885934967980f7a98337ffa997e32816.png" alt="MLPl" data-base62-sha1="5blGpJS73RnLf8F8OViesiMfIpg" width="690" height="417" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MLPl</span><span class="informations">861×521 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can l design such an MLP and do l need a custom  backward procedure or l can use the standard one in pytorch ?<br>
<a class="mention" href="/u/alband">@albanD</a><br>
Thank you for your help.</p>

Eta_C: <pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.w = nn.Parameter(torch.rand(3))  # w1, w2, w3
    def forward(self, x):
        x = torch.sum(x * self.w)
        x = torch.sum(x * self.w)
        return x
</code></pre>

DeepLearner17: <p>Hi  <a class="mention" href="/u/eta_c">@Eta_C</a>,</p>
<p>Where  is the linear layer ?</p>
<p><code> x = torch.sum(x * self.w)</code> for the second layer it makes sense. However, in the first layer each unit has 3 shared connections.</p>

albanD: <p>The way your weights are shared, w1 is multiplied with the first entry, w2 with the second and w3 with the third. THen you sum the three, and this is all the nodes after the first layer. That means that all 3 nodes in the middle here have the same value. Is that expected?</p>

DeepLearner17: <p>Hi <a class="mention" href="/u/alband">@albanD</a>. Yes , it is</p>

albanD: <p>Ok, then that should work for <code>x.size() == [batch, 3]</code> (you can remove the extra dimension in the parameters if you don’t want the batch dimension):</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.w = nn.Parameter(torch.rand(1, 3))  # w1, w2, w3
    def forward(self, x):
        one_node = torch.sum(x * self.w, dim=-1, keepdim=True)
        return one_node * self.w.sum(dim=-1, keepdim=True)
</code></pre>

DeepLearner17: <p>Thank you <a class="mention" href="/u/alband">@albanD</a> for the answer. However, each node of the MLP represents à n by n matrix not a scalar. Hope it’s clear :</p>
<p>Inputs are</p>
<p>x1[n,n] x2[n,n]  x3[n,n]</p>

albanD: <p>And w1, w2 and w3 are scalars?</p>

DeepLearner17: <p>Exactly :  x1[n,n] x2[n,n] x3[n,n] are matrices and  w1, w2, w3 are scalars</p>

albanD: <p>And do you get a batch of <code>x</code>? What would be the size of the input to your module? [batch, 3, n, n] or [3, n, n] ?</p>

DeepLearner17: <p>x.size()= [batch,n, n]</p>

albanD: <p>I’m confused, there is not 3 ? Where does x1,x2,x3 come from?</p>

DeepLearner17: <p>In the MLP figure, the three nodes refers to the three matrices x1,x2,x3  and the output of the MLP is  x_{mlp}=[n,n]</p>

albanD: <p>Could you give a small code sample that shows the input sizes and what you expect as output?<br>
Thanks !</p>

DeepLearner17: <p>Here is  :</p>
<pre><code class="lang-auto">b= 3
n=1000
input=torch.rand(b,n,n)

first_layer_node_1=Relu(w1*w2*w3*input[0])# n by n
first_layer_node_2=Relu(w1*w2*w3*input[1])# n by n
first_layer_node_3=Relu(w1*w2*w3*input[2])# n by n


output_layer=Relu(w1*first_layer_node_1+w2*first_layer_node_2+w3*first_layer_node_3)# n by n
</code></pre>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>You can actually use this code with <code>w1</code>, <code>w2</code> and <code>w3</code> being your weights, and it will perform weight sharing because you reuse them.</p>

DeepLearner17: <p>Thank you a lot <a class="mention" href="/u/alband">@albanD</a></p>