Why arent the shared weights exactly the same?

Shisho_Sama: <p>This is a follow up question from this <a href="https://discuss.pytorch.org/t/how-to-share-weights-between-two-layers/55541/5">thread</a>. Basically I shared weights between two layers. but the problem is, when I visualize  the weights, they are not exactly the same, one of them is a bit off compared to the other. (looks washed out if you will)<br>
here is the samples I’m talking about :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9d601d317a3d09c12e9de88a9f90277775a43be8.png" data-download-href="https://discuss.pytorch.org/uploads/default/9d601d317a3d09c12e9de88a9f90277775a43be8" title="trained_encoder_w"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9d601d317a3d09c12e9de88a9f90277775a43be8_2_489x500.png" alt="trained_encoder_w" data-base62-sha1="mscSWGeiGVnmCT2wLsrkBNcWK0w" width="489" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9d601d317a3d09c12e9de88a9f90277775a43be8_2_489x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9d601d317a3d09c12e9de88a9f90277775a43be8_2_733x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/9d601d317a3d09c12e9de88a9f90277775a43be8.png 2x" data-dominant-color="6C6C6C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">trained_encoder_w</span><span class="informations">752×768 323 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0fc538cbadde537580a9d29468ff041fd933d992.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0fc538cbadde537580a9d29468ff041fd933d992" title="trained_decoder_w_transposed"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fc538cbadde537580a9d29468ff041fd933d992_2_489x500.jpeg" alt="trained_decoder_w_transposed" data-base62-sha1="2fvHTjzrKNYbshwTlTdDE2tBbeq" width="489" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fc538cbadde537580a9d29468ff041fd933d992_2_489x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fc538cbadde537580a9d29468ff041fd933d992_2_733x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0fc538cbadde537580a9d29468ff041fd933d992.jpeg 2x" data-dominant-color="767676"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">trained_decoder_w_transposed</span><span class="informations">752×768 155 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, they are nearly identical, expect for the fact that the decoder’s weight seem washed out (indicates more values are either 0 or very close to 0, compared to the encoders weight.)<br>
However, knowing that both encoder and decoder share the same weight, why am I seeing this?<br>
(I trained this on a sparse autoencoder by the way. and the weights are shared like this :</p>
<pre><code class="lang-auto">weights = nn.Parameter(torch.randn_like(self.encoder[0].weight))
self.encoder[0].weight.data = weights.clone()
self.decoder[0].weight.data = self.encoder[0].weight.data.t()
</code></pre>
<p>What is the reason behind this behavior?<br>
I’d be very grateful to know</p>

Joshua_Clancy: <p>I’m a pytorch noob so take my answer with a grain of salt. But I am working on a similar problem and have just got it working. From your code segment I cannot tell if you have implemented the weight tie in the forward function or not, but that was the problem for me. I originally tied the weights in the <strong>init</strong> but this did not force the weights to update together, so during learning time, they drifted apart. Instead now I have used functional methods in the decoder section, and directly connected the weights in that forward function… so far this seems to work.</p>

Shisho_Sama: <p>Thanks alot good to know. I’ll give that a try and see how it goes:slightly_smiling_face:</p>