How to share weights between two layers?

Shisho_Sama: <p>Hello everyone, hope you are having a great time.<br>
I wanted to create an autoecoder. a simple one. if my memory serves me correctly, back in the day, one way to create an autoencoder was to share weights between encoder and decoder. that is, the decoder was simply using the transpose of the encoder. aside from the practicality of this and whether or not this was for the best or the worst, can you please help me do this?<br>
based on <a href="https://discuss.pytorch.org/t/how-to-share-part-of-weights-in-different-models/26696/4">this</a> discussion, I tried doing :<br>
<code>self.decoder[0].weight = self.encoder[0].weight.t()</code><br>
and this wont work. and I get :</p>
<blockquote>
<p><strong>TypeError</strong> : cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</p>
</blockquote>
<p>So I ended up doing:</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    def __init__(self, embeddingsize=40):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(28*28, embeddingsize),
                                     nn.Tanh())
        self.decoder = nn.Sequential(nn.Linear(embeddingsize, 28*28),
                                     nn.Sigmoid())
       self.decoder[0].weight = nn.Parameter(self.encoder[0].weight.t())

    def forward(self, input):
           ....
           return output
</code></pre>
<p>the network trains and I get no errors, but I’m not sure if it uses the very same weights for both of them or the initial weights are simply used as initial values and<code> nn.Parameter()</code> simply creates a brand new weights for the decoder!</p>
<p>Any helps in this regard is greatly appreciated and Thanks a lot in advance</p>

InnovArul: <p>I guess, you have already read the answer.<br>
I am linking the post just for completion!</p>
<aside class="quote" data-post="13" data-topic="2585">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-and-train-a-tied-autoencoder/2585/13">How to create and train a tied autoencoder?</a> 
  </div>
  <blockquote>
    To me, mixed approach looks better. 
For the record, I have updated the gist to verify that it works too.
  </blockquote>
</aside>


Shisho_Sama: <p>Thanks a lot, It would be much better if you copy/paste all of those codes here as well. its kind of hard to be redirected to another website.<br>
Anyway I have some questions as well.<br>
Why cant we simply do :<br>
<code>self.decoder[0].weight = self.encoder[0].weight.t()</code><br>
and instead we must do :<br>
<code>self.decoder[0].weight.data = self.encoder[0].weight.data.transpose(0,1)</code><br>
doesn’t data only copy the raw values from the source to the destination (i.e encoder to decoder)?<br>
so basically this would be two different weights that happen to have the same initial values! and in backprop they just get tuned independently. (please see the images below)<br>
my second question is, why cant we use <code>.t()</code> instead of <code>transpose(0,1)</code> aren’t they interchangeable?</p>
<p>Also I noticed, simply doing :</p>
<pre><code class="lang-auto">weights = nn.Parameter(torch.randn_like(self.encoder[0].weight))
self.encoder[0].weight.data = weights.clone()
self.decoder[0].weight.data = self.encoder[0].weight.data.transpose(0, 1)
</code></pre>
<p>results in different weight visualization. when I tried to visualize both weights, they look just different!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fd544f1653bfdc98bc32443ac1759efaf3029df4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fd544f1653bfdc98bc32443ac1759efaf3029df4" title="trained_encoder_w"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd544f1653bfdc98bc32443ac1759efaf3029df4_2_489x500.jpeg" alt="trained_encoder_w" data-base62-sha1="A93qJ52Zta9XgYPiM3jtZL1R5DC" width="489" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd544f1653bfdc98bc32443ac1759efaf3029df4_2_489x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd544f1653bfdc98bc32443ac1759efaf3029df4_2_733x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/fd544f1653bfdc98bc32443ac1759efaf3029df4.jpeg 2x" data-dominant-color="767676"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">trained_encoder_w</span><span class="informations">752×768 192 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/eee18b9ae219c8bf822f640c82bdf35bc340c7bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/eee18b9ae219c8bf822f640c82bdf35bc340c7bb" title="trained_decoder_w"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/eee18b9ae219c8bf822f640c82bdf35bc340c7bb_2_489x500.png" alt="trained_decoder_w" data-base62-sha1="y5eRJfXuhB0vNXiJfY2oJVWp5jl" width="489" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/eee18b9ae219c8bf822f640c82bdf35bc340c7bb_2_489x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/eee18b9ae219c8bf822f640c82bdf35bc340c7bb_2_733x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/eee18b9ae219c8bf822f640c82bdf35bc340c7bb.png 2x" data-dominant-color="727272"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">trained_decoder_w</span><span class="informations">752×768 289 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Update:</strong><br>
After transposing the decoders weight and visualizing it, it turns out, they are identical ( decoders weight visualization is a bit washed out though, but they are indeed look alike)</p>

Joshua_Clancy: <p>I have been working on a similar problem… after following <a class="mention" href="/u/innovarul">@InnovArul</a> code in the thread already linked, I was able to get it to work. Though I am unsure how the different methods effect the end result… they seem to all work, is it just a matter of speed? I also have another related question, what if I wish to tie weights between layers that are within different classes or modules. Would I return the weight data, and then pass in the weight data into the other classes? Would this still allow me to tie the weights?</p>
<p>Edit: Of course after staring at <a class="mention" href="/u/innovarul">@InnovArul</a> 's code for an hour… it is straight after I post this question that I figure out what he is doing regarding passing weight data through classes and so figure out my own question. I still wonder about the difference between the three methods though.</p>