How to create and train a tied autoencoder?

username: Chen_HY
cooked: <p>Some papers mentioned a tied auto encoder, in which two W matrices are identical, i.e. W_{decode} = W_{encode}.t.<br>
How can I create such a network which two layer share a matrix but use them differently?</p>

username: smth
cooked: <p>you can use the Functional interface of <code>import torch.nn.functional as F</code> and just have W and W.t() be passed to <code>out_encoder = F.linear(x, W)</code>, <code>out_decoder = F.linear(y, W.t())</code>.</p>
<p>Hope these hints help.</p>

username: Chen_HY
cooked: <p>Thank you very much. I will try that.<br>
BTW, what is the difference between Linear as a Function and Linear as a Module?</p>

username: trypag
cooked: <p>The Module version will take care of the weights by itself, whereas the Functional version will just apply the linear transform by using an external weight matrix, it is based on the concept of immutability, where you compose function with an immutable state.</p>

username: jekbradbury
cooked: <p>If you want to you can also have two modules that share a weight matrix just by setting <code>mod1.weight = mod2.weight</code>, but the functional approach is likely to be less magical and harder to make a mistake with.</p>

username: Chen_HY
cooked: <p>My final choice, A little bit slightly complicated…</p>
<pre><code>class MirrorLinear(nn.Module):
    def __init__(self, in_features, out_features, weight, bias=True):
        super(MirrorLinear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.register_parameter('weight', weight)
        if bias:
            self.bias = nn.Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_bias()

        def reset_bias(self):
            stdv = 1. / math.sqrt(self.weight.size(1))
            if self.bias is not None:
                self.bias.data.uniform_(-stdv, stdv)

        def forward(self, input):
            if self.bias is None:
                return self._backend.Linear()(input, self.weight.t())
            else:
                return self._backend.Linear()(input, self.weight.t(), self.bias)

        def __repr__(self):
            return self.__class__.__name__ + ' (' \
                + str(self.in_features) + ' -&gt; ' \
                + str(self.out_features) + ')'</code></pre>

username: Filipe_Silva
cooked: <p><a class="mention" href="/u/smth">@smth</a>,  your solution assumes we are defining a custom neural network module where W is a parameter of the network, right?</p>
<p>Is there any way to tie weights by only instantiating off-the-shelf nn modules without fiddling around with new parameters?</p>

username: InnovArul
cooked: <p>This would be more appropriate answer for your question.</p>
<aside class="quote" data-post="5" data-topic="2585">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jekbradbury/48/14_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-and-train-a-tied-autoencoder/2585/5">How to create and train a tied autoencoder?</a> 
  </div>
  <blockquote>
    If you want to you can also have two modules that share a weight matrix just by setting mod1.weight = mod2.weight, but the functional approach is likely to be less magical and harder to make a mistake with.
  </blockquote>
</aside>


username: Filipe_Silva
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, I have read that post already, yet, I would like to know if there is another way to achieve that result.</p>

username: InnovArul
cooked: <p>Hi, sorry for the delay. I am not sure if this is what you are looking for.<br>
But, Please take a look at this:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc</a></h4>
<h5>tied_linear.py</h5>
<pre><code class="Python">import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np
import torch.optim as optim

# tied autoencoder using off the shelf nn modules
class TiedAutoEncoderOffTheShelf(nn.Module):
	def __init__(self, inp, out, weight):
		super().__init__()
		self.encoder = nn.Linear(inp, out, bias=False)
		self.decoder = nn.Linear(out, inp, bias=False)</code></pre>
This file has been truncated. <a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I have tried to use two linear layers and shared their weights, as well as tried to use F.linear.</p>
<p>Hope, it helps.</p>

username: Filipe_Silva
cooked: <p>Hi <a class="mention" href="/u/innovarul">@InnovArul</a>,</p>
<p>Thank you for your answer, it was quite enlightening!</p>
<p>Yet, what are your thoughts about this approach:</p>
<pre><code class="lang-auto">class MixedApproachTiedAutoEncoder(nn.Module):
	def __init__(self, inp, out, weight):
		super().__init__()
		self.encoder = nn.Linear(inp, out, bias=False)

	def forward(self, input):
		encoded_feats = self.encoder(input)
		reconstructed_output = F.linear(encoded_feats, self.encoder.weight.t())
		return encoded_feats, reconstructed_output
</code></pre>
<p>I believe that by building the network this way will result in the parameters of the decoder not being differentiated when running backward because the functional module has no state but at the same time as we are performing an operation over a parameter I would expect the network graph to be updated dynamically.</p>
<p>What is wrong with this train of thought?</p>
<p>Cheers.</p>

username: InnovArul
cooked: <p>To me, mixed approach looks better.</p>
<p>For the record, I have updated the gist to verify that it works too.<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc</a></h4>
<h5>tied_linear.py</h5>
<pre><code class="Python">import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np
import torch.optim as optim

# tied autoencoder using off the shelf nn modules
class TiedAutoEncoderOffTheShelf(nn.Module):
    def __init__(self, inp, out, weight):
        super().__init__()
        self.encoder = nn.Linear(inp, out, bias=False)
        self.decoder = nn.Linear(out, inp, bias=False)</code></pre>
This file has been truncated. <a href="https://gist.github.com/InnovArul/500e0c57e88300651f8005f9bd0d12bc" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Filipe_Silva
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, can you explain how the weights for the decoder part are being differentiated?</p>
<p>As far as I know by defining a module we are storing the value of the learnable parameters (weights) unlike happens when we simply call the functional counterpart, thereby, in the mixed approach we have just defined one module/layer in the _<em>init</em>_ function then we passed the output of that layer to the function F, therefore, I was expecting that the weights between the hidden layer and the output layer were not automatically differentiated, right?</p>
<p>Thank you for keeping the gist updated <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: InnovArul
cooked: <p>In my understanding, the gradient still flows to the encoder’s weight because we are just passing the weight (<code>self.encoder.weight.t()</code>), as it is to the function in F. i.e, here the link to the parameter is still intact so that automatic differentiation happens (thanks to the integration of Variable into Tensor, I guess). Instead, if you decide to pass the raw data in <code>encoder.weight</code> (using  <code>self.encoder.weight.data.t()</code>), then we are detaching the differentiation path, hence automatic differentiation will not affect <code>encoder.weight</code>.</p>
<p>Maybe, <a class="mention" href="/u/tom">@tom</a>, <a class="mention" href="/u/smth">@smth</a>, <a class="mention" href="/u/ptrblck">@ptrblck</a> would be able to give more appropriate answer on technicalities of this. For now, this is my understanding. I am not sure if it is 100% correct.</p>

username: tom
cooked: <p>As Arul says, given an <code>nn.Parameter</code> or more generally any a leaf node, a (differentiable) calculation that uses multiple times will propagate (and accumulate) gradients on backward. And this is precisely what you would want for tied weights.<br>
A minor note: <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html#what-about-data" rel="nofollow noopener">Don’ t use <code>.data</code>, it is strictly not as good as <code>.detach()</code></a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Bion_Howard
cooked: <p><a class="mention" href="/u/tom">@tom</a> <a class="mention" href="/u/innovarul">@InnovArul</a></p>
<p>this is a fascinating topic because if we can make scalable tied autoencoders (“coders”) then we can get twice the number of optimization steps on 1/2 the number of parameters – potentially 4x improvement …</p>
<p>how do you make sure tied autoencoder forward step does gradients properly? add reverse function?</p>

username: Di_Wu
cooked: <p>Arul,</p>
<p>Thanks for your code example, it is truly helpful. I just have one question, should we set require_grad=False for the decoder layer after setting self.decoder.weight.data = self.encoder.weight.data.transpose(0,1)?</p>
<p>Thanks a lot.</p>

username: InnovArul
cooked: <p>you do not need to set <code>require_grad=False</code> for decoder.<br>
We expect that the tied weight receives gradients from both encoder and decoder.</p>