Tie weights of conv. autoencoder

nwesemann: <p>I have implemented a convolutional autoencoder that perfectly works without weight sharing among encoder and decoder. I guess you all know how a conv. autoencoder works.<br>
When tieing weights of the decoder to the encoder, i have noticed a weird behaviour of the weights of a standard nn.Conv2d:<br>
For my case the input ist self.conv1 = nn.Conv2d(1,100,(16,5),stride=(16,5),padding=0), the auto-initialized weights for this layer are of size [100,1,16,5].</p>
<p>For the deconv I should use the the functional library with the transpose of these weights, right? This is the mathematically correct way to share weights. So what i would do looks like this<br>
F.conv_transpose2d(out, weight=self.conv1.weight.transpose(0,1), bias=None, stride=(16,5),padding=0)</p>
<p>this throws an error, if I don’t transpose the weights in the conv_transpose2d it doesn’t throw an error.<br>
So, this one works F.conv_transpose2d(out, weight=self.conv1.weight, bias=None,stride=(16,5),padding=0)</p>
<p>This seems like a weird behaviour (and maybe leads to errors in the future), especially because for fully connected (linear) layers it exactly works the way i would expect it to work.</p>
<p>Any ideas on this?</p>
<p>Thanks in advance<br>
niclas</p>

adrien: <p>I was interested in a similar question: what dimensions should be transposed when sharing weights for deconvolution ? is transposing filters redundant ?</p>
<p>I experienced with non symmetric filters:</p>
<pre><code class="lang-auto">self.encoder = nn.Sequential(nn.Conv2d(1, 64, (4, 2), 2, 1))
self.decoder = nn.Sequential(nn.ConvTranspose2d(64, 1, (2, 4), 2, 1))
</code></pre>
<p>does NOT work. The correct decoder is as follows:</p>
<pre><code class="lang-auto">self.decoder = nn.Sequential(nn.ConvTranspose2d(64, 1, (4, 2), 2, 1))
</code></pre>
<p>Therefore I think <code>weight=self.conv1.weight.transpose(0,1)</code> should do the job for transposed convolution</p>

adrien: <p>EDIT:</p>
<p>I report the same error as you with the use of <code>nn.functional.conv_transpose2d()</code>:</p>
<pre><code class="lang-auto">class AE_tied_weights(nn.Module):
    
    def __init__(self, input_dim=28*28, hidden_layers=(), output_dim=2000):

        super(AE_tied_weights, self).__init__()

        self.encoder = nn.ModuleList([nn.Conv2d(1, 64, 4, 2, 1)])
        self.bias = nn.ParameterList([nn.Parameter(torch.randn(1))])

    def forward(self, x):

        h = self.encoder[0](x)
        h = torch.sigmoid(h)
        y = nn.functional.conv_transpose2d(h, weight=self.encoder[0].weight.transpose(0, 1), bias=self.bias[0], stride=2, padding=1)

        return x, h, y
</code></pre>
<p>transposing <code>weight=self.encoder[0].weight.transpose(0, 1)</code> gives the following error:</p>
<pre><code class="lang-auto">y = nn.functional.conv_transpose2d(h, weight=self.encoder[0].weight.transpose(0, 1), bias=self.bias[0], stride=2, padding=1)
RuntimeError: Given transposed=1, weight[1, 64, 4, 4], so expected input[60, 64, 14, 14] to have 1 channels, but got 64 channels instead
</code></pre>
<p>Which I don’t find very consistent with my previous feed back…</p>

KSLTPK89: <p>I experienced the same behaviour and I don’t really know why it works like that…</p>

Rishi_Rao: <p>Did you check the transposing the stride as well ? I don’t know just a thought.</p>

Joshua_Clancy: <p>I’ve got the same error too. Has anyone figured out a way around this?</p>

sneiman: <p>Working on similar problems associated with disentangled representations, and ran across this thread. If I am not mistaken, you are effectively transposing the weights when you swap the in and out channel counts.</p>