How can we provide the output of a Linear layer to a Conv2D

username: jitesh
cooked: <p>I am building an Autoencoder where I need to encode an image into a latent representation of length 100. I am using the following architecture for my model.</p>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size=3)
        self.conv2 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3,stride=2)
        self.conv3 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=3,stride=2)
        
        self.linear = nn.Linear(in_features=128*30*30,out_features=100)

        self.conv1_transpose = nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=2,output_padding=1)
        self.conv2_transpose = nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=3,stride=2,output_padding=1)
        self.conv3_transpose = nn.ConvTranspose2d(in_channels=32,out_channels=3,kernel_size=3,stride=1)  
</code></pre>
<p>Is there any way I could give my <code>Linear</code> layer’s output to a <code>Conv2D</code> or a <code>ConvTranspose2D</code> layer so that I can reconstruct my image? The output is restored if I remove the <code>Linear</code> layer. I want to know how I can reconstruct my image keeping the <code>Linear</code> layer</p>
<p>Any help would be appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Yes that should be possible, if you are able to create a <code>view</code> of the tensor in the expected shape.<br>
<code>nn.ConvTranspose2d</code> expects an input in the shape <code>[batch_size, channels, height, width]</code>.<br>
Since your linear layer is returning 100 output features, you won’t be able to use <code>in_channels=128</code>, but would have to lower it.<br>
You could use <code>out = out.view(out.size(0), 4, 5, 5)</code> on the output of the linear layer and pass it to the transposed conv.</p>

username: jitesh
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
I tried another way, I just added another linear layer like this<br>
<code>self.linear2 = nn.Linear(in_features=100,out_features=128*30*30)</code> and in the forward function, I used<br>
<code>x = x.view(x.size(0),128,30,30))</code> and kept the rest of the code same. When I print the output after every layer in the <code>forward</code> method, I am getting the desired shape. But when I start training, I am getting the following error<br>
<code>RuntimeError: shape '[8, 115200]' is invalid for input of size 2113536</code> despite resizing all images to (128,128) in my dataloader.</p>
<p>Could you please have a look at my <a href="https://colab.research.google.com/drive/1VCJu2YI7wGbLdTwjRourgtgzMfKX-vUy" rel="nofollow noopener">code</a> (Google Colab notebook).</p>

username: jitesh
cooked: <p>Just for reference, here is the sequence of shapes when I pass my model through my network</p>
<pre><code class="lang-auto">torch.Size([1, 3, 128, 128])
torch.Size([1, 32, 126, 126])
torch.Size([1, 64, 62, 62])
torch.Size([1, 128, 30, 30])
torch.Size([1, 115200])
torch.Size([1, 100])
torch.Size([1, 115200])
torch.Size([1, 128, 30, 30])
torch.Size([1, 64, 62, 62])
torch.Size([1, 32, 126, 126])
torch.Size([1, 3, 128, 128])
</code></pre>

username: ptrblck
cooked: <p>Even though you are defining a valid <code>Resize</code> transformation, you are not applying the transformation in your <code>CustomDataset</code>, but instead but use <code>ToTensor()</code>.<br>
If your images have different spatial size, this shape mismatch error will be raised.</p>

username: jitesh
cooked: <p>Oh my god. I can’t believe I missed that! Thanks a lot. It works perfectly now</p>