Does nn.Linear performs Convolution operation?

username: Xanthan
cooked: <p>I’m writing a softmax layer at the end of the model for classification but I’m not sure whether the nn.Linear layer does convolution operation and if it doesn’t then the convolution operation needs to be done.</p>
<p>So suppose I have 5 classes to be classified at the end and the output features of the 2nd last layer is 64*8, so should the last layer be something like this:</p>
<pre><code class="lang-auto">if(layer=="last"):
      output_channels=5
                return nn.Sequential(
                    nn.Conv2d(64*8, output_channels, kernel_size, stride, padding, bias=False),
                    nn.Linear(output_channels,output_channels),
                    nn.Softmax(dim=1)
                    )
</code></pre>
<p>or it should be like this:</p>
<pre><code class="lang-auto">if(layer=="last"):
      output_channels=5
                return nn.Sequential(
                    nn.Linear(64*8,output_channels),
                    nn.Softmax(dim=1)
                    )

</code></pre>

username: KFrank
cooked: <p>Hi Prithviraj!</p>
<p>The short answer is that a <code>Linear</code> can perform a <code>Conv2d</code> as a special<br>
case (assuming appropriate reshaping of the various tensors).</p>
<p>First make sure you understand how <code>Conv2d</code> and <code>Linear</code> treat the<br>
dimensions of the tensors passed into them – which dimensions are<br>
“channels,” which are 2d dimensions over which to convolve, and<br>
so on.</p>
<aside class="quote no-group" data-username="Xanthan" data-post="1" data-topic="119714" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xanthan/48/37484_2.png" class="avatar"> Xanthan:</div>
<blockquote>
<p>I’m not sure whether the nn.Linear layer does convolution operation and if it doesn’t then the convolution operation needs to be done.</p>
</blockquote>
</aside>
<p>Assume that we fully flatten (except for the batch dimension)<br>
the tensor passed into <code>Linear</code> so that it has shape<br>
<code>[nBatch, nChannels * height * width]</code> (and <code>Linear</code> is<br>
<code>Linear (nChannels * height * width, output_channels)</code>).<br>
Then <code>Linear</code> doesn’t perform a convolution, per se, but it is<br>
more general than a convolution in that you could construct a<br>
<code>Linear</code> (that has lots of zeroes for weights) that performs a<br>
convolution (assuming that you appropriately reshape the<br>
output tensor).</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">if(layer=="last"):
      output_channels=5
                return nn.Sequential(
                    nn.Conv2d(64*8, output_channels, kernel_size, stride, padding, bias=False),
                    nn.Linear(output_channels,output_channels),
                    nn.Softmax(dim=1)
                    )
</code></pre>
<p>or it should be like this:</p>
<pre><code class="lang-auto">if(layer=="last"):
      output_channels=5
                return nn.Sequential(
                    nn.Linear(64*8,output_channels),
                    nn.Softmax(dim=1)
                    )
</code></pre>
</blockquote>
</aside>
<p>Note that your first version does <em>not</em> have a non-linearity in between<br>
the <code>Conv2d</code> and <code>Linear</code> layers.  They therefore combine into a single<br>
linear operation on your input tensor.  They can therefore be replaced<br>
by a single <code>Linear</code> layer (assuming that you pay proper attention to<br>
the shapes of the tensors and are willing to flatten the input tensor).<br>
To say this a little bit differently, a <code>Linear</code> is the most general,<br>
fully-connected layer, so it can perform any linear operation, including,<br>
as a special case, a convolution, or a convolution followed by a<br>
<code>Linear</code> (assuming no intervening non-linearity).</p>
<p>(As an aside, assuming that you’re performing “global” classification,<br>
that is assigning the whole image to a single class, rather than<br>
per-pixel classification, such as semantic segmentation, by the time<br>
you get to the “last” layer you have probably reduced things down<br>
global features rather than features that still maintain some spatial<br>
location.  So in your “last” layer there would no longer be any spatial<br>
dimensions over which to convolve.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Xanthan
cooked: <p>Hello Frank,</p>
<p>Firstly thank you for the detailed explanation</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>assuming that you’re performing “global” classification</p>
</blockquote>
</aside>
<p>Yes frank I’m performing global classification.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>are willing to flatten the input tensor</p>
</blockquote>
</aside>
<p>Will there be a need to do that?</p>
<p>my image input shapeat the starting is [batch_size,channels,image_size,image_size]</p>
<p>and by the time it comes to last layer, I believe the shape is [batch_size,64*8,4,4]</p>
<p>and I’m (want) converting this to shape  [batch_size,output_channels,1,1] after the convolution (then the linear and softmax stuff can come).</p>
<p>So now you know the shapes. what do u recommend i should do?</p>

username: KFrank
cooked: <p>Hi Prithviraj!</p>
<aside class="quote no-group" data-username="Xanthan" data-post="3" data-topic="119714" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xanthan/48/37484_2.png" class="avatar"> Xanthan:</div>
<blockquote>
<p>I’m performing global classification.</p>
</blockquote>
</aside>
<p>In that case, you will probably want to use <code>CrossEntropyLoss</code> as<br>
your loss criterion.  Therefore you will not want to use an explicit<br>
<code>Softmax</code> as <code>CrossEntropyLoss</code> has, in effect, <code>Softmax</code> built in.</p>
<aside class="quote no-group">
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>are willing to flatten the input tensor</p>
</blockquote>
</aside>
<p>Will there be a need to do that?</p>
</blockquote>
</aside>
<p>If you don’t want to have a non-linearity (the <code>activation</code> in my example,<br>
below) then the convolution becomes redundant.  So you might as well<br>
just have a single <code>Linear</code> layer (no <code>Conv2d</code>) for which you will need<br>
to flatten the (non-batch dimensions of the) tensor you pass into the<br>
<code>Linear</code>.  This is illustrated below.</p>
<aside class="quote no-group">
<blockquote>
<p>by the time it comes to last layer, I believe the shape is [batch_size,64*8,4,4]</p>
<p>and I’m (want) converting this to shape  [batch_size,output_channels,1,1] after the convolution (then the linear and softmax stuff can come).</p>
<p>So now you know the shapes. what do u recommend i should do?</p>
</blockquote>
</aside>
<p>It really depends on what you want to do.  What is your proposed<br>
<code>Conv2d</code> supposed to accomplish?  What is the final <code>Linear</code> supposed<br>
to accomplish?</p>
<p>In any event, you can shrink your last two dimensions, <code>[4, 4]</code> to<br>
<code>[1, 1]</code> by convolving with a kernel of size <code>4</code>.</p>
<p>Then to pass this to a subsequent <code>Linear</code> you have to get rid of the<br>
trailing <code>[1, 1]</code> dimensions (for example, by <code>squeeze()</code>ing).</p>
<p>These various points are illustrated by the following script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

n648 = 3   # smaller placeholder for 64*8
batch_size = 2
output_channels = 5
conv = torch.nn.Conv2d (n648, output_channels, 4)
activation = torch.nn.Sigmoid()   # for example
lin = torch.nn.Linear (output_channels, output_channels)

t1 = torch.randn (batch_size, n648, 4, 4)
print ('t1.shape =', t1.shape)
t2 = conv (t1)
print ('t2.shape =', t2.shape)
t3 = activation (t2)   # without non-linearity, conv becomes redundant
print ('t3.shape =', t3.shape)
t4 = t3.squeeze()
print ('t4.shape =', t4.shape)

prediction = lin (t4)
print ('prediction.shape =', prediction.shape)
target = torch.randint (output_channels, (batch_size,))
print ('target.shape =', target.shape)

# no softmax between final Linear layer and CrossEntropyLoss

loss_fn = torch.nn.CrossEntropyLoss()
loss = loss_fn (prediction, target)
print ('loss =', loss)

# if you want just a single Linear layer, you can flatten t1

t1_flat = torch.flatten (t1, start_dim = 1)
print ('t1_flat.shape =', t1_flat.shape)
lin2 = torch.nn.Linear (n648 * 4 * 4, output_channels)
prediction2 = lin2 (t1_flat)
print ('prediction2.shape =', prediction2.shape)
loss2 = loss_fn (prediction2, target)
print ('loss2 =', loss2)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
t1.shape = torch.Size([2, 3, 4, 4])
t2.shape = torch.Size([2, 5, 1, 1])
t3.shape = torch.Size([2, 5, 1, 1])
t4.shape = torch.Size([2, 5])
prediction.shape = torch.Size([2, 5])
target.shape = torch.Size([2])
loss = tensor(1.6289, grad_fn=&lt;NllLossBackward&gt;)
t1_flat.shape = torch.Size([2, 48])
prediction2.shape = torch.Size([2, 5])
loss2 = tensor(2.1311, grad_fn=&lt;NllLossBackward&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Xanthan
cooked: <p>Hello Frank,</p>
<p>Firstly sorry for the late reply. I have been pretty caught up lately.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>It really depends on what you want to do. What is your proposed<br>
<code>Conv2d</code> supposed to accomplish? What is the final <code>Linear</code> supposed<br>
to accomplish?</p>
</blockquote>
</aside>
<p>Yes, my aim was to use softmax at the last layer. So, that’s why i was thinking to use Linear as my last layer of the network (so that I can then send use the softmax) then the question arrived that whether it performs convoution or not.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In that case, you will probably want to use <code>CrossEntropyLoss</code> as<br>
your loss criterion. Therefore you will not want to use an explicit<br>
<code>Softmax</code> as <code>CrossEntropyLoss</code> has, in effect, <code>Softmax</code> built in.</p>
</blockquote>
</aside>
<p>So as you said Cross entropy loss has a built in softmax, so i think I can end my model at the last as just one convolution layer with no activation and will straight away send that output to Cross Entropy loss. (Also give your thoughts)</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="119714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>So you might as well<br>
just have a single <code>Linear</code> layer (no <code>Conv2d</code>) for which you will need<br>
to flatten the (non-batch dimensions of the) tensor you pass into the<br>
<code>Linear</code>. This is illustrated below.</p>
</blockquote>
</aside>
<p>I don’t think if there a need to flatten out and send it to linear layer based on what I said above.</p>
<p>And thanks for all the extra effort you put up.</p>