Create FC layer with multiple input channel and one output channel

username: Hussein_Torki
cooked: <p>Hi,<br>
I have 3 feature vector with 128 x 1 dimensions, in the last classification layer, I would like to feed each feature vector as one channel to final FC layer but as you know Pytorch does not support linear layer with multiple input channel.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/236ee236fc6f06a7c5ec4f0287590f2764e1cd3e.png" alt="image" data-base62-sha1="53shgUCsw7JO6nz9myTbqhIFGhw" width="292" height="339"></p>
<p>What I want is a linear layer with:<br>
Input: 3 channel: each channel 128x1 feature vector<br>
output: 1 channel 100 class probabilities.</p>
<p>Is it different than concatenating all 3 feature vectors and feed to linear layer?<br>
384*1 single channel feature vactor?</p>

username: ptrblck
cooked: <p>Could you describe what “channels” would represent in a linear layer?<br>
<code>nn.Linear</code> accepts inputs with an additional dimension as <code>[batch_size, *, in_features]</code> and will be applied sequentially on the additional dims, but I don’t understand what a channel would represent here.</p>

username: Hussein_Torki
cooked: <p>Each channel is the output of one specific feature extractor. For example assume we have 3 different encoders as feature extractors (F1,F2,F3). Each encoder have different loss function or depth or architecture.<br>
In the final classification layer we want to use all information extracted in each encoders. One simple method is feeding the output of all of those pre-trained encoders as channels to the final classification linear layer. In the linear evaluation protocol we cannot add more than single linear layer so I want to add just one linear layer with multiple input channel.<br>
1-I can concatenate all encoders output and fed to one channel linear layer but I 'm not sure what will happened in back-propagation? Could it distinguish between each encoder feature space? Could it learn properly in comparison to multi-channel input?<br>
2-After that I can use channel attention to learn suitable view and use the output of attention as the input of linear layer.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Hussein_Torki" data-post="3" data-topic="162309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hussein_torki/48/31821_2.png" class="avatar"> Hussein_Torki:</div>
<blockquote>
<p>One simple method is feeding the output of all of those pre-trained encoders as channels to the final classification linear layer. In the linear evaluation protocol we cannot add more than single linear layer so I want to add just one linear layer with multiple input channel.</p>
</blockquote>
</aside>
<p>The issue is that a linear layer doesn’t have any notion of “channels” and expects “features” while your explanation points to feed the input “as channels” to the linear layer.</p>
<p>If you want to sequentially apply the linear layer to each of these “channels” (given as an additional dimension in your output) you could directly pass them to the layer as:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 3, 4)
lin = nn.Linear(in_features=4, out_features=7)

out = lin(x)
print(out.shape)
# torch.Size([2, 3, 7])
</code></pre>
<p>Here the linear layer will be applied on each index in <code>dim1</code> separately as if you would iterate this dimension.</p>
<aside class="quote no-group" data-username="Hussein_Torki" data-post="3" data-topic="162309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hussein_torki/48/31821_2.png" class="avatar"> Hussein_Torki:</div>
<blockquote>
<p>1-I can concatenate all encoders output and fed to one channel linear layer but I 'm not sure what will happened in back-propagation? Could it distinguish between each encoder feature space? Could it learn properly in comparison to multi-channel input?</p>
</blockquote>
</aside>
<p>Again, the same issue that linear layers don’t work on channels but pure features. Concatenating tensors will not break the computation graph and you will be able to calculate the gradients. I don’t know <em>how</em> you want to apply the linear layer on the inputs so also cannot comment on its ability to learn properly.</p>

username: Hussein_Torki
cooked: <p>Thanks, I saw similar answer to previous question <a href="https://discuss.pytorch.org/t/nn-linear-with-multi-dimensional-channel-datas/95959/4">here</a>. I present my implementation for concatenating the encoders output and feed them as the input of linear layer.</p>
<pre><code class="lang-auto">class Res50_test(nn.Module):
  def __init__(self, st):
    super().__init__()
    self.encoders=nn.ModuleList()
    
    for i in range(3):
      self.encoders.append(torchvision.models.resnet50(zero_init_residual=True))
      self.encoders[i].load_state_dict(torch.load(st.trained_models[i]))
    for child in self.encoders.children():
                    for param in child.parameters():
                        param.requires_grad = False
    
    sizes = [2048*3,100]
    self.classifier = nn.Linear(sizes[-2], sizes[-1], bias=False)

    def forward(self, x):
            emb=[]
            for i,encoder in enumerate(self.encoders):
                emb.append(encoder(x))
                emb=torch.cat(emb,dim=1)
            logit=self.classifier(emb)
            return logit,emb
</code></pre>
<p>I wanted to differentiate between the output effect of each encoder in classifier.</p>

username: ptrblck
cooked: <p>Thanks for the link. The author of this post shared this:</p>
<pre><code class="lang-python">Here is the pseudo code:

input_matrix[ X x Y x Z x Channels]

linear_layer = nn.linear( in_channel, out_channel)

output_matrix = linear_layer(input_matrix)
</code></pre>
<p>which fits my code snippet and will apply the linear layer on the additional dimensions.</p>
<p>Your approach also seems to use the same approach by creating the additional <code>dim1</code> in <code>emb</code>.</p>

username: Hussein_Torki
cooked: <p>Yes, You are right, your answer for that(link) question is correct. But if I use additional dim1 in emb and feed it to the linear layer, the output shape will be (3,100). It means that the classifier considers each feature space (encoder) separately and creates 3 logits for each class. I want the to consider them together but differentiate between each encoders output.<br>
I think, as you said before, there are not any notion of “channels” for linear layers so my only remaining solution is concatenating them. Do you have any idea to differentiate between each encoders output as the input of linear layer?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Hussein_Torki" data-post="7" data-topic="162309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hussein_torki/48/31821_2.png" class="avatar"> Hussein_Torki:</div>
<blockquote>
<p>But if I use additional dim1 in emb and feed it to the linear layer, the output shape will be (3,100). It means that the classifier considers each feature space (encoder) separately and creates 3 logits for each class. I want the to consider them together but differentiate between each encoders output.</p>
</blockquote>
</aside>
<p>In that case I might have misunderstood your code snippet and it seems <code>emb</code> is still 2-dimensional.<br>
Try to follow the approach form my code snippet and <code>stack</code> the features in a new dimension at <code>dim1</code>.</p>