Error change output pretrained model

Alphonsito25: <p>Hi, I am trying to edit the input and output channel of Mobilenet_v3_small. I have created a class for this (avoid the encoder part). The change is based on the mobilenet source code.</p>
<pre><code>input_net = 3 (RGB image)
output_net = 2 (2 classes)
last_channel = 1024 (Default in Mobilenet_v3_small)
class mi_Net(nn.Module):
    def __init__(self):
        super(mi_Net, self).__init__()

        self.model = models.mobilenet_v3_small(pretrained=True)
        for param in self.model.parameters():
            param.requires_grad = False
        
        #Change input and output Mobilenet
        self.model.classifier[0] = nn.Linear(input_net, last_channel)
        self.model.classifier[3] = nn.Linear(last_channel, output_net)
        
        #Encoder
        self.encoder = nn.Sequential(
            nn.Linear(input_encoder, 128),
            nn.ReLU(),
            nn.Linear(128, output_encoder),
        )
        
        self.out = nn.Linear(4, 2)

    def forward(self, x, y):
    	#Mobilenet
        x = self.model(x)  	
        #Encoder
        y = self.encoder(y)
        z = torch.cat(x, y)
        z = self.out(z)
        return z
</code></pre>
<p>But when I compile I get this:<br>
mat1 and mat2 shapes cannot be multiplied (4x576 and 3x1024)</p>
<p>I changed input_net = 576 (I am not sure about this)<br>
And now the error is:<br>
mat1 and mat2 shapes cannot be multiplied (68x3 and 17x128)</p>
<p>I would like to solve the first error without change input_net, but I dont know how.</p>
<p>Thank you!!!</p>

ptrblck: <aside class="quote no-group" data-username="Alphonsito25" data-post="1" data-topic="157372">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d6d6ee/48.png" class="avatar"> Alphonsito25:</div>
<blockquote>
<p>I changed input_net = 576 (I am not sure about this)</p>
</blockquote>
</aside>
<p>This would be the right change assuming you are not changing the shapes of the intermediate activations since the original <code>classifier[0]</code> layer expects <code>576</code> input features:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.mobilenet_v3_small(pretrained=False)
print(model)
...
#   (classifier): Sequential(
#     (0): Linear(in_features=576, out_features=1024, bias=True)
#     (1): Hardswish()
#     (2): Dropout(p=0.2, inplace=True)
#     (3): Linear(in_features=1024, out_features=1000, bias=True)
#   )
# )
</code></pre>
<p>However, I don’t see the reason why you would like to replace the trained <code>nn.Linear</code> layer with a new one, so you might want to consider only replacing the last one in case you want to change the number of output features.</p>
<aside class="quote no-group" data-username="Alphonsito25" data-post="1" data-topic="157372">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d6d6ee/48.png" class="avatar"> Alphonsito25:</div>
<blockquote>
<p>And now the error is:<br>
mat1 and mat2 shapes cannot be multiplied (68x3 and 17x128)</p>
</blockquote>
</aside>
<p>I guess this error is raised in <code>self.encoder(y)</code>. I don’t know how <code>input_encoder</code> is set (I guess it’s set to <code>17</code>) and don’t know where <code>y</code> is coming from so you would need to make sure <code>self.encoder</code> accepts the <code>y</code> input.</p>

Alphonsito25: <p>input_encoder is set in 17 because the input is a tensor of 17 keypoints. I changed it to 1 because it is 1 tensor. But it returns me an error, because input_encoder must be 3. I dont understand it.</p>
<p>Once I changed to 3 (in order to works), it returns me this:</p>
<pre><code>    z = torch.cat(z)
RuntimeError: Tensors must have same number of dimensions: got 2 and 3
</code></pre>
<p>Why returns me this if I have set both outputs to 2?</p>

ptrblck: <p>I would recommend to check the shapes and number of dimensions of all inputs to the model as well as from each output activation before trying to concatenate them.<br>
The latest issue is caused by a dimension mismatch.</p>

Alphonsito25: <p>I want to transform the “y” input (tensor of 17 rows and 3 columns, dimension 3) to an output tensor of 1 row and 2 columns, dimension 2. Because currently the output is a tensor of 17 rows and 2 columns, dimension 3. What layer should I introduce to the encoder? Thank you!</p>

ptrblck: <p>Sorry, I don’t fully understand the shapes, as you are mentioning rows and columns for 3D tensors so it seems the 3rd dimension is missing. Could you post them as shapes e.g. <code>[2, 3, 4]</code> which would be a 3D tensor with the corresponding shapes in each dimension?</p>

Alphonsito25: <p>Of course, what I mean is at the input I have this (print before self.encoder(y)):<br>
(tensor of 17 rows and 3 columns, dimension 3)</p>
<pre><code>tensor([[[ 0.0253, -0.2194,  0.9556],
         [ 0.0759, -0.2682,  0.9194],
         [-0.0253, -0.2682,  0.9463],
         [ 0.1770, -0.3170,  0.9249],
         [-0.1265, -0.3170,  0.9088],
         [ 0.3288,  0.0244,  0.9021],
         [-0.3288, -0.0244,  0.8784],
         [ 0.3794,  0.4633,  0.8549],
         [-0.4300,  0.4145,  0.8708],
         [ 0.1265,  0.5608,  0.8571],
         [-0.2276,  0.4633,  0.8436],
         [ 0.1770,  0.9997,  0.7719],
         [-0.2276,  0.9997,  0.7444],
         [ 0.1770,  1.7312,  0.8381],
         [-0.2276,  1.7312,  0.8332],
         [ 0.1265,  2.0725,  0.1256],
         [-0.1770,  2.0725,  0.0970]]])
</code></pre>
<p>At the output I have this (print after self.encoder(y)):<br>
(tensor of 17 rows and 2 columns, dimension 3)</p>
<pre><code>tensor([[[0.1626, 0.2519],
         [0.1674, 0.2395],
         [0.1650, 0.2347],
         [0.1734, 0.2375],
         [0.1595, 0.2026],
         [0.1249, 0.3276],
         [0.1085, 0.2589],
         [0.0321, 0.4811],
         [0.0249, 0.3825],
         [0.0079, 0.4788],
         [0.0122, 0.4129],
         [0.0000, 0.6507],
         [0.0000, 0.6006],
         [0.0000, 0.9070],
         [0.0000, 0.8434],
         [0.0000, 0.8709],
         [0.0000, 0.8039]]])
</code></pre>
<p>And what I want at the output is a tensor like this:<br>
(tensor of 1 row and 2 columns, dimension 2)</p>
<pre><code>tensor([[-0.0739, -0.2400]])
</code></pre>

ptrblck: <p>If you want to reduce a tensor in the shape <code>[1, 17, 2]</code> to <code>[1, 2]</code> you could use any reduction operation such as e.g. <code>torch.mean</code>, <code>torch.sum</code> etc.<br>
Also a linear layer would work in case you want to reduce it, but the right approach depends on your use case in the end:</p>
<pre><code class="lang-python">x = torch.tensor([[[0.1626, 0.2519],
                   [0.1674, 0.2395],
                   [0.1650, 0.2347],
                   [0.1734, 0.2375],
                   [0.1595, 0.2026],
                   [0.1249, 0.3276],
                   [0.1085, 0.2589],
                   [0.0321, 0.4811],
                   [0.0249, 0.3825],
                   [0.0079, 0.4788],
                   [0.0122, 0.4129],
                   [0.0000, 0.6507],
                   [0.0000, 0.6006],
                   [0.0000, 0.9070],
                   [0.0000, 0.8434],
                   [0.0000, 0.8709],
                   [0.0000, 0.8039]]])


lin = nn.Linear(17, 1)
out = lin(x.permute(0, 2, 1))
print(out.shape)
# torch.Size([1, 2, 1])
out = out.squeeze(2)
print(out.shape)
# torch.Size([1, 2])
</code></pre>