Adding layers to MobileNetV2

Osnat: <p>Hi all,</p>
<p>I’m new to the DL field and pytorch.<br>
I want to use pretrained MobileNetV2 but instead of its classifier, add 1x1 convolutional layer+max pooling+convolutional layer+other linear layers (all this in order to reduce the output to less dimensions so I can cluster it later on).</p>
<p>When I tried to run the next code I got “Expected 4-dimensional input for 4-dimensional weight [1280, 1280, 2, 2], but got 2-dimensional input of size [2, 1280] instead”.</p>
<p>Here’s the code:<br>
model = models.mobilenet_v2(pretrained=True)<br>
for param in model.parameters():<br>
param.requires_grad = False</p>
<p><span class="hashtag">#modify</span> last layer of the net<br>
fc = nn.Sequential(<br>
nn.Conv2d(1280, 1280, kernel_size=(2, 2), stride=(2, 2), bias=False),<br>
nn.MaxPool2d(kernel_size=2, stride=2, padding=0),<br>
nn.Conv2d(1280, 320, kernel_size=(1, 1), stride=(1, 1), bias=False),<br>
nn.Dropout(0.2),<br>
nn.BatchNorm1d(320),<br>
nn.ReLU(),<br>
nn.Dropout(0.3),<br>
nn.Linear(320,NUM_CLASSES),<br>
n.LogSoftmax(dim=1)<br>
)<br>
model.classifier = fc<br>
criterion = nn.NLLLoss()<br>
optimizer = torch.optim.Adam(model.classifier.parameters(), lr=0.0015)<br>
model.to(DEVICE)<br>
summary(model, input_size=(3, 224, 224))</p>
<p>Is there someone who managed to that successfully? Or tell me where am I wrong?</p>
<p>Thanks a lot in advance!</p>

ptrblck: <p>The original <code>forward</code> implementation will flatten the activation output in <a href="https://github.com/pytorch/vision/blob/42aa9b263eaea3553369ff39268cfe295c7a57b0/torchvision/models/mobilenet.py#L166">this line of code</a> before passing it to the <code>classifier</code>.<br>
You could add a custom module at the beginning of your <code>nn.Sequential</code> container and reshape the activation back or alternatively create a custom model and override the <code>forward</code> method.</p>
<p>Once this is solved, note that your <code>nn.Sequential</code> module is missing an <code>nn.Flatten</code> layer after the last <code>nn.Conv2d</code> layer (and also uses <code>nn.BatchNorm1d</code>, which might be a typo).</p>

Osnat: <p>Thank you very much for answering!</p>
<p>I wanted to implement your suggestion of reshaping the activation back but I couldn’t figure out how to go from 2-dimensional weight [2,1280] to 4-dimensional [1280, 1280, 2, 2] (I guess the 2,2 are due to the kernel and stride).</p>
<p>So, I tried to define the forward only on the mobilenetv2.features as written below (thought this way I will just avoid the activation function) but that resulted in the next error “Expected 4-dimensional input for 4-dimensional weight [1280, 1280, 1, 1], but got 2-dimensional input of size [2, 62720] instead”.</p>
<p>Here’s the current code:</p>
<p>class Flatten(nn.Module):<br>
def forward(self, input):<br>
return input.view(input.size(0), -1)</p>
<p>class MobileNet_ConvAdd(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MobileNet_ConvAdd, self).<strong>init</strong>()<br>
self.model_M = models.mobilenet_v2(pretrained=True)<br>
self.model_M.classifier = nn.Identity()<br>
for param in self.model_M.parameters():<br>
param.requires_grad = False</p>
<pre><code>    self.MobileNet_ConvAdd_conv1 = nn.Sequential(
                                                nn.Flatten(),
                                                nn.Conv2d(1280, 1280, kernel_size=(2, 2), stride=(2, 2), bias=False),
                                                nn.MaxPool2d(kernel_size=2, stride=2, padding=0),
                                                nn.ReLU(),
                                                nn.Conv1d(1280, 320, kernel_size=(1, 1), stride=(1, 1), bias=False),
                                                nn.AvgPool2d(kernel_size=2, stride=2, padding=0),
                                                nn.Flatten()
                                                )

    self.MobileNet_ConvAdd_fc2 = nn.Sequential(
                                    nn.Dropout(0.2),
                                    nn.BatchNorm1d(1280), #320
                                    nn.ReLU(),
                                    nn.Dropout(0.3),      
                                    nn.Linear(320,NUM_CLASSES),
                                    nn.LogSoftmax(dim=1)
                                    )

def forward(self, x):
    x = self.model_M.features(x)
    x = self.MobileNet_ConvAdd_conv1(x)
    x = self.MobileNet_ConvAdd_fc2(x)
    return x</code></pre>

ptrblck: <p>The first <code>nn.Flatten()</code> layer in <code>self.MobileNet_ConvAdd_conv1</code> would flatten the incoming tensor, which will create a shape mismatch in the following <code>nn.Conv2d</code>.<br>
<code>nn.X2d</code> layers expect an input activation of <code>[batch_size, channels, height, width]</code>, while the <code>nn.Linear</code> layer expects an activation of <code>[batch_size, in_features]</code> (in the default setup).</p>
<p>Remove the first <code>Flatten</code> layer and make sure you really want to use <code>nn.BatchNorm1d</code> in the second <code>nn.Sequential</code> container.</p>