Extract deep features from inception_v3

algeriapy: <p>When I tried to extract deep features using trained inception_v3 model</p>
<p>model = torchvision.models.inception_v3(pretrained=True)<br>
model.fc = nn.Linear(2048, 1)<br>
model.load_state_dict(torch.load(’./models/Beauty_inception_reg.pt’))<br>
feature_extractor = torch.nn.Sequential(*list(model.children())[:-1])</p>
<p>I got the following error :<br>
File “c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\conv.py”, line 345, in forward<br>
return self.conv2d_forward(input, self.weight)</p>
<p>File “c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\conv.py”, line 342, in conv2d_forward<br>
self.padding, self.dilation, self.groups)</p>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight 192 768 1 1, but got 2-dimensional input of size [1, 1000] instead</p>
<p>any Ideas what is causing that error ??</p>

ptrblck: <p>Wrapping child modules into an <code>nn.Sequential</code> container will only work in simple use cases, where each module is called sequentially and no functional calls are used in the <code>forward</code> method.</p>
<p>As you can see <a href="https://github.com/pytorch/vision/blob/34810c0c8e192caf29f05d7e4a02f36a9e9ebc02/torchvision/models/inception.py#L131-L184">here</a>, the <code>Inception</code> model uses some functional pooling layers, conditions, dropout, and a flattening inside the <code>forward</code>. All these functional API calls will be missing in your <code>feature_extractor</code>.</p>
<p>You could thus derive a custom model and manipulate the <code>forward</code> method as you wish or alternatively replace the unwanted layers with an <code>nn.Identity</code> to get the desired output.</p>

algeriapy: <p>Thanks a lot.<br>
I solved it by adding</p>
<p>model.fc = nn.Identity()</p>

algeriapy: <p>I have another question<br>
I am trying to train Resnet50 and Inception_v3 simultaneously<br>
I make the following class</p>
<pre><code class="lang-auto">class TwoInputsNet(nn.Module):
  def __init__(self):
    super(TwoInputsNet, self).__init__()
    self.model1 = torchvision.models.resnet50(pretrained=True)
    self.model1.fc = nn.Linear(2048, 1024)        
    self.model2 = torchvision.models.inception_v3(pretrained=True)
    self.model2.fc = nn.Linear(2048, 1024)
    self.fc2 = nn.Linear(2048, 1)


  def forward(self, input1, input2):
    c = self.model1(input1)
    f = self.model2(input2)
    combined = torch.cat((c,torch.Tensor(f)), dim=1)

    out = self.fc2(combined)
    return out
</code></pre>
<p>and I am getting the following error:</p>
<pre><code class="lang-auto">  File "D:/Face Beauty Pytorch/transfer_learning_inception_resnet50_regression.py", line 152, in &lt;module&gt;
    preds = model(images1, images2)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)

  File "D:/Face Beauty Pytorch/transfer_learning_inception_resnet50_regression.py", line 82, in forward
    f = self.model2(input2)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torchvision\models\inception.py", line 132, in forward
    aux = self.AuxLogits(x)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torchvision\models\inception.py", line 332, in forward
    x = self.conv1(x)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torchvision\models\inception.py", line 353, in forward
    x = self.bn(x)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\modules\batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)

  File "c:\users\fares\appdata\local\programs\python\python37\lib\site-packages\torch\nn\functional.py", line 1666, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 768, 1, 1])
</code></pre>
<p>Could you suggest any solution to this issue ??</p>

ptrblck: <p><code>Inception_v3</code> needs more than a single sample during training as at some point inside the model the activation will have the shape <code>[batch_size, 768, 1, 1]</code> and thus the batchnorm layer won’t be able to calculate the batch statistics.<br>
You could set the model to <code>eval()</code>, which will use the running statistics instead or increase the batch size.</p>

algeriapy: <p>Thanks a lot,<br>
I am getting new error</p>
<blockquote>
<p>File “D:/Face Beauty Pytorch/transfer_learning_inception_resnet50_regression.py”, line 90, in forward<br>
combined = torch.cat(c,torch.Tensor(f), dim=1)</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>

ptrblck: <p>Could you post the shapes of <code>c</code> and <code>f</code>, please?</p>

algeriapy: <p>Sure, I tried to summarize everything:<br>
When I tried to print the shape of c and f I got</p>
<blockquote>
<p>torch.Size([2, 1024])</p>
<p>print(f.shape)<br>
AttributeError: ‘InceptionOutputs’ object has no attribute ‘shape’</p>
</blockquote>
<p>so I tried</p>
<pre><code class="lang-auto">   def forward(self, input1, input2):
     c = self.model1(input1)
     f = self.model2(input2) 
     print(c.shape)
     l, ll= f
     print(l.shape)
     combined = torch.cat(c,torch.Tensor(f), dim=1) 
     out = self.fc2(combined)
     return out
</code></pre>
<p>and got</p>
<blockquote>
<p>torch.Size([2, 1024])<br>
torch.Size([2, 1024])</p>
<pre><code>combined = torch.cat(c,torch.Tensor(f), dim=1)
</code></pre>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>
<p>then I tried</p>
<pre><code class="lang-auto">  def forward(self, input1, input2):
    c = self.model1(input1)
    f = self.model2(input2) 
    print(c.shape)
    l, ll= f
    print(l.shape)
    combined = torch.cat(c,torch.Tensor(l), dim=1)
    out = self.fc2(combined)
    return out
</code></pre>
<p>the code did not run at all and it did not return any error</p>
<pre><code class="lang-auto">runfile('D:/Face Beauty Pytorch/transfer_learning_inception_resnet50_regression.py', wdir='D:/Face Beauty Pytorch')
  0%|          | 0/1100 [00:00&lt;?, ?it/s]
</code></pre>

ptrblck: <p>Thanks for the update.<br>
<code>InceptionOutputs</code> is a <code>namedtuple</code> containing the <code>logits</code> and <code>aux_logits</code>, so your approach of passing the first return value (<code>logits</code>) to <code>torch.cat</code> should be correct.</p>

algeriapy: <p>Thanks so much,<br>
It is working now</p>

jerpint: <p>If anyone is looking for a way to extract the features of inception_v3 layer by layer:</p>
<pre><code class="lang-auto">from torchvision.models.inception import Inception3
from torchvision.models.utils import load_state_dict_from_url
import torch.nn.functional as F


model_urls = {
    # Inception v3 ported from TensorFlow
    'inception_v3_google': 'https://download.pytorch.org/models/inception_v3_google-1a9a5a14.pth',
}
def inception_v3_sliced(pretrained=False, progress=True, stop_layer=3, **kwargs):
    r"""Inception v3 model architecture from
    `"Rethinking the Inception Architecture for Computer Vision" &lt;http://arxiv.org/abs/1512.00567&gt;`_.
    .. note::
        **Important**: In contrast to the other models the inception_v3 expects tensors with a size of
        N x 3 x 299 x 299, so ensure your images are sized accordingly.
    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
        progress (bool): If True, displays a progress bar of the download to stderr
        aux_logits (bool): If True, add an auxiliary branch that can improve training.
            Default: *True*
        transform_input (bool): If True, preprocesses the input according to the method with which it
            was trained on ImageNet. Default: *False*
    """
    if pretrained:
        if 'transform_input' not in kwargs:
            kwargs['transform_input'] = True
        if 'aux_logits' in kwargs:
            original_aux_logits = kwargs['aux_logits']
            kwargs['aux_logits'] = True
        else:
            original_aux_logits = True
        kwargs['init_weights'] = False  # we are loading weights from a pretrained model
        class Inception3Mod(Inception3):
          def __init__(self, stop_layer, **kwargs):
            super(Inception3Mod, self).__init__(**kwargs)
            self.stop_layer = stop_layer
          def _forward(self, x):
            layers = [
             self.Conv2d_1a_3x3,
             self.Conv2d_2a_3x3,
             self.Conv2d_2b_3x3,
             'maxpool',
             self.Conv2d_3b_1x1,
             self.Conv2d_4a_3x3,
             'maxpool',
             self.Mixed_5b,
             self.Mixed_5c,
             self.Mixed_5d,
             self.Mixed_6a,
             self.Mixed_6b,
             self.Mixed_6c,
             self.Mixed_6d,
             self.Mixed_6e,
             self.Mixed_7a,
             self.Mixed_7b,
             self.Mixed_7c,
            ]

            for idx in range(self.stop_layer):
              layer = layers[idx]
              if layer == 'maxpool':
                x = F.max_pool2d(x, kernel_size=3, stride=2)
              else:
                x = layer(x)
            return x, None


        model = Inception3Mod(**kwargs, stop_layer=stop_layer)
        state_dict = load_state_dict_from_url(model_urls['inception_v3_google'],
                                              progress=progress)
        model.load_state_dict(state_dict)
        if not original_aux_logits:
            model.aux_logits = False
            del model.AuxLogits
        return model

    return Inception3Mod(**kwargs)
</code></pre>
<p>then call</p>
<p><code>model = inception_v3_sliced(pretrained=True, stop_layer=12)</code></p>

jorisvane: <p>Can you show the full code where you managed to get inceptionv3 without the classification layer working?</p>

algeriapy: <p>Hi Jorisvane,</p>
<p>model = torchvision.models.inception_v3(pretrained=True)<br>
model.fc = nn.Identity()<br>
model.eval</p>
<p>then you can have the deep features</p>