Transfer learning usage with different input size

Dr_John: <p>VGG16 and Resnet require input images to be of size 224X224X3. I know my question may be stupid, but is there any chance to use these pretrained networks on a datasets with different input sizes (for example, black and white images of size 224X224X1? or images of different size, which I don’t want to resize?)</p>
<p>Thanks!</p>

ptrblck: <p>In your first use case (different number of input channels) you could add a conv layer before the pre-trained model and return 3 out_channels.</p>
<p>For different input sizes you could have a look at the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py#L29">source code of vgg16</a>. There you could perform some model surgery and add an adaptive pooling layer instead of max pooling to get your desired shape for the classifier (<code>512*7*7</code>).</p>
<p>Note that the performance of your pre-trained model might differ for different input sizes.</p>

Dr_John: <p>Thanks a lot ptrblck! The way to implement the first part of your answer is something like -</p>
<pre><code class="lang-auto">    model = models.vgg16(pretrained=True)
    first_conv_layer = list(nn.Conv2d(1, 3, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True))
    first_conv_layer .extend(list(model.features))  
    model.features= nn.Sequential(*first_conv_layer )  

</code></pre>
<p>?</p>

ptrblck: <p>Looks good with some minor tweaks.<br>
I think you’ll get an error calling <code>list(nn.Conv2d...)</code>.<br>
Also, since we didn’t change the input size, we should take care to return the same spatial dimensions after the first conv layer. Using <code>kernel_size=3</code> and <code>padding=1</code> will result in the same size.<br>
Here is the code:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, 224, 224)
model = models.vgg16(pretrained=False) # pretrained=False just for debug reasons
first_conv_layer = [nn.Conv2d(1, 3, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True)]
first_conv_layer.extend(list(model.features))  
model.features= nn.Sequential(*first_conv_layer )  
output = model(x)
</code></pre>

Dr_John: <p>Thanks a lot ptrblck!</p>

pvardanis: <p>Assuming we use a pretrained vgg, let’s say we add a conv layer in the beggining (i.e. if we want to have as input a tensor with 2 channels instead of 3). And we also drop the last fc layers to match our output size. When training we usually freeze all params in model.features with requires_grad=False and we only train model.classifier part. Now that we also added a conv layer in the beginning, do we need to train this part from model.features as well? Example code:</p>
<pre><code class="lang-auto">def build_model(self):

        if self.model_type in ['vgg-16', 'vgg-19']:
            
            assert self.image_size == 224, "ERROR: Wrong image size."
            model = torchvision.models.vgg16(pretrained=True) if self.model_type == 'vgg-16' else torchvision.models.vgg19(pretrained=True)
            if self.input_ch != 3:
                first_conv_layer = [nn.Conv2d(self.input_ch, 3, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True)]
                first_conv_layer.extend(list(model.features))  
                model.features= nn.Sequential(*first_conv_layer)  

            model.classifier[-1] = nn.Linear(4096, 1000)
            model.classifier.add_module('7', nn.ReLU())
            model.classifier.add_module('8', nn.Dropout(p=0.5, inplace=False))
            model.classifier.add_module('9', nn.Linear(1000, self.output_ch))
            model.classifier.add_module('10', nn.LogSoftmax(dim=1))
            
            for param in model.features.parameters(): # disable grad for trained layers
                param.requires_grad = False

            return model.to(self.device)```</code></pre>

ptrblck: <p>Yes, you probably should train the first layer. Otherwise it’ll stay randomly initialized.<br>
You could of course freeze all other layers, which are already pretrained. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

pvardanis: <p>I guess the following code should do the work:</p>
<pre><code class="lang-auto">            assert self.image_size == 224, "ERROR: Wrong image size."
            model = torchvision.models.vgg16(pretrained=True) if self.model_type == 'vgg-16' else torchvision.models.vgg19(pretrained=True)
            if self.input_ch != 3:
                first_conv_layer = [nn.Conv2d(self.input_ch, 3, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True)]
                first_conv_layer.extend(list(model.features))  
                model.features= nn.Sequential(*first_conv_layer)  

            for param in model.features[1:].parameters(): # disable grad for trained layers
                param.requires_grad = False
</code></pre>

pvardanis: <p>Hi again,</p>
<p>I’m trying to train a pretrained vgg-16 on the Fashion MNIST dataset. I added an extra conv layer at the beggining and modified the last layers to end up to 10 nodes. Also, I applied <code>nn.LogSoftmax(dim=1)</code> at the end, because I’m using <code>nn.NLLLoss()</code>. I disabled gradients for all gradients on model.features, except the first layer, so only the first conv layer and model.classifier layers are trained. I ran my code in two PCs one with 5gb GPU memory and one with 6gb, and both failed. Then, I tried Google Collab on GPU and still it takes too long to run even one epoch. This is my code:</p>
<pre><code class="lang-auto">if self.model_type in ['vgg-16', 'vgg-19']:

            

            assert self.image_size == 224, "ERROR: Wrong image size."

            model = torchvision.models.vgg16(pretrained=True) if self.model_type == 'vgg-16' else torchvision.models.vgg19(pretrained=True)

            if self.input_ch != 3:

                first_conv_layer = [nn.Conv2d(self.input_ch, 3, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True)]

                first_conv_layer.extend(list(model.features))  

                model.features= nn.Sequential(*first_conv_layer)  

            model.classifier[-1] = nn.Linear(4096, 1000)

            model.classifier.add_module('7', nn.ReLU())

            model.classifier.add_module('8', nn.Dropout(p=0.5, inplace=False))

            model.classifier.add_module('9', nn.Linear(1000, self.output_ch))

            model.classifier.add_module('10', nn.LogSoftmax(dim=1))

            

            for param in model.features[1:].parameters(): # disable grad for trained layers

                param.requires_grad = False
</code></pre>
<p>Is this normal, or am I doing something wrong?</p>
<p>UPDATE: For <code>batch_size=64</code> it took more than 10 minutes for 1 epoch. Is this normal? I’m going to try a larger batch_size as well to see what happens.</p>
<p>UPDATE 2: For a batch size of 256 I got this error:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 784.00 MiB (GPU 0; 15.90 GiB total capacity; 14.87 GiB already allocated; 323.88 MiB free; 14.88 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>Wow, 14.88 GB already reserved? Isn’t that too much?</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/vgg-16-training-time-google-collab/72304/2">here</a>.</p>

Muhammad_Abubakar: <p>I am building a classifier using MRIs  with pretrained alexnet so my batch size has become the number of MRI slices for example one MRI have 30 slices so the input shape becomes [30, 3 , 256, 256] but i want to parallelize the training by passing batches of  MRIs, lets say batches of 8 MRIs and the input shape will be [8, 30, 3, 256, 256]. How can i alter Alexnet or any pretrained model from torchvision to accept this input size?</p>

ptrblck: <p>You could reshape the input such that the batches and sliced are both in dim0, which would thus increase the batch size via <code>x = x.view(-1, 3, 256, 256)</code>.<br>
This would treat each slice as an own input in the same way as your previous approach.</p>
<p>Alternatively, you might want to treat the slice dimension as the depth dimension.<br>
In that case, you would need to change the model architecture, since you would need 3D layers such as <code>nn.Conv3d</code>.</p>

Muhammad_Abubakar: <p>Now I am getting this error RuntimeError: CUDA out of memory. Tried to allocate 62.00 MiB (GPU 0; 8.00 GiB total capacity; 5.90 GiB already allocated; 55.97 MiB free; 5.96 GiB reserved in total by PyTorch)</p>

ptrblck: <p>This would mean that a batch size of 240 images is too large for your current device.<br>
You could either decrease the batch size, the number of slices, or try to use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

lesolorzanov: <p>And what about if I have images bigger than 224? maybe like 562x562x3? what would be the best way to modify the network?</p>

Mrig: <p>I didn’t get the 5th line</p>
<pre><code class="lang-auto">model.features= nn.Sequential(*first_conv_layer ) 
</code></pre>
<p>whats it is doing? I</p>

ptrblck: <p>This line unwraps the <code>list</code> and passes each element sequentially to the <code>nn.Sequential</code> container.<br>
If you would pass the <code>list</code> directly, you would get a <code>TypeError</code>:</p>
<pre><code class="lang-python">my_list = []
my_list.append(nn.Linear(1, 1))
my_list.append(nn.Linear(1, 1))
my_list.append(nn.Linear(1, 1))
module = nn.Sequential(my_list)
&gt; TypeError: list is not a Module subclass

module = nn.Sequential(*my_list) # works
</code></pre>

sammy007: <p>I know we can add a layer before and fc at the end to match the no of classes. But when we train model by freezing the pretrained part, from where, the layer added at input side will get gradients… Even if it gets from the latest layer which is being trained does dimensions match and is it right to receive gradient from that layer, and its a partial derivative related calculations and it shd be receiving gradients from the layer next to it,</p>

Hongnan_G: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> My virgin post will be dedicated here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>First off, I really appreciate your swift and detailed responses, most of the PyTorch forums I visited has your name. It is really not easy replying to so many issues and questions.</p>
<p>Now, although not a complete beginner, but I am relatively new to PyTorch, and I have decided that it is my native language from now. The question is relatively straightforward, for pretrained models like <code>VGG16</code> or even more advanced models from <a href="https://github.com/rwightman/pytorch-image-models" rel="noopener nofollow ugc">pytorch-image-models</a>, the question on whether I need to input the shape that the pretrained model is trained on remains a puzzle to me. From my basic understanding, Convolutional Layers are invariant of image size, but Dense Layers aren’t. But I was doing resizing of image sizes blatantly in PyTorch’s pipeline without any regards to why it worked like a charm.</p>
<p>In short, I use either <code>albumentations</code> or <code>torchvision</code> to resize my images to say 512x512 (as usually this image size seems to be the favourite among Kagglers). Then I will have a very simple class wrapper around <code>nn.module</code> as follows:</p>
<pre><code class="lang-auto">class CustomPretrainedModel(nn.Module):
    def __init__(self, pretrained: bool=True):
        super().__init__()

        self.model = create_pretrained_model_here(......)
        n_features = self.model.classifier.in_features
        self.model.classifier = nn.Linear(n_features, num_classes)
        

    def forward(self, input_neurons):
        output_predictions = self.model(input_neurons)
        return output_predictions
</code></pre>
<p>Then I run the pipeline without any hiccups. So my main question is, the flexibility to input any image size in PyTorch’s own native pretrained models/other famous PyTorch based repos is mainly due to the change in <code>nn.AdaptiveAvgPool2d((x,x))</code>? I am not too familiar with it so it would be great to know how pretrained models can handle all types of image sizes  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Thanks for the kind words and yes you are correct. The input shapes are flexible due to the usage of the adaptive pooling layers, which make sure the output activation shape matches the desired feature dimension of the linear layers.<br>
Note that larger inputs would work without a problem, but the model would be still limited to a min. input size, since the intermediate activations could become empty otherwise.</p>