VGG feature extractor pretrained, with bigger input images – pytroch

MikeTensor: <p>How does pytorch init. the other paramters of a vgg16 <strong>feature extractor</strong> (no classifier), if the input images are much bigger as when pretrained?</p>
<p>So lets say I use 700 x 1200 colour images (not cropped) as input.<br>
Create the vgg16-feature-extractor-model and load the pretrained values.</p>
<p>From my point of view the parameters for the standard vgg16 size are given (224 x 224).<br>
But what happens with the parameters “outside” of this area – does pytorch initialize them …</p>
<ul>
<li>randomly?</li>
<li>kind of ‘interpolation’ based on the pretrained values? ,</li>
<li>…</li>
</ul>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> any idea? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>P.S. Is there any official source for this certain behaviour?</p>

ptrblck: <p>The <code>classifier</code> of <code>VGG</code> is defined <a href="https://github.com/pytorch/vision/blob/78c271974f94585f45cd696f66d08dae538a9207/torchvision/models/vgg.py#L43-L51">here</a> as:</p>
<pre><code class="lang-python">        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, num_classes),
        )
</code></pre>
<p>and does not change the <code>in_features</code> depending on the input shape.<br>
This might sound strange as a larger input should have more “features”.<br>
However, <code>torchvision</code> classification models use adaptive pooling layers (as seen <a href="https://github.com/pytorch/vision/blob/78c271974f94585f45cd696f66d08dae538a9207/torchvision/models/vgg.py#L67">here</a>) to create the desired activation output shape to avoid shape mismatch errors in the first linear layers. This allows you to use variable input shapes as long as the inputs are not too small, which would raise errors in e.g. conv or pooling layers.</p>

MikeTensor: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I think there is a misunderstanding.<br>
I am only referring to the vgg16 feature_extractor itself (without the classifier).</p>
<p>So the Parameters for the weights of the pretrained weights are in orange (224 x 224).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/159b0596819be4155401b284d965b627156f3531.png" alt="grafik" data-base62-sha1="3589TypDqWsguMOktSWgWCyK22Z" width="449" height="185"></p>
<p>What happens if my input is larger (blue area).<br>
How are those parameters outside the orange (pretrained area) initialized?</p>
<blockquote>
<p>So lets say I use 700 x 1200 colour images (not cropped) as input (blue are in the figure above)<br>
Create the vgg16-feature-extractor-model and load the pretrained values (orange area = pretrained).</p>
<p>From my point of view the parameters for the standard vgg16 size are given (224 x 224) → orange area.<br>
But what happens with the <strong>parameters “outside” of this area (blue area)</strong> – does pytorch initialize them …</p>
<pre><code>randomly?
kind of ‘interpolation’ based on the pretrained values? ,
…
</code></pre>
</blockquote>
<blockquote>
<p>P.S. Is there any official source for this certain behaviour?</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="MikeTensor" data-post="3" data-topic="176366">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/5daacb/48.png" class="avatar"> MikeTensor:</div>
<blockquote>
<p>So the Parameters for the weights of the pretrained weights are in orange (224 x 224).</p>
</blockquote>
</aside>
<p>I think you there is a misunderstanding between the (input) activation shape and trainable parameters.<br>
The weights of conv layers are typically small filter kernels with a spatial shape of <code>3x3</code> or <code>5x5</code>.<br>
The orange area represents the spatial input size and the conv layer will apply a convolution to it using the smaller kernel. The same logic applies to the blue box and the filter kernel will just have to “move around more” (of course internally conv layers don’t have to use a nested loop to shift the filter kernel.)</p>
<p><a href="https://github.com/vdumoulin/conv_arithmetic#convolution-animations">These animations</a> might help.<br>
E.g. take a look at the first animation. You can increase the size of the blue input which will allow the dark blue filter to be moved more creating a larger spatial output size.</p>