Image size with pretrained model

username: mv000
cooked: <p>Hello experts,<br>
I am using various pretrained models like ViT and CvT to do image classification.</p>
<p>My images are in RGB format, but one dimension is much larger than the pretrained model was trained on.  Specifically I have images with dimension (3x224x1000) and the pretrained is (3x224x224).</p>
<p>To resize the images to the required dimensions, I want to apply a <strong>trainable/learnable dimensionality reduction</strong> technique that operates <strong>gradually</strong>. However, techniques like convolutional layers and max-pooling tend to reduce the size too aggressively, which results in a significant loss of information (and max-pool isn’t trainable).</p>
<p>For example:</p>
<pre><code class="lang-auto">## this moves too fast, losing information
self.conv = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=(3,3), stride=(1,2), padding=1) 
</code></pre>
<p>The stride in the width dimension cuts the size in half too quickly. I’m looking for a <strong>trainable</strong>, <strong>high-fidelity</strong> method to reduce the dimensions more <strong>gradually</strong> along the width axis without losing too much information in the process.</p>
<p>Any suggestions or guidance would be greatly appreciated!</p>
<p>Thank you!</p>

username: J_Johnson
cooked: <p>If the image size is 224x1000, why not just cut it into 4 smaller images and run those through the vision part of the model? Then you can stack those on the batch dimension through the feed forward part of the pretrained model, and then concat them before a final trainable linear layer. That seems to make the most sense, vs. trying to train something from scratch on the input side.</p>

username: mv000
cooked: <p>I am worried about losing context and info by cutting through objects when I make the split.  I suppose I could use a sliding window with overlap, and/or attention heads.  It’s an interesting option, and I’m looking into it.  Thanks for the suggestion!</p>
<p>I’m looking for as many options as possible since this is a hard classification task (subtle differences decide the class). And there is something desirable about simply adding a trainable dimension reduction layer in front of the model. The model will have to be trained for a long time to fine-tune anyway, so I’m not worried about training things from scratch.</p>
<p>Any other options?</p>

username: KFrank
cooked: <p>Hi mv!</p>
<aside class="quote no-group" data-username="mv000" data-post="1" data-topic="209913" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/bb73d2/48.png" class="avatar"> mv000:</div>
<blockquote>
<p>My images are in RGB format, but one dimension is much larger than the pretrained model was trained on.  Specifically I have images with dimension (3x224x1000) and the pretrained is (3x224x224).</p>
</blockquote>
</aside>
<p>One approach would be to use <code>adaptive_avg_pool2d()</code> to<br>
reduce your width dimension as gradually as you like.  You<br>
could add trainability to this by interposing some trainable<br>
convolution layers.</p>
<p>You could also recognize that as you reduce your width by a<br>
factor of two with a <code>stride = 2</code> convolution, you can “move”<br>
spatial information into the channels dimension by increasing<br>
the value of <code>out_channels</code> of your stride-2 convolution.  You<br>
can then use subsequent convolutions to move the information<br>
that you put in the channels dimension back into your reduced<br>
width dimension.</p>
<p>Something like this (but you would most likely want some<br>
non-linear activations to get good trainability):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.4.1
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; img = torch.randn (3, 224, 1000)
&gt;&gt;&gt;
&gt;&gt;&gt; conv = torch.nn.Conv2d (3, 3, 3, padding = 'same')                # for trainability
&gt;&gt;&gt;
&gt;&gt;&gt; i1 = torch.nn.functional.adaptive_avg_pool2d (img, (None, 900))   # gradually reduce
&gt;&gt;&gt; i1.shape
torch.Size([3, 224, 900])
&gt;&gt;&gt; i1a = conv (i1)                                                   # trainable / learnable
&gt;&gt;&gt; i2 = torch.nn.functional.adaptive_avg_pool2d (i1a, (None, 800))   # gradually reduce
&gt;&gt;&gt; i2.shape
torch.Size([3, 224, 800])
&gt;&gt;&gt;
&gt;&gt;&gt; # transfer some spatial information into channels dimension
&gt;&gt;&gt; convA = torch.nn.Conv2d (in_channels = 3, out_channels = 12, kernel_size = 3, stride = (1, 2), padding = 1)
&gt;&gt;&gt; convB = torch.nn.Conv2d (in_channels = 12, out_channels = 3, kernel_size = 3, padding = 'same')
&gt;&gt;&gt;
&gt;&gt;&gt; iA = convA (img)
&gt;&gt;&gt; iA.shape
torch.Size([12, 224, 500])
&gt;&gt;&gt; iB = convB (iA)
&gt;&gt;&gt; iB.shape
torch.Size([3, 224, 500])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: mv000
cooked: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="209913">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>adaptive_avg_pool2d</code></p>
</blockquote>
</aside>
<p>This is also an interesting option!</p>
<p>I will have to try the different methods to see where I get best results.  Good note about “moving” spatial info to the channels.</p>
<p>Thanks again!</p>
<p>(all suggestions continue to be welcome!)</p>