Resize images (torch.Tensor) already loaded on GPU keeping their aspect ratio

username: evgeniititov
cooked: <p>Hello everyone,</p>
<p>Could anyone give me a hand with the following please.<br>
Context: I am working on a system that processed videos. I take N frames, .cat() them in a batch and move to GPU.<br>
Then, I want to run this batch through a neural net (YOLO). In order to do it, I need to resize each image in the batch to the standard 416 x 416 size keeping the aspect ratio. The main problem here - I cannot move it back to CPU, we’re trying to minimize host-device data transfers, so it needs to be done directly on GPU.</p>
<p>Could anyone please explain how to do it?</p>
<p>What I am doing now:</p>
<ol>
<li>I am looping over my torch.Tensor (batch of images on GPU) and want to resize each tensor separately. I am sure there’s a better way to do it all together.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49a8f7aed35dc809617a179cf615f097a0808973.png" data-download-href="https://discuss.pytorch.org/uploads/default/49a8f7aed35dc809617a179cf615f097a0808973" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49a8f7aed35dc809617a179cf615f097a0808973.png" alt="image" data-base62-sha1="avCRJ51fNC8Ji2PxVkJvOsfdz7J" width="690" height="325" data-dominant-color="2C2D27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1076×508 31.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is obviously not working.</li>
</ol>
<p>Thank you in advance!</p>
<p>Eugene</p>

username: ptrblck
cooked: <p>You could use <code>F.interpolate</code> to reshape the tensors instead of the current copy-approach.<br>
Let me know, if that would work for you.</p>

username: evgeniititov
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks a lot for a quick reply.</p>
<p>Yes, I’ve tried this method and it is working. However, I need my image to be a square (416x416). When I use the method you suggested I am getting:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5c5b06cfd37e399f8ffcbe8c096b239905acc85.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5c5b06cfd37e399f8ffcbe8c096b239905acc85" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5c5b06cfd37e399f8ffcbe8c096b239905acc85.png" alt="image" data-base62-sha1="nEukrnlpIHlTy3Z4l88RP1mlvBX" width="690" height="207" data-dominant-color="2D2E27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×312 22.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f262b05c644447603fdc848c8ea339e1a0213a92.png" alt="image" data-base62-sha1="yAeZfllGi4bmAT6n267DWIkuvWW" width="418" height="313"></p>
<p>If I am not mistakes, now I need to create a tensor of the shape I need and fill it with (126, 126, 126) colour and somehow combine with resized image. Is that correct? Could you please give me a hand with this or share I link where I could read how to do it?</p>
<p>Thanks in advance.<br>
Eugene</p>

username: ptrblck
cooked: <p>What kind of shape does <code>tensor</code> have and what are you passing as <code>new_dimension</code>?</p>

username: evgeniititov
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>TENSOR SHAPE: torch.Size([5, 3, 1080, 1920]) - a batch of frames (5 in this case), rgb.<br>
NEW DIMENSIONS: (234, 416) - recalculated frame size with the preserved aspect ratio.</p>
<p>No doubts you’ve already understood what I want, but just to reiterate I would love each image to be not (234, 416) but (416, 416), the rest should be filled with (126, 126, 126) colour.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Ah, thanks for the explanation. I totally misunderstood the use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you check, if this would work?</p>
<pre><code class="lang-python">x = torch.randn(5, 3, 1080, 1920)
y = F.interpolate(x, (234, 416))
res = torch.ones(5, 3, 416, 416) * 126.
res[:, :, 91:325, :] = y
</code></pre>

username: evgeniititov
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>It seems to be working almost perfectly! Thank you very much. The only 2 issues is the colour background we added is white:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5a0be48b97e5aa4a50e4672265f9b2f398fa019.png" alt="image" data-base62-sha1="utQ5I9wUNgU29rwWgdJkRVBTVuV" width="416" height="493"></p>
<p>And when I test it for <em>tall</em> image (height &gt; width), it crashes with the following error:<br>
<strong>The expanded size of the tensor (416) must match the existing size (312) at non-singleton dimension 3.  Target sizes: [5, 3, 234, 416].  Tensor sizes: [3, 416, 312]</strong></p>

username: ptrblck
cooked: <p>Could the output be somehow “normalized” such that the gray color would be visualized as white?<br>
The background should have the color <code>[126, 126, 126]</code>, so I’m unsure why it’s shown as white.</p>
<p>I’m getting a gray background here:</p>
<pre><code class="lang-python">x = torch.randint(0, 255, (5, 3, 1080, 1920)).float()
y = F.interpolate(x, (234, 416))
res = torch.ones(5, 3, 416, 416) * 126.
res[:, :, 91:325, :] = y
plt.imshow(res[0].permute(1, 2, 0).byte())
</code></pre>
<p>Assuming that a “tall” image should be resizes as <code>[416, 234]</code>, you might need to change the code to:</p>
<pre><code class="lang-python">if tall:
    res[:, :, :, 91:325] = y
else:
    res[:, :, 91:325, :] = y
</code></pre>
<p>or</p>
<pre><code class="lang-python">if tall:
    idx = 3
else:
    idx = 2
res.index_copy_(idx, torch.arange(91, 325), y)
</code></pre>

username: evgeniititov
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I think you might be right, the background color should be correct. Thank you for helping with this.</p>
<p>I am sorry I think I failed to specify that the system can technically work with images of any size. Tall, wide, with different proportions. If I am not mistaken, the 91 and 325 numbers are hard coded for the 1080, 1920 case. Could we somehow use the new dimensions I am calculating to generalize this step, so it works with images of any random size?</p>
<p>UPDATE: Done. Everything’s working.</p>
<p>Thank you for your help and time!</p>
<p>Eugene</p>

username: evgeniititov
cooked: <p>In case anyone ever finds this thread having the same question. Here’s the working code I ended up with:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ec74ab9cebaa797a3c05a3220991a47396e662e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9ec74ab9cebaa797a3c05a3220991a47396e662e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ec74ab9cebaa797a3c05a3220991a47396e662e.png" alt="image" data-base62-sha1="mECq7ihgKNokp8ctZOdvIOjBYwS" width="690" height="351" data-dominant-color="2C2D27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×529 34.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>