Fastest way to process images for Resnet

username: johnny12
cooked: <p>I have a dataset of (N, H, W, C) images with values in [0, 255]. Some are saved as np.uint8, some as torch.uint8.<br>
I need to process them to pass them to Resnet, and I need to do it efficiently.<br>
What I currently do is</p>
<pre><code class="lang-auto">import torchvision.transforms as T

transforms = nn.Sequential(
    T.Resize(256, interpolation=3, antialias=None),
    T.CenterCrop(224),
    T.ConvertImageDtype(torch.float), # also divides by 255 if input is uint8
    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
)

img4resnet = transforms(img.permute(0, 3, 1, 2).contiguous())
</code></pre>
<p>This is slow, especially if I need to pass many images at once.<br>
There is ToTensor which is faster than permute + contiguous and accepts np.array, but it works with only 1 image at the time (I get an error saying I tried to pass 4D input and it expects 3D). If I have many images I am not sure if looping would be faster…</p>
<p>I have found other old issues asking something similar, but they are all years old.</p>
<p>Finally, I see that the latest version of torchvision now gets the preprocessing transformation from the weights. Is it faster? Can I just replace my current transform?</p>

username: eqy
cooked: <p>I’m not sure that the recent changes to <code>torchvision</code> actually improve the speed of the transforms rather than making them more convenient, but it could be worthwhile to update and see if the implementation(s) have become more efficient anyway.</p>
<p>However, for recent generations of GPUs with tensor cores, you may find that a channels-last memory layout (NHWC) actually yields faster execution in convolutional models such as ResNet, so I would also check if leaving the images in channels-last speeds things up in addition to removing a preprocessing step.</p>

username: johnny12
cooked: <p>Thanks for your reply, but torchvision Resnet must get channel first because of conv2d. Is there a way to pass it with channel last?</p>
<pre><code class="lang-auto">from torchvision import models
import torch

model = models.resnet18(pretrained=True, progress=False)
x = torch.rand(1,32,32,3)
model(x)

[...]
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 32, 32, 3] to have 3 channels, but got 32 channels instead
</code></pre>

username: eqy
cooked: <p>Does converting the model to <code>torch.channels_last</code> work for you?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; model = torchvision.models.resnet50().to(memory_format=torch.channels_last)
&gt;&gt;&gt; input = torch.randn(4, 3, 224, 224).to(memory_format=torch.channels_last)
&gt;&gt;&gt; model(input)
tensor([[ 1.0137,  0.3668, -0.4023,  ..., -0.4069,  0.9567, -0.3115],
        [ 0.9884,  0.3564, -0.6024,  ..., -0.5426,  0.8428, -0.3755],
        [ 1.0849,  0.2691, -0.3835,  ..., -0.3646,  0.9449, -0.1472],
        [ 0.8947,  0.1875, -0.3154,  ..., -0.4034,  0.7230, -0.4413]],
       grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>Also note that channels-last inputs would still have <em>shape</em> [N, C, H, W]:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; input.shape
torch.Size([4, 3, 224, 224])
&gt;&gt;&gt; input.is_contiguous()
False
&gt;&gt;&gt; input.is_contiguous(memory_format=torch.channels_last)
True
</code></pre>
<p>You can verify this by checking e.g., <code>img.permute(0, 3, 1, 2).is_contiguous(memory_format=torch.channels_last)</code> without calling <code>.contiguous()</code> (which would convert it to channels-first format).</p>

username: johnny12
cooked: <p>I didn’t know about <code>memory_format=torch.channels_last</code>, thanks!<br>
I tried it but the speed seems exactly the same.</p>

username: eqy
cooked: <p>What GPU are you running on?</p>
<p>You can also look at <code>torch.compile</code> as a means to increasing performance:<br>
<a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html</a></p>

username: ptrblck
cooked: <p>NIT: channels-last should be used with <code>amp</code> as I would not expect to see any major perf. improvements in FP32.</p>