Nn.interpolate function unexpected behaviour

xvdp: <p>Hi folks, a while ago I built myself a translation of the torchvision.transforms library that operates over tensors.<br>
To substitute PIL (or accimage) resize() i use nn.interpolate(), which at first sight works (in -linear) mode but with slight different results on high contrast pixel differences, no difference in mean, but different std.<br>
On -cubic mode its a bit stranger, min max go beyond original value range., e.g instead of 0-1 range I will get something like -0.1 to 1.15</p>
<p>Is this particularly important, no, data can still be trained. But benchmarking results are different.<br>
Why I did this, utilizing tensor augmentation is of value, as demonstrated recently in Nvidia’s ADAStyleGAN</p>
<p>I am curious why the differences. I dug into the code and while PIL is easy to read, torch requires a bit more time devoted to the issue.</p>
<p>I got same results in cpu and cuda., both equally different from pil and accimage.</p>

xvdp: <p>example<br>
differences appear minor, but they change benchmarking results</p>
<pre data-code-wrap="python"><code class="lang-python">import io
import requests
import numpy as np
import PIL
import matplotlib.pyplot as plt
import torch
from torch.nn.functional import interpolate
![difpytorch_pil|690x268](upload://lFVUHsLWLC5JzwxOvPmZznqUsum.jpeg) 
def test_dif(dtype="float64", mode="bilinear", show=False):

    sig = lambda x, msg="": print("%s\tmin: %.4f, max: %.4f mean: %.4f, std: %.4f \tshape: %s, dtype: %s"%(msg, x.min(), x.max(), x.mean(), x.std(), str(tuple(x.shape)), str(x.dtype)))
    url = ("https://ichef.bbci.co.uk/news/976/cpsprodpb/10207/production/_116155066_campercats.png")
    pimg = PIL.Image.open(io.BytesIO(requests.get(url).content))
    tensor = torch.from_numpy((np.array(pimg)/255).astype(dtype)).permute(2, 0, 1).contiguous()
    tensor = tensor.view(1, *tensor.shape)
    print(dtype, mode, "Resize test vs interpolate, ",  pimg.size)

    new_size = [512, (512*pimg.size[0])//pimg.size[1]]
    resample = {"bilinear":PIL.Image.BILINEAR, "bicubic":PIL.Image.BICUBIC}

    pimg_sz = (np.array(pimg.resize(size=new_size[::-1], resample=resample[mode]))/255).astype(dtype)
    ptensor = interpolate(tensor, size=new_size, mode=mode, align_corners=False)
    
sig(ptensor, msg="  interpolate() ")
    sig(pimg_sz, msg="  Image.resize()")
    if show:
        ntensor = ptensor[0].numpy().transpose(1, 2, 0)
        diff = ntensor - pimg_sz
        diff = (diff - diff.min())/(diff.max() - diff.min())
        plt.figure(figsize=(18, 7))
        # plt.subplot(131)
        # plt.imshow(pimg_sz)
        # plt.subplot(132)
        # plt.imshow(ntensor)
        # plt.subplot(133)
        plt.imshow(diff)
        plt.tight_layout()
        plt.show()
if __name__ == "__main__":
    test_dif("float64", "bicubic")
    test_dif("float32", "bicubic")
    test_dif("float64", "bilinear")
    test_dif("float32", "bilinear", show=True)
</code></pre>
<p>Result</p>
<pre><code class="lang-auto">float64 bicubic Resize test vs interpolate,  (976, 549)
  interpolate()         min: -0.0877, max: 1.0536 mean: 0.3292, std: 0.2270     shape: (1, 3, 512, 910), dtype: torch.float64
  Image.resize()        min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2266      shape: (512, 910, 3), dtype: float64
float32 bicubic Resize test vs interpolate,  (976, 549)
  interpolate()         min: -0.0877, max: 1.0536 mean: 0.3292, std: 0.2270     shape: (1, 3, 512, 910), dtype: torch.float32
  Image.resize()        min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2266      shape: (512, 910, 3), dtype: float32
float64 bilinear Resize test vs interpolate,  (976, 549)
  interpolate()         min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2261      shape: (1, 3, 512, 910), dtype: torch.float64
  Image.resize()        min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2260      shape: (512, 910, 3), dtype: float64
float32 bilinear Resize test vs interpolate,  (976, 549)
  interpolate()         min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2261      shape: (1, 3, 512, 910), dtype: torch.float32
  Image.resize()        min: 0.0000, max: 1.0000 mean: 0.3292, std: 0.2260      shape: (512, 910, 3), dtype: float32
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97eb27968f3ed39c05181c14c9b698ea73a5030a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/97eb27968f3ed39c05181c14c9b698ea73a5030a" title="difpytorch_pil"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97eb27968f3ed39c05181c14c9b698ea73a5030a_2_690x268.jpeg" alt="difpytorch_pil" data-base62-sha1="lFVUHsLWLC5JzwxOvPmZznqUsum" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97eb27968f3ed39c05181c14c9b698ea73a5030a_2_690x268.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97eb27968f3ed39c05181c14c9b698ea73a5030a_2_1035x402.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97eb27968f3ed39c05181c14c9b698ea73a5030a_2_1380x536.jpeg 2x" data-dominant-color="B3B3B3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">difpytorch_pil</span><span class="informations">1800×700 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

xvdp: <p>ignore the comment on bicubic, interpolate. I notice, I have to clamp it.</p>

ptrblck: <aside class="quote no-group" data-username="xvdp" data-post="1" data-topic="106684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xvdp/48/11617_2.png" class="avatar"> xvdp:</div>
<blockquote>
<p>a while ago I built myself a translation of the torchvision.transforms library that operates over tensors.</p>
</blockquote>
</aside>
<p>That sounds cool, but note that <code>torchvision</code> implements the transformations now for tensors step by step, so you might consider using it directly or contribute instead to it.</p>
<p>E.g. <code>resize</code> seems to work on tensors already:</p>
<pre data-code-wrap="python"><code class="lang-python">import torchvision.transforms.functional as TF

out = TF.resize(torch.randn(3, 24, 24), (12, 12))
print(out.shape)
&gt; torch.Size([3, 12, 12])
</code></pre>

xvdp: <p>thank you ptrblk! Im not activelly developing my transformations - I just happen to use them,  when I noticed  slightly odd benchmarking numbers<br>
But Ill look at the stack and see if theres any I can contribute - to the project. Appreciated!</p>

xvdp: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, for reference, just to see if anything changed, I installed current torchvision, mod’d my script to</p>
<pre><code class="lang-python">from torchvision.transforms.functional import resize
ttensor = resize(img=tensor, size=new_size, interpolation=torchvision.transforms.InterpolationMode.BILINEAR)
diff = ttensor.numpy().transpose(1, 2, 0) - pimg_sz
#...
</code></pre>
<p>it does seem that resize uses nn.interpolate just like i did, or same resampling algo - so the cat still has the shadow. It isnt important as these bilinear mode isnt lossless.</p>
<p>Nice to see functional.affine() !</p>

xvdp: <p>Just to close the discussion with a bit of info.<br>
i tried, torchvision.resize(), torch.nn.interpolate(), PIL.Image.resize(), cv2.resize(), and tensorflow.image.resize()</p>
<p>torchvision.resize() and torch.nn.interpolate() return image diff of 0.<br>
torch 1.5 and tensorflow 2.2 have a slight differetnce(image attachhed)<br>
torch 1.8 and tensorflow 2.2 are identical -</p>
<p>all others are different from each other! curiously, since models tend to learn surface, they will all return slightly different benchmarks. Which is what led me to look at this.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dac880be0c6bdde3108c26713ace06080cec289.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dac880be0c6bdde3108c26713ace06080cec289" title="ResizeDiffs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dac880be0c6bdde3108c26713ace06080cec289_2_666x500.png" alt="ResizeDiffs" data-base62-sha1="dmG3cuSuGwCI1kz6Z1MprImdL8B" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dac880be0c6bdde3108c26713ace06080cec289_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dac880be0c6bdde3108c26713ace06080cec289_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dac880be0c6bdde3108c26713ace06080cec289_2_1332x1000.png 2x" data-dominant-color="969696"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ResizeDiffs</span><span class="informations">3000×2250 707 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>gist: <a href="https://gist.github.com/xvdp/64618a603d9ed2bf460a252013062ef6" class="inline-onebox" rel="noopener nofollow ugc">cv2 PIL.Image torch and tensorflow have sligthly different interpolation algorithms; curious · GitHub</a></p>