Using torchvision.transforms with numpy arrays

sequoia19: <p>Hello,</p>
<p>I am trying to perform transformations using torchvision.transforms (specifically <code>transforms.RandomAdjustSharpness</code>) on images that are currently stored as numpy arrays. I have experimented with many ways of doing this, but each seems to have its own issues.</p>
<p><strong>Method 1</strong>:  Converting numpy arrays to torch tensors, then applying transformation.<br>
The documentation for <code>RandomAdjustSharpness</code> says that the input image can be a tensor. However:</p>
<pre><code class="lang-auto">img = np.load('file.npy').newbyteorder().byteswap() # float32
img = torch.from_numpy(img)
transf = transforms.RandomAdjustSharpness(sharpness_factor=2, p=1)
img = transf(img)
</code></pre>
<p>results in:</p>
<pre><code class="lang-auto">File ~/miniconda3/envs/general/lib/python3.10/site-packages/torchvision/transforms/transforms.py:1961, in RandomAdjustSharpness.forward(self, img)
   1953 """
   1954 Args:
   1955     img (PIL Image or Tensor): Image to be sharpened.
   (...)
   1958     PIL Image or Tensor: Randomly sharpened image.
   1959 """
...
    828     ],
    829 )
    830 result_tmp = conv2d(result_tmp, kernel, groups=result_tmp.shape[-3])

IndexError: tuple index out of range
</code></pre>
<p><a href="https://stackoverflow.com/questions/68273819/randomadjustsharpness-gives-indexerror-tuple-index-out-of-range" rel="noopener nofollow ugc">This post</a> (which is also referenced <a href="https://discuss.pytorch.org/t/randomadjustsharpness-gives-indexerror-tuple-index-out-of-range/125974">here</a>) seems to imply that the transform asumes a batch dimension, so adding an additional dimension to the tensor should work. But their solution:</p>
<pre><code class="lang-auto">img = np.load('file.npy').newbyteorder().byteswap()
img = torch.from_numpy([img])
transf = transforms.RandomAdjustSharpness(sharpness_factor=2, p=1)
img = transf(img)
</code></pre>
<p>results in:</p>
<pre><code class="lang-auto">TypeError: expected np.ndarray (got list)
</code></pre>
<p>Using np.exand_dims “works” but produces a tensor of all ones (no matter the <code>sharpness_factor</code> I choose):</p>
<pre><code class="lang-auto">img = np.load('file.npy').newbyteorder().byteswap()
og_img = np.copy(img)
img = torch.from_numpy(np.expand_dims(img, axis=0))
transf = transforms.RandomAdjustSharpness(sharpness_factor=2, p=1)
img = transf(img)[0]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3))
im1 = ax1.imshow(og_img); plt.colorbar(im1, ax=ax1); ax1.set_title('original')
im2 = ax2.imshow(img); plt.colorbar(im2, ax=ax2); ax2.set_title('transformed')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80537e036ed3fd44af83c4f9ea0b22212b689b68.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/80537e036ed3fd44af83c4f9ea0b22212b689b68" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80537e036ed3fd44af83c4f9ea0b22212b689b68_2_690x259.jpeg" alt="image" data-base62-sha1="ijdZQs60ldIqsrSig4g5KaMQwnS" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80537e036ed3fd44af83c4f9ea0b22212b689b68_2_690x259.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80537e036ed3fd44af83c4f9ea0b22212b689b68_2_1035x388.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80537e036ed3fd44af83c4f9ea0b22212b689b68_2_1380x518.jpeg 2x" data-dominant-color="88BAB8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1519×571 84.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Method 2</strong>: Converting np array to PIL image, then applying transformation.<br>
Since I could not get this to work with a tensor, I tried to converting to a PIL image first (as the documentation says PIL images should also work). I am converting to PIL using <code>transforms.ToPILImage</code>:</p>
<pre><code class="lang-auto">img = np.load('file.npy').newbyteorder().byteswap()
transf = transforms.ToPILImage() 
img = transf(img)
</code></pre>
<p>Converting back to a numpy array with <code>img = np.squeeze(transforms.functional.pil_to_tensor(img).numpy())</code> does reproduce the image I expect</p>
<p>However, using <code>img.show()</code> on the PIL image results in an all whilte image, and trying to apply the sharpening transform to the PIL image does not work.</p>
<pre><code class="lang-auto">img = np.load('file.npy').newbyteorder().byteswap()
transf1 = transforms.ToPILImage() 
transf2 = transforms.RandomAdjustSharpness(sharpness_factor=1, p=1)
img = transf1(img)
img = transf2(img)
</code></pre>
<p>results in:</p>
<pre><code class="lang-auto">File ~/miniconda3/envs/general/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/general/lib/python3.10/site-packages/torchvision/transforms/transforms.py:1961, in RandomAdjustSharpness.forward(self, img)
   1953 """
...
     30 if image.mode == "P":
     31     raise ValueError("cannot filter palette images")
---&gt; 32 return image.filter(*self.filterargs)

ValueError: image has wrong mode
</code></pre>
<p>This error persists when I change <code>mode</code> in <code>ToPILImage().</code> Using <code>mode="F"</code> (since my numpy array dtype is <code>float32</code>) gives the same <code>image has wrong mode</code> error. Changing to any other mode (e.g. <code>I</code>) results in <code>ValueError: Incorrect mode (I) supplied for input type . Should be F</code>.</p>
<p>It seems like using transforms with numpy arrays should be a previously-solved problem, so I am wondering what the correct way to do this is. Thank you so much for your help!</p>

ptrblck: <p>Transforming the numpy array to a tensor works for me:</p>
<pre><code class="lang-python">img = np.random.uniform(0, 1, (3, 224, 224))
img = torch.from_numpy(img)
transf = transforms.RandomAdjustSharpness(sharpness_factor=2, p=1)
out = transf(img)

import matplotlib.pyplot as plt

plt.imshow(img.permute(1, 2, 0).numpy())
plt.imshow(out.permute(1, 2, 0).numpy())
</code></pre>

sequoia19: <p>Aha! The issue is just that the <code>sharpness_factor</code>  values listed in the <code>transforms.RandomAdjustSharpness</code> documentation assume the image is normalized to [0, 1]. I can thus either normalize the images or use appropriately scaled <code>sharpness_factor</code>  values to get 0x, 1x, 2x, etc increases in sharpness. Thanks!</p>