Torchvision transformation arbitrary channels

username: zeneofa
cooked: <p>Hi, I have “images” with a large number of channels and there are some transformations that I would like to apply to these. Specifically I would like to perform rotations, cropping etc. These are all implemented in torchvision, however these require PIL images as input. This is problem, as PIL images require a specific number range of channels. Is there a way to support an arbitrary number of channels?</p>
<p>If not, is there an efficient way to take a tensor, and for each channel convert it to a PIL image, perform my transformations, convert back to tensors and stack along the channel dimension? Kinda like a along-a-dimension map-reduce function (that is hopefully efficient, and in and of itself would be endlessly useful for other applications that I am working on).</p>
<p>Ideally I would like to not rely on PIL at all.</p>

username: crcrpar
cooked: <p>Hi,</p>
<p>I think what you want to do can be realized by writing a loader function able to load your multi-channel images and convert them into numpy.ndarrays because <a href="https://github.com/pytorch/vision/blob/master/torchvision/transforms/transforms.py#L61" rel="nofollow noopener">torchvision.transforms.ToTensor</a> can handle PIL.Image and numpy.ndarrays.</p>

username: himat
cooked: <p>How does this solve the problem?<br>
Transforms like <code>torchvision.transforms.RandomHorizontalFlip</code> need to take in a PIL image, so using <code>ToTensor</code> doesn’t make sense.</p>
<p><a class="mention" href="/u/zeneofa">@zeneofa</a> were you able to figure out a solution?</p>

username: thiagodma
cooked: <p>I managed to solve it in two ways:</p>
<ol>
<li>Apply the Pytorch’s transforms channel by channel:</li>
</ol>
<pre><code class="lang-auto">tfms = transforms.Compose([transforms.Resize(224),
                           transforms.RandomHorizontalFlip(),
                           transforms.RandomVerticalFlip(),
                           transforms.ToTensor()])

x=[]
#set a seed so the same transforms are applied to each channel
seed = np.random.randint(2147483647)
for ch in img:
    random.seed(seed)
    x.append(tfms(Image.fromarray(ch)))

#this is the multichannel transformed image (a torch tensor)
img_tfm = torch.cat(x)
</code></pre>
<ol start="2">
<li>Use fastai transforms. I think this the best approach because it is really easy and ready to go:</li>
</ol>
<pre><code class="lang-auto">#use fastai's transforms
tfms = get_transforms(flip_vert=True,max_warp=None,max_rotate=45,max_lighting=None,max_zoom=1.1)

#img is a torch tensor (how many channels you need)
fastai_img = fastai.vision.Image(img)
            
#apply the transforms to the image
for tfm in tfms:
    fastai_img = fastai_img.apply_tfms(tfm,size=224)

#this is the multichannel transformed image (a torch tensor)
img_tfm = fastai_img.data
</code></pre>

username: omerbart
cooked: <aside class="quote no-group" data-username="thiagodma" data-post="4" data-topic="18631">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thiagodma/48/24110_2.png" class="avatar"> thiagodma:</div>
<blockquote>
<pre><code class="lang-auto">seed = np.random.randint(2147483647)
for ch in img:
    random.seed(seed)
    x.append(tfms(Image.fromarray(ch)))
</code></pre>
</blockquote>
</aside>
<p>Actually you should use torch.manual_seed and not python random.seed, because the transforms functions use torch.rand(1) to generate a random number and you may get different results for different channels.</p>

username: Siddharth_Shrivasta2
cooked: <p>We need to use both the random seed expressions:</p>
<pre><code class="lang-auto">random.seed(seed) 
torch.manual_seed(seed)
</code></pre>
<p>For details: <a href="https://github.com/pytorch/pytorch/issues/42331" class="inline-onebox" rel="noopener nofollow ugc">The random seed for torchvision transforms is not fixed in PyTorch 1.6 · Issue #42331 · pytorch/pytorch · GitHub</a></p>