How to use RGBA or 4 channel images in a DCGAN model in pytorch

GERMAIN_NICOLAS_MORA: <p>Hello, many thanks to all contributors in advance.</p>
<p>I am trying to apply pytorch’s DCGAN model however when I try to run the dataset it defaults to 3 channel images even though my images are 4 channel RGBA.</p>
<p>How to use RGBA or 4 channel images in a DCGAN model in pytorch?</p>
<pre><code class="lang-auto">
dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
# Create the dataloader
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)

</code></pre>
<p>This is the code I use to generate the dataset, I would like to modify it to be able to use my images with format [647, 4, 256,256]</p>

ptrblck: <p>You would need to modify the first conv layer of the model which I guess would currently accept tensors with 3 channels.<br>
However, are you sure the alpha channel contains any (useful) information and is not just set to the highest value for all pixels? In the latter case, I would claim you can directly remove this channel as it doesn’t contain any information.</p>

GERMAIN_NICOLAS_MORA: <p>When executing my code without any correction, I get the following error</p>
<pre><code class="lang-auto">Given groups=1, weight of size [32, 4, 4, 4], expected input[1, 3, 256, 256] to have 4 channels, but got 3 channels instead 
</code></pre>
<p>This is because dataset.dset.ImageFolder is converting the images to 3 channels, how can I make it read all four channels?</p>
<p>The information of the alpha channel is of supreme importance since the images are physical representations of the axes X, Y, Z, Z’ of a physical magnitude.</p>
<p>Thank you very much for taking the time to respond!</p>

ptrblck: <p>You could provide a custom <code>loader</code> to <code>ImageFolder</code> as the default would transform the images to RGB as seen <a href="https://github.com/pytorch/vision/blob/96aa3d928c6faaf39defd846966421679244412d/torchvision/datasets/folder.py#L244-L248">here</a>.</p>

Faezeh_Bidjarchian: <p>Hi,<br>
As you know, we can use only 1 and 3 channels in Keras, etc. But I just realized that<br>
we can use the desired channels in Pytorch based on this link:</p><aside class="quote" data-post="7" data-topic="68171">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vahid_sadeghi/48/16430_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/number-of-input-channels-in-pytorch/68171/7">Number of input channels in pytorch</a> 
  </div>
  <blockquote>
    thanks for your help 
may you explain more about how can I fuse 16 images to one image? 
if I shape an array with 16 channels does it work in pyotch as input? 
does pytorch give image as input or can I input array of numbers in the train and test folders? 
thanks in advance
  </blockquote>
</aside>
<p>
“You can write a custom dataset class for reading images and make them <strong>16</strong> channels input.<br>
Check [link] for writing a custom dataset.”<br>
But according to what you wrote here, finally “ImageFolder” converts the input to RGB channels?<br>
def pil_loader(path: str) → Image.Image:<br>
with open(path, “rb”) as f:<br>
img = Image.open(f)<br>
return img.convert(“<strong>RGB</strong>”)<br>
Is that correct?</p>

ptrblck: <p>A custom <code>Dataset</code> allows you to load and process the data the way you want. It would come down to loading an image with 16 channels as this it not a standard image format. However, if you have stored these samples as binary data you could just load them.</p>

Faezeh_Bidjarchian: <p>Many thanks for answering.<br>
I have another question. I could load them and the conv layer can deal with these images? or this binary image converts to an RGB format?<br>
In Keras, we have to use 1 or 3 channels. There is not this constraint for Pytorch? We can load images with any formats and with any number of channels?</p>

ptrblck: <aside class="quote no-group" data-username="Faezeh_Bidjarchian" data-post="7" data-topic="158252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/faezeh_bidjarchian/48/64134_2.png" class="avatar"> Faezeh_Bidjarchian:</div>
<blockquote>
<p>and the conv layer can deal with these images?</p>
</blockquote>
</aside>
<p>You are creating the conv layer and as long as you specify <code>in_channels=16</code> in the first conv layer it will be able to process inputs with 16 channels.</p>
<aside class="quote no-group" data-username="Faezeh_Bidjarchian" data-post="7" data-topic="158252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/faezeh_bidjarchian/48/64134_2.png" class="avatar"> Faezeh_Bidjarchian:</div>
<blockquote>
<p>We can load images with any formats and with any number of channels?</p>
</blockquote>
</aside>
<p>Again, you can create a custom <code>Dataset</code> and load whatever you want. Samples with 16 channels are not images, so PIL or OpenCV won’t be able to load these samples. However, as long as you can load the sample somehow inside the <code>Dataset</code> you will be able to use it also to train your model.</p>

Faezeh_Bidjarchian: <p>Oh. Finally, I could understand your meaning! The main challenge is the load of the images! If we can load them, we can train the networks with them! Is that correct?<br>
In the conv layers, Pytorch can consider weights for any number of channels! So, the main challenge is that we must load images in the format of gray, RGB, or a tensor (In PyTorch we deal with tensors)? For example, if I have FITS images I can not train my image in Pytorch. Is that correct? Because this is an unknown format in Pytorch!</p>

ptrblck: <p>PyTorch doesn’t know anything about image formats and just expects tensors.<br>
If you can load your 16 channels using any library and transform it somehow to a tensor (e.g. from numpy arrays) it will work.<br>
What PyTorch cares about is:</p>
<pre><code class="lang-python">input = torch.randn(1, 16, 224, 224)
first_conv = nn.Conv2d(16, 32, 3, 1, 1)

out = first_conv(input)
</code></pre>
<p>You have to make sure <code>input</code> is a tensor with 16 channels.</p>

Faezeh_Bidjarchian: <p>Many thanks for your guidance and time.</p>