Caltech256 containing grayscale images

Ricu: <p>I was trying to use the Caltech256 dataset to finetune the resnet 50 model. The images contained in the dataset are supposed to be 3 channel rgb images. However, when applying transformations, I got an error because some pictures only contained one channel.</p>
<pre><code class="lang-auto">from torchvision import models, datasets

resnet_weights = models.ResNet50_Weights.DEFAULT
caltech256 = datasets.Caltech256(
    root="data",
    download=True,
    transform=resnet_weights.transforms()
)

for i in range(len(caltech256)):
  print(i)
  print(caltech256[i][0].shape)
</code></pre>
<p>Running this code returns the error</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)

&lt;ipython-input-17-ed84e6e9f7b9&gt; in &lt;module&gt;
      1 for i in range(len(caltech256)):
      2   print(i)
----&gt; 3   print(caltech256[i][0].shape)

4 frames

/usr/local/lib/python3.9/dist-packages/torchvision/datasets/caltech.py in __getitem__(self, index)
    211 
    212         if self.transform is not None:
--&gt; 213             img = self.transform(img)
    214 
    215         if self.target_transform is not None:

/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.9/dist-packages/torchvision/transforms/_presets.py in forward(self, img)
     59             img = F.pil_to_tensor(img)
     60         img = F.convert_image_dtype(img, torch.float)
---&gt; 61         img = F.normalize(img, mean=self.mean, std=self.std)
     62         return img
     63 

/usr/local/lib/python3.9/dist-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    358         raise TypeError(f"img should be Tensor Image. Got {type(tensor)}")
    359 
--&gt; 360     return F_t.normalize(tensor, mean=mean, std=std, inplace=inplace)
    361 
    362 

/usr/local/lib/python3.9/dist-packages/torchvision/transforms/functional_tensor.py in normalize(tensor, mean, std, inplace)
    938     if std.ndim == 1:
    939         std = std.view(-1, 1, 1)
--&gt; 940     return tensor.sub_(mean).div_(std)
    941 
    942 

RuntimeError: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]
</code></pre>
<p>From the print statements during execution it can be seen, that it break for i=15.</p>
<p>Checking this without the normalization it can be seen that the sample for i=15 contains only one channel:</p>
<pre><code class="lang-auto">from torchvision import models, datasets, transforms

resnet_weights = models.ResNet50_Weights.DEFAULT
caltech256 = datasets.Caltech256(
    root="data",
    download=True,
    transform=transforms.ToTensor()
)
print(caltech256[15][0].shape)
</code></pre>
<p>Is there something I am missing? How should one deal with that problem?</p>

ptrblck: <p>Based on <a href="https://www.imageprocessingplace.com/root_files_V3/image_databases.htm">this overview</a> it seems both image types are expected:</p>
<blockquote>
<p>Caltech256 dataset. (RGB and grayscale images of various sizes in 256 categories for a total of 30608 images).</p>
</blockquote>
<p>However, I don’t see an easy way to convert the images to RGB besides deriving a custom <code>Dataset</code> and calling <code>img = img.convert("RGB")</code> after <a href="https://github.com/pytorch/vision/blob/6ca9c76adb6daf2695d603ad623a9cf1c4f4806f/torchvision/datasets/caltech.py#L201-L208">this <code>Image.Open</code></a> call.</p>

Ricu: <p>Thanks for your reply. I did not know that mixing RGB and grayscale image was a viable option in training models.</p>
<p>I will therefore look into creating a custom dataset.</p>

zuliani99: <p>An other possible solution is to replicate the grey channel three times in order to create a new tensor with dimension [3, w, h]. In my case I have to compute the mean and std in order to normalize the random split, and I’ve done like it follow:</p>
<pre><code class="lang-auto">def init_caltech256():
    if len(cls_datasets['caltech256']['train_idxs']) == 0 and len(cls_datasets['caltech256']['test_idxs']) == 0:
        
        download_caltech256()
        
        logger.info(' =&gt; Computing mean and std for a random train test split...')
        
        rand_perm = torch.randperm(cls_datasets['caltech256']['n_images'])
        cls_datasets['caltech256']['train_idxs'] = rand_perm[:10000].tolist()
        cls_datasets['caltech256']['test_idxs'] = rand_perm[:10000].tolist()
                
        train_data = datasets.Caltech256('./datasets/', download=False, transform=v2.Compose([v2.Resize((224,224))]))
        
        images = []
        for i in cls_datasets['caltech256']['train_idxs']:
            if len(np.asarray(train_data[i][0]).shape) == 3: images.append(np.asarray(train_data[i][0]))
            else: images.append(np.repeat(np.asarray(train_data[i][0])[:, :, np.newaxis], 3, axis=2))
                
        images = np.concatenate(images)
                
        mean = np.mean(images, axis=(0, 1)) / 255 
        std = np.std(images, axis=(0, 1)) / 255
                
        cls_datasets['caltech256']['transforms']['train'].append(v2.Normalize(mean=mean, std=std))
        cls_datasets['caltech256']['transforms']['train'] = v2.Compose(cls_datasets['caltech256']['transforms']['train'])
                
        cls_datasets['caltech256']['transforms']['test'].append(v2.Normalize(mean=mean, std=std))
        cls_datasets['caltech256']['transforms']['test'] = v2.Compose(cls_datasets['caltech256']['transforms']['test'])

        logger.info(' DONE\n')
</code></pre>