MNIST iterator not working after normalization

Torcione: <p>After defining the transformation</p>
<pre><code class="lang-auto">            transform = transforms.Compose([
                        transforms.ToTensor(), #NOTE: Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]
                        transforms.Normalize((DataMean,), (DataStd,))]) 
</code></pre>
<p>where <code>DataMean</code>  <code>DataStd</code> are mean and std computed over the train data<br>
I define the train dataset</p>
<pre><code class="lang-auto">            train_data = datasets.MNIST(root = ./data, train = True, download = True, transform = transform)

</code></pre>
<p>and build the dataloader (for my problem I need a separate dataloader for each class “i”)</p>
<pre><code class="lang-auto">TrainDL[i] = torch.utils.data.DataLoader(train_data, batch_size = 100, 
                                                       sampler = train_sampler, num_workers = 0)     

</code></pre>
<p>Later in the code I want to iterate over the dataloaders, namely</p>
<pre><code class="lang-auto">        for EvalKey in TrainDL:
            for dataval,labelval in TrainDL[EvalKey]:
</code></pre>
<p>Here I get the following error</p>
<blockquote>
<p>ValueError: Too many dimensions: 3&gt;2</p>
</blockquote>
<p>I read <a href="https://discuss.pytorch.org/t/mnist-dataloader-return-an-extra-iten-for-eg-mean-pixel-value-of-the-image/48954">here</a> that this is due to the Normalization transformation. Is not clear to me why the third dimension should cause this problem (CIFAR10 also has 3 channels and works perfectly fine). Is it due to a difference between the expected dimension of get item and thart you got after initialization? what is the easiest way to solve the problem?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="Torcione" data-post="1" data-topic="180106">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/torcione/48/52091_2.png" class="avatar"> Torcione:</div>
<blockquote>
<p>Is not clear to me why the third channel should cause this problem (CIFAR10 also has 3 channels and works perfectly fine).</p>
</blockquote>
</aside>
<p><code>MNIST</code> images have a single channel only so normalizing these samples with 3 stats values will fail.</p>

Torcione: <p>sorry I meant 3 dimensions (I corrected my question), referring to ,your answer from the post in the link, i.e.:</p>
<blockquote>
<p><code>img</code> will have 3 dimensions (<code>[1, 28, 28]</code>), which will yield this error.</p>
</blockquote>
<p>In my case for the normalization I defined two functions to compute directly mean and std on the MNIST dataset ( DataMean):</p>
<pre><code class="lang-auto">        elif (self.DatasetName == 'MNIST'):
            imgs = [item[0] for item in self.train_data]  # item[0] and item[1] are image and its label
            imgs = torch.stack(imgs, dim=0).numpy()
            
            # calculate mean 
            mean = imgs[:,:,:].mean()

            return mean    
</code></pre>
<p>and std ( DataStd):</p>
<pre><code class="lang-auto">        elif (self.DatasetName == 'MNIST'):
            imgs = [item[0] for item in self.train_data] # item[0] and item[1] are image and its label
            imgs = torch.stack(imgs, dim=0).numpy()
            
            # calculate std
            std = imgs[:,:,:].std()
 
            
            return(std)            
            
</code></pre>
<p>What am I doing wrong? How could I fx it?</p>

ptrblck: <p>I don’t know as your code is not executable, but this works for me:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(root="~/python/data", download=False)

imgs = dataset.train_data
            
# calculate mean 
mean = imgs.float().mean()
std = imgs.float().std()

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=(mean,), std=(std,)),
])
dataset = datasets.MNIST(root="~/python/data", download=False, transform=transform)
loader = DataLoader(dataset, batch_size=2)

for data, target in loader:
    print(data.shape)
    print(target)
</code></pre>

Torcione: <p>Thank you for the answer!</p>
<p>The first time you define the dataset (for the mean and std computation) shouldn’t you include</p>
<pre><code class="lang-auto">transform = transforms.ToTensor()
</code></pre>
<p>in order to compute the two quantity on the same scale that we use later for the “real” dataset ([0,1])?</p>

ptrblck: <p>Yes, it’s indeed missing and you could add the transformation but would then need to iterate the dataset.<br>
Alternatively, you could also normalize the already calculated <code>mean</code> and <code>std</code>.</p>

Torcione: <p>Why would I need to iterate the dataset in this case?</p>
<p>Thank you a lot for your answer, it make me realize that probably the problem, in my case, is not in the normalization step but probably in the moment I define different dataloaders for different classes, splitting the dataset. I’m trying to understand where exactly the problem arise as using cifar10 (with exactly the same splitting logic) I don’t get any error</p>

ptrblck: <p>Because the internal <code>.train_data</code> won’t be directly transformed.<br>
Each sample will be transformed in the <code>__getitem__</code> on-the-fly using the passed transformation while the internal <code>.train_data</code> attribute will still hold the original data.</p>

Torcione: <p>I see! so for cifar10 this does not represent a problem as the dimension keep the same before and after transformation, while for MNIST I got an extra channel dimension, right?</p>
<p>But why is this the case only when I split the dataset on multiple dataloader and not when I keep a single dataloader for the whole dataset?</p>

ptrblck: <p>It’s still unclear what exactly is failing in your code as my code snippets work as you can see.<br>
The <code>MNIST</code> dataset has a single channel and <code>Normalize</code> thus expects stats with a single value, too.</p>

Torcione: <p>Thank you for the explanation; as I understood that the problem may not be in the normalization step I created a more detailed post where I provide the whole pipeline (<a href="https://discuss.pytorch.org/t/code-extention-from-cifar10-to-mnist-not-working/180371">Code extention from CIFAR10 to MNIST not working</a>)</p>