Computing the mean and std of dataset

kuzand: <p>Hello. So I am trying to compute the mean and the standard deviation per channel of my train dataset (three-channel images of different shapes).<br>
For the mean I can do it in two ways, but I get slightly different results.</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms

dataset = datasets.ImageFolder('train',
                 transform=transforms.ToTensor())
</code></pre>
<p>First computation:</p>
<pre><code class="lang-auto">mean = 0.0
for img, _ in dataset:
    #mean += img.sum([1,2])/torch.numel(img[0])
    mean += img.mean([1,2])
mean = mean/len(dataset)
print(mean)
# tensor([0.3749, 0.3992, 0.4505])
</code></pre>
<p>Second computation:</p>
<pre><code class="lang-auto">sumel = 0.0
countel = 0
for img, _ in dataset:
    sumel += img.sum([1, 2])
    countel += torch.numel(img[0])
mean = sumel/countel
print(mean)
# tensor([0.3802, 0.4003, 0.4513])
</code></pre>
<p>Any idea why there is this small difference in the two computations?</p>
<p>Similarly for the std</p>
<pre><code class="lang-auto">sumel = 0.0
countel = 0
for img, _ in dataset:
    img = (img - mean.unsqueeze(1).unsqueeze(1))**2
    sumel += img.sum([1, 2])
    countel += torch.numel(img[0])
std = torch.sqrt(sumel/countel)
</code></pre>
<p>Is it a correct way to compute it?</p>

kuzand: <p>Ok, I think this small discrepancy is due to some numerical issue (floating point error?).<br>
Anyway, the above method of computing mean and std is not efficient for big datasets. So it is better to use the dataloader, after first resizing the images.<br>
I found here <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560?u=kuzand" class="inline-onebox">About Normalization using pre-trained vgg16 networks</a> how compute the mean and std (thanks to <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a>),</p>
<pre><code class="lang-auto">loader = data.DataLoader(dataset,
                         batch_size=10,
                         num_workers=0,
                         shuffle=False)

mean = 0.
std = 0.
for images, _ in loader:
    batch_samples = images.size(0) # batch size (the last batch can have smaller size!)
    images = images.view(batch_samples, images.size(1), -1)
    mean += images.mean(2).sum(0)
    std += images.std(2).sum(0)

mean /= len(loader.dataset)
std /= len(loader.dataset)
</code></pre>
<p>However I have doubts for the correctness of the computation of the std. In the above code the std’s of all the images are summed and at the end they are averaged by the total number of images. But I think that the total std should be computed over all the pixel values of all the images in the dataset, as in my previous post.</p>

vmirly1: <p>I think in the other post by <a class="mention" href="/u/ptrblck">@ptrblck</a>, he is computing the mean and std over the pixels not over samples in the batch. So, then that code in <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560?u=kuzand">About Normalization using pre-trained vgg16 networks</a> is correct, since the goal is to compute the mean and std for each batch and then take the average of these two quantities over the entire dataset.</p>

kuzand: <p>What about this one:</p>
<pre><code class="lang-auto">dataset = datasets.ImageFolder('train', transform=transforms.Compose([transforms.Resize(256),
                             transforms.CenterCrop(224),
                             transforms.ToTensor()]))

loader = data.DataLoader(dataset,
                         batch_size=10,
                         num_workers=0,
                         shuffle=False)

mean = 0.0
for images, _ in loader:
    batch_samples = images.size(0) 
    images = images.view(batch_samples, images.size(1), -1)
    mean += images.mean(2).sum(0)
mean = mean / len(loader.dataset)

var = 0.0
for images, _ in loader:
    batch_samples = images.size(0)
    images = images.view(batch_samples, images.size(1), -1)
    var += ((images - mean.unsqueeze(1))**2).sum([0,2])
std = torch.sqrt(var / (len(loader.dataset)*224*224))
</code></pre>
<p>This also gives reasonable values for std, but different from <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> std.</p>

vmirly1: <p>yes! Because this is computing a different std. In the other code, the purpose was to compute the mean and std for each batch over the pixels, and then take the average of them. This is specifically applicable for <strong>Batch-Normalization</strong>.</p>
<p>However, the purpose of this code that you have posted is to compute the mean of the entire data and std over the entire data, which is different from batch-normalization.</p>

MariosOreo: <p>I have tried the method wrote by <a class="mention" href="/u/ptrblck">@ptrblck</a> (thanks to prtblck), it works but the cost on <strong>CPU</strong> is too much, more than 1200%, how can I change the code or settings that make it more efficient.</p>
<p>Thanks in advance. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

ptrblck: <p>Usually you would only compute it once on your dataset.<br>
Why do you think the CPU utilization is too high?</p>

jcowles: <p>This version should run much faster and compute the same result as std(), though you need to be careful about overflow:</p>
<pre><code class="lang-auto">  #
  # True standard deviation
  #

  loader = torch.utils.data.DataLoader(
      your_dataset,
      batch_size=10,
      num_workers=0,
      shuffle=False
  )

  mean = 0.
  meansq = 0.
  for data in loader:
      mean = data.mean()
      meansq = (data**2).mean()

  std = torch.sqrt(meansq - mean**2)
  print("mean: " + str(mean))
  print("std: " + str(std))
  print()
</code></pre>

JessicaDuFirst: <p>I tried your code, but one error raised:  mean=data.mean()<br>
AttributeError: ‘list’ object has no attribute ‘mean’</p>

IngoMarquart: <p>I mean, for the two codes you posted, each does different things.</p>
<p>The first is the mean over the means in each image.</p>
<p>The second is the mean over all images.</p>
<p>Why should they be the same?</p>

pete7.62: <p>If all images are of the same size and all batches are of the same size they’re mathematically equal. In OP’s case (three-channel images of different shapes) that’s of course not the case.</p>

thisismexp: <p>It might be a bit too late, but PIL provides nice functionality for your problem in the ImageStat.Stat class. Its calculations are based on the histogram of the images and therefore only need O(1) memory, but it only considers one image. In order to deal with more images, I extended the Stat class by introducing an <code>__add__</code> method which combines two histograms of the given objects (and therefore is a bit like concatenating two images and generate the Stat object out of them):</p>
<pre><code class="lang-auto">class Stats(ImageStat.Stat):
    def __add__(self, other):
        return Stats(list(map(add, self.h, other.h)))
</code></pre>
<p>The histogram is stored in h, both histograms (of self and other) are summed up and then a new Stat class is initialized with the new histogram instead of an image.</p>
<p>Using this new “Stats” class i could do something like:</p>
<pre><code class="lang-auto">loader = DataLoader(dataset, batch_size=10, num_workers=5)

statistics = None
for data in loader:
    for b in range(data.shape[0]):
        if statistics is None:
            statistics = Stats(tf.to_pil_image(data[b]))
        else:
            statistics += Stats(tf.to_pil_image(data[b]))
</code></pre>
<p>And from there on use normal Stat calls like:</p>
<pre><code class="lang-auto">print(f'mean:{statistics.mean}, std:{statistics.stddev}')
# mean:[199.59, 156.30, 170.59], std:[31.30, 31.28, 35.95]
</code></pre>
<p>Note that although this is quite a neat solution, it is by far not the most efficient.</p>

Hong_Cheng: <pre><code class="lang-auto">import tensorflow as tf
from PIL import ImageStat
</code></pre>
<p>This works for me, thanks, better and stable than other method I tried</p>

prateekgupta891: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/vmirly1">@vmirly1</a> I stilll have a doubt regarding normalization. I want to normalize my data, what approach should I take ?<br>
I dont want to resize my data, as image size varies from (20,80,3) to (253,80,3) it varies a lot.<br>
And I wanna try Resnet’s which uses batch normalization, do i need to a batch normalization?<br>
And again if I dont resize it, while using the code with the data loader, I get an error saying “invalid argument”.<br>
Your clarification and help would much appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

ptrblck: <p>Could you post the complete error message please?</p>

prateekgupta891: <pre><code class="lang-auto">#doesnt work without without batch size equal to 1.
dataset = datasets.ImageFolder(('train'),transform = transforms.ToTensor())
loader = torch.utils.data.DataLoader(dataset,
                                     batch_size = 10,
                                     num_workers = 0,
                                     shuffle = False)
mean = 0.0
for images,_ in loader:
  print(images.shape)
  batch_samples = images.size(0)
  
  images = images.view(batch_samples,images.size(1),-1)
  print(images.shape)
  mean += images.mean(2).sum(0)
  print(images.mean(2).sum(0))
  break
mean = mean/len(loader.dataset)

</code></pre>
<p>Error Message:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0e13ee67b23c90641ecc7fee3b016890a474fdf.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0e13ee67b23c90641ecc7fee3b016890a474fdf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0e13ee67b23c90641ecc7fee3b016890a474fdf.png" alt="image" data-base62-sha1="tNPTPI7uRauK9PEf8MLfLpvVx4r" width="690" height="232" data-dominant-color="F7F7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1201×404 22.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vmirly1: <p>I think there is no issue in the mean computation, but the problem is in the data-loader. You cannot put images of different sizes in a batch. So, you may want to try iterating thought the dataset with a batch_size=1</p>

caped-vigilante: <p>This is my solution:</p>
<pre><code class="lang-auto">mean = 0.0
meansq = 0.0
count = 0

for index, data in enumerate(train_loader):
    mean = data.sum()
    meansq = meansq + (data**2).sum()
    count += np.prod(data.shape)

total_mean = mean/count
total_var = (meansq/count) - (total_mean**2)
total_std = torch.sqrt(total_var)
print("mean: " + str(total_mean))
print("std: " + str(total_std))</code></pre>

l00p: <aside class="quote no-group" data-username="thisismexp" data-post="13" data-topic="34949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thisismexp/48/19761_2.png" class="avatar"> thisismexp:</div>
<blockquote>
<p><code>ImageStat.Stat</code></p>
</blockquote>
</aside>
<p>Gave this a go like this:</p>
<h1><a name="httpsdiscusspytorchorgtcomputing-the-mean-and-std-of-dataset3494912-1" class="anchor" href="#httpsdiscusspytorchorgtcomputing-the-mean-and-std-of-dataset3494912-1"></a><a href="https://discuss.pytorch.org/t/computing-the-mean-and-std-of-dataset/34949/12" class="inline-onebox">Computing the mean and std of dataset - #12 by pete7.62</a></h1>
<p>class Stats(PIL.ImageStat.Stat):<br>
def <strong>add</strong>(self, other):<br>
return Stats(list(map(add, self.h, other.h)))</p>
<pre><code class="lang-auto">loader = my_train_dataloader(batch_size=64, balance=False, return_path=False, verbose=False, augment=False, resize=False)

statistics = None
toPIL=transforms.ToPILImage()

print(PIL.__version__)

for data, _ in loader:
    for b in range(data.shape[0]):
        if statistics is None:
            print(type(toPIL(data[b])))
            statistics = Stats(toPIL(data[b]))
        else:
            statistics += Stats(toPIL(data[b]))
print(f'mean:{statistics.mean}, std:{statistics.stddev}')

# PIL version (pillow-simd)
6.2.2.post1
# confirming that I'm passing the right thing
&lt;class 'PIL.Image.Image'&gt;
</code></pre>
<p>but got this error:</p>
<pre><code class="lang-auto">&lt;ipython-input-22-53e0795c502a&gt; in __add__(self, other)
      2 class Stats(PIL.ImageStat.Stat):
      3     def __add__(self, other):
----&gt; 4         return Stats(list(map(add, self.h, other.h)))
      5 
      6 loader = my_train_dataloader(batch_size=64, balance=False, return_path=False, verbose=False, augment=False, resize=False)

NameError: name 'add' is not defined
</code></pre>

l00p: <p>probably your dataloader is returning (tensor, label) so you could modify it like:</p>
<pre><code class="lang-auto">for data, _ in loader:
</code></pre>