Why image datasets need normalizing with means and stds specified like in transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])?

seberino: <p>Why image datasets need normalizing with means and standard deviations specified like in transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ?</p>
<p>Where did those numbers come from and why isn’t this done already in the datasets?</p>
<p>Chris</p>

ptrblck: <p>These stats are calculated from the ImageNet dataset, which was used to pretrain classification models in <code>torchvision</code>. You don’t need to use these stats and could also try to use the stats calculated from your custom dataset.</p>

seberino: <p>ptrblck</p>
<p>Thanks. But aren’t we only concerned with getting the values between 0 and 1?<br>
To get the values between 0 and 1 we just need something like a min-max normalization right?<br>
Why do we <em>care</em> what the mean and standard deviation are?</p>
<p>Even the dataset in torchvision.datasets.CIFAR10 apparently needs a similar step with<br>
data_means = [0.4914, 0.4822, 0.4465]<br>
data_stds = [0.2023, 0.1994, 0.2010]<br>
torchvision.transforms.Normalize(mean = data_means, std = data_stds)</p>

Mah_Neh: <p>I’ll only reply the first question: no, we are not concerned with values between 0 and 1.</p>
<p>0-255 pixel values can blow up the gradients and the model wont converge, because it oscillates between large values.</p>
<p>We could imagine concatenating the red channel of all images in this single array <code>[0,1,45,250,...]</code></p>
<p>By normalising we make sure that the new array has <code>mean=0</code> and a <code>std=1</code>, which is more or less to say that we centre it. If you plot it now, the values in the red channel would be equally distributed around 0 (average value is 0), and most values won’t be too large.</p>
<p>I can not just justify why would it matter to centre rather than just <code>.div(255)</code> but it seems a common practice.</p>
<p>I think what was suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a> is a good idea, and common practice i.e to normalise from your dataset; I sometimes use <code>.div(255)</code> and haven’t had real issues.</p>
<p>Anyways, this is my interpretation, I am also learning.</p>
<p>You may benefit <a href="https://inside-machinelearning.com/en/why-and-how-to-normalize-data-object-detection-on-image-in-pytorch-part-1/" rel="noopener nofollow ugc">from his post</a>.</p>

ptrblck: <p><code>Normalize</code> will create outputs with a zero-mean and unit-variance as <a class="mention" href="/u/mah_neh">@Mah_Neh</a> also explained. This normalization is also sometimes called standardization, z-scoring, or whitening.<br>
In ML literature you can often read that whitening a signal is beneficial for the convergence with some theoretical analysis. E.g. I’m quite sure Bishop explains it also in his fantastic “Pattern Recognition and Machine Learning” book.<br>
This approach is thus also used commonly for deep learning training. Normalizing the values to the range of <code>[0, 1]</code> could also work and you could test if your use case would see a difference between <code>[0, 1]</code> values and standardized values.</p>

seberino: <p>I’ve noticed difference numbers for the CIFAR10 means and stds.</p>
<p>For example, in the PyTorch docs they use 0.5 everywhere!?</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<pre><code class="lang-auto">transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)
</code></pre>

ptrblck: <p>I guess nobody calculated these stats for CIFAR10 or the calculated stats didn’t yield any significant improvement over the default <code>0.5</code>.</p>

seberino: <p>ptrblck</p>
<p>We normalize training data the way we think best for training. However, when a model<br>
is used in production for inferences, the real testing data is <em>not</em> normalized!? Why<br>
isn’t that a problem? Didn’t the training phase create a model that now expects a certain<br>
normalization?</p>
<p>Chris</p>

Mah_Neh: <p>We have to normalise before inference, as a pre-processing step.</p>

seberino: <p>Thanks.  I really appreciate it.</p>
<p>cs</p>