Is my calculation of normalisation correct

bing: <p>Hi,<br>
I am applying 0 Mean &amp; Std Dev-1 normalisation to my train and dev set but it is giving me poor metrics in comparison to without normalization after applying them.<br>
Am I calculating the mean &amp; Std dev correctly?</p>
<pre><code class="lang-auto">train_mean = []
train_std = []

for i,image in enumerate(tqdm(train_loader, 0)):
    numpy_image = image[0].numpy()
    batch_mean = np.mean(numpy_image, axis=(0, 2, 3))
    batch_std = np.std(numpy_image, axis=(0, 2, 3))
    train_mean.append(batch_mean)
    train_std.append(batch_std)  
train_mean = torch.tensor(np.mean(train_mean, axis=0))
train_std = torch.tensor(np.mean(train_std, axis=0))
</code></pre>
<p>My application of normalisation from the values obtained from above in the dataset class below -</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as TF
image = TF.normalize(image, [0.7259, 0.7320, 0.6922], [0.2350, 0.1569, 0.1024]) # noqa

</code></pre>

tom: <p>Strictly speaking, you are not supposed to aggregate the standard deviation that way.<br>
Is there a reason why the RGB channels have vastly different stds? This seems surprising, but of course, only you know your data.</p>
<p>Best regards</p>
<p>Thomas</p>

bing: <p>Data is pretty similar in terms of appearance, is the difference in standard deviation a lot?<br>
Am I calculating the std dev correctly?</p>

bing: <p>My bad, I was calculating mean&amp;std incorrectly.<br>
These are the new values with the updated code, are these correct or they still have problems -</p>
<blockquote>
<p>Mean: tensor([-1.7882, -3.5365, -5.7360])<br>
Std Dev: tensor([1.0839, 1.0510, 1.1245])</p>
</blockquote>
<pre><code class="lang-auto">var = 0.0
mean = 0.0
nimages = 0
for i,(images,_) in enumerate(tqdm(train_loader, 0)):
    batch_samples = images.size(0)
    images = images.view(batch_samples, images.size(1), -1)
    nimages+= images.size(0)
    mean += images.mean(2).sum(0) 
    var += images.var(2).sum(0)
mean /= nimages
var /= nimages
std = torch.sqrt(var)
</code></pre>

tom: <p>Can you say something about the images and your pipeline to load them?<br>
The common thing is to use PIL + ToTensor and then you’d get 0…1 images, which the statistics say you don’t have.<br>
I mentioned that before, but strictly speaking you cannot aggregate variance like that but should use <code>M2 = (images ** 2).mean(dim=(0, 2, 3))</code> or so and then subtract <code>mean**2</code> at the end. (see <a href="https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance" class="inline-onebox" rel="noopener nofollow ugc">Algorithms for calculating variance - Wikipedia</a> for more sophistication)</p>
<p>Best regards</p>
<p>Thomas</p>

bing: <p>These are medical images.<br>
Below is my <code>dataset class</code></p>
<pre><code>from PIL import Image
import torchvision.transforms.functional as TF

 def __init__(self, list_IDs, labels, root_dir, train_transform=False, valid_transform=False, test_transform=True): # noqa
    """To instantiate labels and the data distribution."""
    self.labels = labels
    self.list_IDs = list_IDs
    self.dir = root_dir
    self.race_folder = glob.glob(os.path.join(root_dir, '*.JPG'))
    self.train_transform = train_transform
    self.valid_transform = valid_transform
    self.test_transform = test_transform

def train_transforms(self, image):
    """To train transformation."""
    image = TF.to_tensor(image)
    image = TF.normalize(image, [-1.7882, -3.5365, -5.7360],[1.0839, 1.0510, 1.1245]) 

def __getitem__(self, index):
    """Generate one sample of data."""
    ID = self.list_IDs[index]
    image = Image.open(os.path.join(self.dir, ID))
    y = self.labels[ID]
    if self.train_transform:
        image = self.train_transforms(image)
    elif self.valid_transform:
        image = self.valid_transforms(image)
    else:
        image = self.test_transforms(image)
    img = np.array(image)
    return img, y
</code></pre>

tom: <p>Image is PIL? What kind of transforms are you using?</p>

bing: <p>Yes, I imported <code>Image</code> from <code>PIL</code>.<br>
Transforms are just <code>to_tensor</code> and then <code>normalise</code>. You could see it in the <code>train_transform</code> function above.</p>

tom: <p>But then something is fishy with the averages you get - to_tensor should give you tensors with values 0…1, so you cannot possibly get negative means or std/var &gt; 1.<br>
Be extra sure that your normalization gets the same images (maybe up to random) as your training.</p>
<p>Best regards</p>
<p>Thomas</p>

bing: <p>Yesss, you were right. I was calculating the normalisation values twice.<br>
Below are my new mean and std_dev values, do you think these are correct, Also how you are able to see that values are not correct, just by looking at them?</p>
<pre><code class="lang-auto">Mean: tensor([0.3057, 0.1771, 0.1048])
Std Dev: tensor(0.2547, 0.1649, 0.1152])
</code></pre>
<p>Thanks.</p>

tom: <p>I think these the new values are plausible. The red is quite a bit stronger than green and blue, which would be funny for “natural images”, but of course the medical images will look differently. Do the images look reddish when you open them in an image viewer?</p>
<p>Best regards</p>
<p>Thomas</p>

bing: <p>Yes, those are fundus images which are red in colour. I’ll also fix the <code>std dev</code> value as you suggested above, I still quite didn’t understand it but I’ll figure it out.<br>
Thanks.</p>