Getting negative mean and std for a subset of my dataset

Mona_Jalal: <p>I have created a subset of my data (a bit smaller) and now I have negative values in my std and mean tensors:</p>
<p>This subset is designed in a way that I would have same number of positive and negative classes.</p>
<pre><code class="lang-auto">train mean and std: tensor([0.0050, 0.0225, 0.0250]) tensor([0.9833, 0.9977, 0.9932])
val mean and std: tensor([-0.0584, -0.0225, -0.0385]) tensor([1.0157, 1.0408, 1.0215])
test mean and std: tensor([-0.1491, -0.0664, -0.0715]) tensor([1.0436, 1.0221, 1.0180])
</code></pre>
<p>Is any part of the code below wrong?</p>
<pre><code class="lang-auto"># get the mean var std of train, test and val set for data transform
def get_mean_std(loader):
    # VAR[X] = E[X**2] - E[X]**2
    channels_sum, channels_squared_sum, num_batches = 0, 0, 0
    for data, _ in loader:
        channels_sum += torch.mean(data, dim=[0,2,3])
        channels_squared_sum += torch.mean(data**2, dim=[0,2,3])
        num_batches += 1
    
    mean = channels_sum/num_batches
    std = (channels_squared_sum/num_batches - mean**2)**0.5
    return mean, std

train_mean, train_std = get_mean_std(dataloaders_dict['train'])
print(train_mean, train_std)
test_mean, test_std = get_mean_std(dataloaders_dict['test'])
print(test_mean, test_std)
val_mean, val_std = get_mean_std(dataloaders_dict['val'])
print(val_mean, val_std)
</code></pre>
<p>Here are the values I had for the larget set of my dataset:</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(input_size),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7031, 0.5487, 0.6750], [0.2115, 0.2581, 0.1952])
    ]),
    'val': transforms.Compose([
        transforms.Resize(input_size),
        transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7016, 0.5549, 0.6784], [0.2099, 0.2583, 0.1998])
    ]),
    
    'test': transforms.Compose([
        transforms.Resize(input_size),
        transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7048, 0.5509, 0.6763], [0.2111, 0.2576, 0.1979])
    ])
}
</code></pre>
<p>I also understand that I should only use the train mean and std also for val and test and don’t calculate it for those since it would leak into val and test based on what <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned in a previous post. But I don’t understand how it happens. Is there any scientific paper talking about this phenomena?</p>

Mona_Jalal: <p>I fixed this problem and I set it as solution. However, my question on <a class="mention" href="/u/ptrblck">@ptrblck</a> comment is still present. I would like to learn mathematically or with a proof why we want to use the mean and std value of train set for normalization of val and test set too. Any feedback is really appreciated</p>
<pre><code class="lang-auto">train mean and std: tensor([0.7057, 0.5569, 0.6816]) tensor([0.2047, 0.2554, 0.1914])
val mean and std: tensor([0.6925, 0.5451, 0.6686]) tensor([0.2145, 0.2682, 0.2022])
test mean and std: tensor([0.6702, 0.5377, 0.6640]) tensor([0.2191, 0.2640, 0.2034])
</code></pre>
<p>I had forgotten to comment the normalize values for my larger dataset – very dumb sorry <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        #transforms.RandomResizedCrop(input_size),
        transforms.Resize((input_size, input_size)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        #transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    
    'test': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        #transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
}
</code></pre>

Mona_Jalal: <aside class="quote no-group" data-username="Mona_Jalal" data-post="2" data-topic="147701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p><code>[0.2047, 0.2554, 0.1914]</code></p>
</blockquote>
</aside>
<p>Could you please confirm if I should use this values for Normalize?</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        #transforms.RandomResizedCrop(input_size),
        transforms.Resize((input_size, input_size)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7057, 0.5569, 0.6816], [0.2047, 0.2554, 0.1914])
    ]),
    'val': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        #transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7057, 0.5569, 0.6816], [0.2047, 0.2554, 0.1914])
    ]),
    
    'test': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        #transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transforms.Normalize([0.7057, 0.5569, 0.6816], [0.2047, 0.2554, 0.1914])
    ])
}
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Mona_Jalal" data-post="2" data-topic="147701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>I would like to learn mathematically or with a proof why we want to use the mean and std value of train set for normalization of val and test set too.</p>
</blockquote>
</aside>
<p>The idea is that the val and test sets are “new” or “unseen” data.<br>
If you are using these splits to calculate any stats or use them in any other way, they are not new anymore as you are leaking the data information into your training.<br>
This is also why a validation <em>and</em> test set is needed.<br>
Even though you are not training with the validation dataset, you can still use it for e.g. early stopping, which uses some information about the validation data for the training.<br>
The same applies for the normalization. Think about the real use case of deploying the model where you would usually not have access to the new data which will be passed to your model.</p>

Mona_Jalal: <p>Thank you so much for your explanation. This is very helpful.</p>