How to normalize a tensor to 0 mean and 1 variance?

evansnd: <p>Hi I’m currently converting a tensor to a numpy array just so I can use sklearn.preprocessing.scale<br>
Is there a way to achieve this in PyTorch? I have seen there is torchvision.transforms.Normalize but I can’t work out how to use this outside of the context of a dataloader. (I’m  trying to use this on a tensor during training)</p>
<p>Thanks in advance</p>

ptrblck: <p>You could add the normalization in the <code>__getitem__</code> function of your <code>Dataset</code>:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, X, y, transform=None):
        self.data = X
        self.target = y
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]

        # Normalize your data here
        if self.transform:
            x = self.transform(x)

        return x, y
    
    def __len__(self):
        return len(self.data)
</code></pre>
<p>In this use case, you could set <code>transform</code> to something like this:</p>
<pre><code class="lang-auto">transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])
</code></pre>

yvanscher: <p>Would this do it?</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms

mu = 2
std = 0.5
t = torch.Tensor([1,2,3])
(t - 2)/0.5
# or if t is an image
transforms.Normalize(2, 0.5)(t)
</code></pre>
<p>see:</p>
<p><a href="https://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.Normalize" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.Normalize</a></p>

evansnd: <p>Thanks but this one won’t work for my use case , as I am not trying to do this when I load the data, but as part of another calculation that I am performing during training.</p>

evansnd: <aside class="quote no-group" data-username="yvanscher" data-post="3" data-topic="18766">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yvanscher/48/2412_2.png" class="avatar"> yvanscher:</div>
<blockquote>
<p>transforms.Normalize(2, 0.5)(t)</p>
</blockquote>
</aside>
<p>Yeah I tried this and I always get an error:</p>
<p>" for t, m, s in zip(tensor, self.mean, self.std):<br>
TypeError: zip argument <span class="hashtag-raw">#2</span> must support iteration"</p>

evansnd: <p>Oh, the mean and std need to be arrays?</p>

yvanscher: <p>so you cant zip self.mean and self.std if they are sinlge values. zip takes multiple iterables and returns packaged tuples.</p>
<pre><code class="lang-auto">means = [self.mean] * tensor.size()[0]
stds = [self.std] * tensor.size()[0]
for t, m, s in zip(tensor, means, stds):
  # do stuff
</code></pre>
<p>turn the means and stds into a length n array where n is the length of ‘tensor’ or tensors</p>

evansnd: <p>Great! I see. thank you very much.</p>

billtubbs: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="18766">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>transforms.Normalize</p>
</blockquote>
</aside>
<p>I haven’t figured out how to use <code>transforms.Normalize</code> on input data that is not an image.  I get <code>TypeError: tensor is not a torch image.</code>  Is there any way to use this method on non-images?</p>

ptrblck: <p><code>Normalize</code> works on tensors, so the error message might come from another transformation:</p>
<pre><code class="lang-python">norm = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
x = torch.randn(3, 224, 224)
out = norm(x)
</code></pre>

billtubbs: <p>That only works because your tensor has the dimensions of an Image.  If you look at the documentation, it says <code>torchvision.transforms.Normalize</code> is used to <code>Normalize a tensor image with mean and standard deviation</code>.  The argument is described as a</p>
<blockquote>
<p><strong>tensor</strong>  (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <em>Tensor</em> </a>) – Tensor image of size (C, H, W) to be normalized.</p>
</blockquote>
<p>My data is sequence data of dimension <code>torch.Size([4, 589, 4])</code></p>

billtubbs: <p>Actually, you’re right the error does go away if I get the dimensions right:</p>
<pre><code class="lang-auto">norm = transforms.Normalize((30, 30, 30, 30), (20, 25, 30, 35))
x = torch.randn(4, 589, 4)
out = norm(x)
</code></pre>
<p>But I don’t think this is applying the normalization correctly.  The data from my data loader is shaped <code>[batch_size, seq_length, x_dim]</code> so the scaling should be applied to the last dimension, whereas I think normalize is applying the scaling across the first dimension (the set of image colour maps).</p>

akashs: <p>Is it possible to extend/apply the transforms.Normalize to normalize multidimensional tensor in custom pytroch dataset class? I have a tensor with shape <code>(S x C x W x H)</code> and I want to normalize on C dimension.</p>

mhxin: <p>thanks, and i have a question on how to set mean and std for each channel, are they calculated from dataset?</p>

ptrblck: <p>Yes, you can calculate the mean and std from your training dataset or use some “default” values e.g. from ImatgeNet.</p>

krioux5: <p>Is there a way to apply different transforms to the mask vs input? For example, I want to apply all deformation transforms to both, but i only want to normalize and totensor the predicted masks (not target)</p>

ptrblck: <p>I would recommend to use the functional API for these use cases, as it allows you to apply the same “random” transformation on the data and target, and can also be used to call some transformations on one of these tensors separately.<br>
Have a look at <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.</p>

ZimoNitrome: <p>Me and <a class="mention" href="/u/filipandersson245">@FilipAndersson245</a> found out that the correct way to unnormalize is:</p>
<pre><code class="lang-auto">x * std + mean
</code></pre>
<p>We also had to clamp a few values outside of [0,1].</p>
<p>For a single image the code would look something like this:</p>
<pre><code class="lang-auto">def inv_normalize(img):
    mean = torch.Tensor([0.485, 0.456, 0.406]).unsqueeze(-1)
    std= torch.Tensor([0.229, 0.224, 0.225]).unsqueeze(-1)
    img = (img.view(3, -1) * std + mean).view(img.shape)
    img = img.clamp(0, 1)
    return img
</code></pre>
<p>Feel free to help if the code can be written in a simpler way!</p>

_joker: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am also trying to do <code>transform.Normalize(mean, std)</code> outside data-loader but somewhere in the training process. I am not sure how would I do this for a <strong>batch of images</strong>.</p>
<p>Also, I am using <code>F.normalize(tensor, p=1, dim=1)</code> inside my model. Now, If I am loading the data with <code>transforms.Normalize(mean, std)</code> does it mean I am applying the same Normalization twice?</p>
<p>I saw the source for <code>transforms.Normalize</code> and it appears to be using <code>F.normalize(tensor, self.mean, self.std, self.inplace)</code> which I am not sure is the same thing or different.</p>

ptrblck: <p>To apply <code>transforms.Normalize</code> on a batch you could either run this transformation in a loop on each input or normalize the data tensoe manually via:</p>
<pre><code class="lang-python">x = (x - mean) / std
</code></pre>
<p>Inside <code>transforms.Normalize</code> the <code>torchvision.transforms.functional</code> API will be used as <code>F.normalize</code>.<br>
This is <em>not</em> the same methods as <code>torch.nn.functional.normalize</code> and will accept different input arguments.</p>