MNIST Normalization

1414b35e42c77e0a57dd: <p>I’m confused about normalization process to MNIST.<br>
In 64 batch size, I think <code>transforms.Normalize((0.1307), (0.3081)</code> is right.</p>
<p>In my case, I already do <code>train_dataset.train_data.float() / 255.</code> in MNIST class so I can get (0.1307),(0.3081).<br>
then do not need to use normalization any more?<br>
This is my code.</p>
<pre><code class="lang-auto">class MNIST(data.Dataset):
    def __init__(self, split, transform=None):
        # self.transform = transformers.get_basic_transformer()
        self.split = split
        self.transform = transform

        if split == "train":

            train_dataset = datasets.MNIST(
                root="data/mnist", train=True, download=True, transform=None)
            self.X = train_dataset.train_data.float() / 255.
            self.y = train_dataset.train_labels

            np.random.seed(2)
            ind = np.random.choice(
                len(train_dataset), len(train_dataset), replace=False)
            self.X = self.X[ind]
            self.y = self.y[ind]

        elif split == "val":
            test_dataset = datasets.MNIST(root="data/mnist",
                                          train=False, download=True, transform=None)

            self.X = test_dataset.test_data.float() / 255.
            self.y = test_dataset.test_labels

    def __getitem__(self, index):
        X, y = self.X[index].clone(), self.y[index].clone()

        X = self.transform(X[None])

        return X, y

    def __len__(self):
        """Return size of dataset."""
        return self.X.shape[0]
</code></pre>
<pre><code class="lang-auto">    pre_process = transforms.Compose(
        [transforms.Normalize(mean=[0.5], std=[0.5])]) DO I HAVE TO DO THIS ?
</code></pre>
<ul>
<li>What about test dataset? Should I re-calculate mean and std from test data?</li>
</ul>

ptrblck: <p><code>transforms.Normalize</code> will standardize the data such that it’ll have a zero mean and unit variance.<br>
You don’t need to apply it, but it might help your training.</p>
<aside class="quote no-group" data-username="1414b35e42c77e0a57dd" data-post="1" data-topic="49080">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1414b35e42c77e0a57dd/48/7689_2.png" class="avatar"> 1414b35e42c77e0a57dd:</div>
<blockquote>
<p>What about test dataset? Should I re-calculate mean and std from test data?</p>
</blockquote>
</aside>
<p>No, you should apply the same statistics calculated from the training dataset.<br>
Otherwise this would be considered as data leakage.</p>

1414b35e42c77e0a57dd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks a reply !</p>
<p>My questions was not clear.<br>
What I want to ask you is that <code>train_dataset.train_data.float() / 255.</code> makes normalization<br>
then, <code>[transforms.Normalize(mean=[0.5], std=[0.5])])</code> is not needed ?</p>

ptrblck: <p>I think your question was clear, but my answer probably wasn’t. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Dividing by <code>255.</code> will normalize the data to the range <code>[0, 1]</code>.<br>
<code>transforms.Normalize</code> will standardize this data to zero mean and unit variance.<br>
So, <code>Normalize</code> might yield advantages for your training, but your model might also learn with fine without it.</p>
<p>Anyway, I would recommend using it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>