Shouldn't mean & std be calculated across both the train & validation datasets?

username: oat
cooked: <p><strong>[Context]</strong><br>
Book: <a href="https://www.manning.com/books/deep-learning-with-pytorch" rel="noopener nofollow ugc">Deep Learning with PyTorch by Eli Stevens, Luca Antiga, and Thomas Viehmann</a> <a class="mention" href="/u/lantiga">@lantiga</a><br>
Jupyter Notebook: <a href="https://github.com/deep-learning-with-pytorch/dlwpt-code/blob/master/p1ch7/2_birds_airplanes.ipynb" rel="noopener nofollow ugc">Part 1 Chapter 7</a></p>
<p>As shown below the <strong>means</strong> and <strong>standard deviations</strong> for the RGB channels used in normalization for both the CIFAR10 <strong>training</strong> and <strong>validation</strong> datasets are the same.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88c3b4704ea2b3b809ba7874c8550835fc7c61aa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/88c3b4704ea2b3b809ba7874c8550835fc7c61aa" title="Screen Shot 2021-02-13 at 20.21.26 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88c3b4704ea2b3b809ba7874c8550835fc7c61aa_2_345x217.jpeg" alt="Screen Shot 2021-02-13 at 20.21.26 PM" data-base62-sha1="jvSemw3nglAUDGHBN1YgZqDcYQi" width="345" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88c3b4704ea2b3b809ba7874c8550835fc7c61aa_2_345x217.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88c3b4704ea2b3b809ba7874c8550835fc7c61aa_2_517x325.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88c3b4704ea2b3b809ba7874c8550835fc7c61aa_2_690x434.jpeg 2x" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-13 at 20.21.26 PM</span><span class="informations">817×516 92 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And, they were calculated based on the training set only, as shown in <a href="https://github.com/deep-learning-with-pytorch/dlwpt-code/blob/master/p1ch7/1_datasets.ipynb" rel="noopener nofollow ugc">a previous notebook</a> in the same chapter:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d66ae353583725f789d98666fca16cf5a6a83f5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4d66ae353583725f789d98666fca16cf5a6a83f5" title="Screen Shot 2021-02-13 at 20.29.35 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d66ae353583725f789d98666fca16cf5a6a83f5_2_411x500.jpeg" alt="Screen Shot 2021-02-13 at 20.29.35 PM" data-base62-sha1="b2IKWBbwgl4eqGKhrebRqH6aUux" width="411" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d66ae353583725f789d98666fca16cf5a6a83f5_2_411x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d66ae353583725f789d98666fca16cf5a6a83f5_2_616x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d66ae353583725f789d98666fca16cf5a6a83f5_2_822x1000.jpeg 2x" data-dominant-color="E5DFDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-13 at 20.29.35 PM</span><span class="informations">1045×1269 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>[Question]</strong><br>
Shouldn’t we calculate these <strong>means</strong> and <strong>standard deviations</strong> across the <strong>entire dataset</strong>, i.e. across <strong>both the training and validation datasets</strong> by stacking them up, as shown below?</p>
<p>Or, The way of “calculating means and stds for normalization using training dataset only” in the book is correct because it is to <strong>prevent the information in the validation dataset from “contaminating” normalization</strong>, and consequently, affecting the training?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec" title="Screen Shot 2021-02-13 at 20.45.22 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec_2_447x500.png" alt="Screen Shot 2021-02-13 at 20.45.22 PM" data-base62-sha1="afVVq7Mv2dWd2ayqEEd1tkothEw" width="447" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec_2_447x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec_2_670x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/7/47e2d9d2ad0a2c53bcb7015a12e857cbcc34c1ec.png 2x" data-dominant-color="222222"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-13 at 20.45.22 PM</span><span class="informations">806×900 72.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JuanFMontesinos
cooked: <p>validation or test sets are supposed to be unknown.<br>
If you use any information from them, you are cheating the results.</p>

username: oat
cooked: <p>Thanks. So, it’s to prevent the information in validation/test datasets from “contaminating” the training, right?</p>

username: JuanFMontesinos
cooked: <p>Yes, in fact, the reason why there exist validation and test sets is because we are still using the information from validation . When you save a checkpoint or weights based on validation, you are taking the best model for that subset but it doesn’t mean that it’s the best model for test set.</p>
<p>Simply, when you validate or test you cannot use prior information from those sets.</p>