[Discussion] Why normalise according to ImageNet mean and std dev for transfer learning?

tlim: <p>According to the Pytorch official website, it is advised to use the following transform (normalisation as used for training under ImageNet):</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
</code></pre>
<p>I seen many scripts that uses pre-trained models provided by Pytorch and follow along with the recommendation of normalising according to the mean and standard deviation of ImageNet.</p>
<p>I do not understand why is it not recommended by Pytorch to normalise according to individual datasets considering that the purpose of normalisation is to bring the dataset to a standard normal distribution to ease the optimization process (gradient descent)? This is especially true for the purpose of fine-tuning where every weight and bias parameters is being changed according to a new dataset.</p>
<p>If we use ImageNet normalisation on our own dataset, it will not bring the our dataset to a standard normal distribution. Wouldn’t the effect of normalisation (following ImageNet) be the same as literally just subtracting random numbers that we pick out of thin air (from the perspective of our own dataset)?</p>

ptrblck: <aside class="quote no-group" data-username="tlim" data-post="1" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tlim/48/32364_2.png" class="avatar"> tlim:</div>
<blockquote>
<p>According to the Pytorch official website, it is advised to use the following transform (normalisation as used for training under ImageNet):</p>
</blockquote>
</aside>
<p>Could you post the link where it’s advised to use the ImageNet stats and not to calculate them from your dataset, please?</p>
<aside class="quote no-group" data-username="tlim" data-post="1" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tlim/48/32364_2.png" class="avatar"> tlim:</div>
<blockquote>
<p>I do not understand why is it not recommended by Pytorch to normalise according to individual datasets considering that the purpose of normalisation is to bring the dataset to a standard normal distribution to ease the optimization process (gradient descent)? This is especially true for the purpose of fine-tuning where every weight and bias parameters is being changed according to a new dataset.</p>
</blockquote>
</aside>
<p>I would claim these stats work on “natural” images (or in other words ImageNet-like images) and could yield a good baseline. Your training might benefit from custom stats, but you would need to check if for your use case (and please update this thread in case you have some results).<br>
In my experience your training might suffer, if you are using images from another domain (such as medical data), but it also depends on your use case and I don’t want to create a general assumption that it cannot work.</p>

tlim: <p>Link: <a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.16 documentation</a></p>
<p>From link:<br>
"<br>
All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel<br>
RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. You can use the following transform to normalize:</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
</code></pre>
<p>"</p>
<p>There’s no mention about not calculating them from own dataset (I never said that).</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would claim these stats work on “natural” images (or in other words ImageNet-like images) and could yield a good baseline. Your training might benefit from custom stats, but you would need to check if for your use case (and please update this thread in case you have some results).<br>
In my experience your training might suffer, if you are using images from another domain (such as medical data), but it also depends on your use case and I don’t want to create a general assumption that it cannot work.</p>
</blockquote>
</aside>
<p>This is where I am confused, I cannot figure out why would there be any potential benefit at all from the custom stat. What does ImageNet mean and standard deviation has anything to do with our own datasets ? The only exception I can think of is if someone is using the pre-trained model to improve the performance of the model on ImageNet itself. Given the diversity of ImageNet, our own data has to been equally diverse on similar categories to have any sort of use for the custom state.</p>
<p>This is just what I think, happy to discuss further.</p>

ptrblck: <p>Thanks for the link. I understand the part more as an explanation on how to use or fine-tune the pretrained model in order to get the same results. E.g. if you skip the normalization and just feed the raw ImageNet dataset into it, the performance will be worse.</p>
<aside class="quote no-group" data-username="tlim" data-post="3" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tlim/48/32364_2.png" class="avatar"> tlim:</div>
<blockquote>
<p>This is where I am confused, I cannot figure out why would there be any potential benefit at all from the custom stat.</p>
</blockquote>
</aside>
<p>As already said, you can calculate the custom stats and it would be interesting to see, if you are seeing any benefits. Reusing the ImageNet stats seems to work in practice for “a lot” of use cases but the “lot” depends on your research area (e.g. see the point about medical data from before).</p>
<aside class="quote no-group" data-username="tlim" data-post="3" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tlim/48/32364_2.png" class="avatar"> tlim:</div>
<blockquote>
<p>What does ImageNet mean and standard deviation has anything to do with our own datasets ?</p>
</blockquote>
</aside>
<p>The ImageNet dataset used to pretrain the <code>torchvision</code> models contains a variety of “natural” images from 1000 categories. If your custom image dataset contains similar images of e.g. dogs, cats, strawberries, etc. the same stats could work. Did you compute the stats from your images and compared them to the ImageNet ones? If so, how large was the difference?</p>

tlim: <p>I agree that if you skip normalisation (to normal distribution) and feed the raw ImageNet dataset, the performance will be worse for ImageNet but I am not too sure whether is that true for any other dataset.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>As already said, you can calculate the custom stats and it would be interesting to see, if you are seeing any benefits. Reusing the ImageNet stats seems to work in practice for “a lot” of use cases but the “lot” depends on your research area (e.g. see the point about medical data from before).</p>
</blockquote>
</aside>
<p>(<strong>Test 1</strong>): To test the above.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="115670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The ImageNet dataset used to pretrain the <code>torchvision</code> models contains a variety of “natural” images from 1000 categories. If your custom image dataset contains similar images of e.g. dogs, cats, strawberries, etc. the same stats could work. Did you compute the stats from your images and compared them to the ImageNet ones? If so, how large was the difference?</p>
</blockquote>
</aside>
<p>(<strong>Test 2</strong>): To test the above. My theory is that the ImageNet stats is useless because even if the dataset consist of similar categories of images to ImageNet, the custom stats of ImageNet does not in any way correlate due to the fact that it has way more categories (ex: 3 same categories vs 997 categories that are non-existent in your own dataset).</p>
<p>I will figure out how to best verify the 2 tests, get back to this post when I have done so.  Thank you for the reply, you are really active in this forum!</p>

tlim: <p><strong>Test</strong> (Nature images:  <a href="https://www.kaggle.com/puneet6060/intel-image-classification" class="inline-onebox" rel="noopener nofollow ugc">Intel Image Classification | Kaggle</a>)</p>
<p><strong>Experiment 1.1</strong>: ImageNet Custom Stats vs None</p>
<p>ResNet-18:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/851622762145d4ca2b0b7ab77c39011f962807b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/851622762145d4ca2b0b7ab77c39011f962807b5" title="resnet-18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/851622762145d4ca2b0b7ab77c39011f962807b5_2_690x451.png" alt="resnet-18" data-base62-sha1="iZkVstJHqb6MXggUfJOE7w8gATj" width="690" height="451" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/851622762145d4ca2b0b7ab77c39011f962807b5_2_690x451.png, https://discuss.pytorch.org/uploads/default/original/3X/8/5/851622762145d4ca2b0b7ab77c39011f962807b5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/851622762145d4ca2b0b7ab77c39011f962807b5.png 2x" data-dominant-color="ECECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet-18</span><span class="informations">942×617 48.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>ResNet-34:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09c78fded9d8cf469516655257aae475b38a13eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/09c78fded9d8cf469516655257aae475b38a13eb" title="Screenshot from 2021-04-01 09-13-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09c78fded9d8cf469516655257aae475b38a13eb_2_520x500.png" alt="Screenshot from 2021-04-01 09-13-46" data-base62-sha1="1ovR7FHr9hTpVZLAt1RIyvfsADV" width="520" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09c78fded9d8cf469516655257aae475b38a13eb_2_520x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09c78fded9d8cf469516655257aae475b38a13eb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09c78fded9d8cf469516655257aae475b38a13eb.png 2x" data-dominant-color="E4E4E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-01 09-13-46</span><span class="informations">654×628 50.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can see the general trend being that the normalised experiments are slightly above the experiment without normalisation most of the epochs.</p>
<p><strong>Conclusion 1</strong>: Using custom ImageNet stats for the <strong>pytorch pretrained</strong> models provides better results than without normalisation as it is trained on ImageNet.</p>
<p><strong>Experiment 1.2:</strong> ImageNet Custom Stats vs Own Custom Stats</p>
<p>ResNet-18:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aae347c63514c059e4efd04fa2f4b0b15bd5254.png" data-download-href="https://discuss.pytorch.org/uploads/default/2aae347c63514c059e4efd04fa2f4b0b15bd5254" title="Screenshot from 2021-04-01 10-10-31"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aae347c63514c059e4efd04fa2f4b0b15bd5254_2_690x347.png" alt="Screenshot from 2021-04-01 10-10-31" data-base62-sha1="65zhKx94tZphmbI1XzA8DGyocGo" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aae347c63514c059e4efd04fa2f4b0b15bd5254_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aae347c63514c059e4efd04fa2f4b0b15bd5254_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aae347c63514c059e4efd04fa2f4b0b15bd5254.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-01 10-10-31</span><span class="informations">1269×639 94.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>ResNet-34 (Unstable, need to go more epochs to smooth out the accuracy):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/8547021b0241b85defd8a68b8119982e27feb8a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/8547021b0241b85defd8a68b8119982e27feb8a9" title="Screenshot from 2021-04-01 11-34-09"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8547021b0241b85defd8a68b8119982e27feb8a9_2_690x357.png" alt="Screenshot from 2021-04-01 11-34-09" data-base62-sha1="j11DyqIQHXfzaA3rNcgvn3EcDcB" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8547021b0241b85defd8a68b8119982e27feb8a9_2_690x357.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8547021b0241b85defd8a68b8119982e27feb8a9_2_1035x535.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/8547021b0241b85defd8a68b8119982e27feb8a9.png 2x" data-dominant-color="EAEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-01 11-34-09</span><span class="informations">1263×655 117 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Conclusion 2</strong>: Using your own dataset stats gives better result.</p>
<p>For further experiments into other domains like medical images and stock technical charts photos, I think it is unnecessary as it is obvious to me that your own dataset custom stats will be superior.</p>
<p><strong>Personal Opinion:</strong> Just always find your own dataset mean and standard deviation, normalise accordingly. Keep this as a common practice.  It only takes roughly 5 minutes to write the script and run it.</p>

tlim: <p>Add on from previous post, there’s a strong and weak method to find the mean and standard deviation of your dataset:</p>
<p>View: <a href="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L142-L200" class="inline-onebox" rel="noopener nofollow ugc">CoarseNet/utils/preprocess.py at master · Nikronic/CoarseNet · GitHub</a></p>
<p>Running another test with ResNet-18, it can be observe that the Strong Method does give a little performance boost (own == weak, own_strong == strong):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a00c5379d5205bf9cfd83ec1b418889e80ded4f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/a00c5379d5205bf9cfd83ec1b418889e80ded4f8" title="Screenshot from 2021-04-01 14-11-48"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a00c5379d5205bf9cfd83ec1b418889e80ded4f8_2_680x500.png" alt="Screenshot from 2021-04-01 14-11-48" data-base62-sha1="mPQNPHifwuekLUBug6ioSuzZOK4" width="680" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a00c5379d5205bf9cfd83ec1b418889e80ded4f8_2_680x500.png, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a00c5379d5205bf9cfd83ec1b418889e80ded4f8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a00c5379d5205bf9cfd83ec1b418889e80ded4f8.png 2x" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-01 14-11-48</span><span class="informations">878×645 64.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The mean from both methods are the same, the difference is in the calculations for standard deviation.</p>

Grantor_Shadow: <p><a class="mention" href="/u/tlim">@tlim</a> Thanks for this thread, helped me a lot !</p>

satrya-sabeni: <p>Awesome stuff, this has really cleared up what effect the Imagenet normalisation actually has.</p>