Do we do whitening on batch basis or on dataset basis?

username: BarCodeReader
cooked: <p>I just feel a bit confused about whitening, like PCA or similar techniques.</p>
<p>Common approach is to substract mean and divide std on dataset(i.e. mean is dataset mean, std is dataset std)</p>
<p>However, I feel we should do this on batch basis. because we use minibatch to feed into NN and each batch we cannot garentee the batch_mean and batch_std is the same as dataset_mean and dataset_std.</p>
<p>but i did not see anyone doing this on batch basis…</p>
<p>is there something wrong with my understanding? or i miss something?</p>

username: ptrblck
cooked: <p><code>nn.BatchNorm</code> layers will do exactly this for each batch.<br>
The <a href="https://en.wikipedia.org/wiki/Standard_score">Standard score</a> on the other hand is usually calculated for the whole train set.</p>

username: BarCodeReader
cooked: <p>Thanks for the explaination.<br>
Then my confusion is why we apply standard score on dataset basis? is there any specific reason? Because the first idea come to my mind is to perform everything on batch. My thinking is we only feed in mini-batch to NN, thus we should perform everything on batch-basis…and this is actually how human learns right? we don’t need to know what is the real distribution of the data/datasest, we just try to learn based on samples we have on hand…<br>
Really thanks if you can give me some hints.</p>

username: ptrblck
cooked: <p>I think the main reason is, that you don’t necessarily have batches during testing, which might create bad estimates of the mean and std.<br>
You estimate the statistics of your dataset using the trainset and try to make sure the validation and test data is sampled from the same distribution.</p>
<p>The same would apply e.g. if you use batch norm layers with a batch size of 1.<br>
The running estimates will be noisy and thus the validation and test error might be large.</p>
<p>Relying on a specific batch size seems like a bad idea in the general use case.</p>

username: BarCodeReader
cooked: <p>Thanks for your explaination, I think it’s quite valid and make sense.<br>
So currently, we have a dataset, and we perform whitening/normalization, then we apply dataloader to generate minibatch, the minibatch maynot follow the dataset distribution right, i.e. mean for minibatch maynot be exactly 0(like the red dots shown in graph)? so that is why we also apply batchnrom to make sure the batch is 0-centered:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d86f9d8c0fc77601eb0e4ab5151314558882f00.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d86f9d8c0fc77601eb0e4ab5151314558882f00" title="111"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d86f9d8c0fc77601eb0e4ab5151314558882f00_2_690x144.png" alt="111" data-base62-sha1="4dd1WmomhypZPg6EvxI1QIQTZv2" width="690" height="144" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d86f9d8c0fc77601eb0e4ab5151314558882f00_2_690x144.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d86f9d8c0fc77601eb0e4ab5151314558882f00_2_1035x216.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d86f9d8c0fc77601eb0e4ab5151314558882f00_2_1380x288.png 2x" data-dominant-color="9092AD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">111</span><span class="informations">1912×400 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>but what I firstly come in mind is somthing like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ab12963d4a92857750d510d839d381d29a1439d.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ab12963d4a92857750d510d839d381d29a1439d" title="222"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab12963d4a92857750d510d839d381d29a1439d_2_690x139.png" alt="222" data-base62-sha1="aEKTNrVLTWZWAILeKdLzcFt7OAB" width="690" height="139" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab12963d4a92857750d510d839d381d29a1439d_2_690x139.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab12963d4a92857750d510d839d381d29a1439d_2_1035x208.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab12963d4a92857750d510d839d381d29a1439d_2_1380x278.png 2x" data-dominant-color="6F6DAB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">222</span><span class="informations">2014×408 151 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I dont really care the distribution of the dataset, I just directly use them to generate minibatch(red dots), but for every minibatch i do whitening(we calculate the matrix every batch) so i make sure every minibatch is 0-centered, and then maybe I don’t need BatchNorm layers??</p>
<p>and also this approach should not have any problem during testing, because the distribution of test is not important as we whitening on batch</p>
<p>This is something which I feel the main stream approach is different from mine…<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"> and I get confused about it…I want to know where I miss something…</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="BarCodeReader" data-post="5" data-topic="64554">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barcodereader/48/18717_2.png" class="avatar"> BarCodeReader:</div>
<blockquote>
<p>and also this approach should not have any problem during testing, because the distribution of test is not important as we whitening on batch</p>
</blockquote>
</aside>
<p>How would you whiten the test samples, if you are only using one sample?<br>
This would mean, that each red dot would be centered, wouldn’t it?</p>

username: BarCodeReader
cooked: <p>actually I am also think about this…so based on current approach(i.e. whitening on dataset basis), how the batch_norm is working if we set Batch_size = 1 in both training and testing? the image(dot) will also be in center right?</p>

username: ptrblck
cooked: <p>During training the batch norm layers will update their running estimates of the mean and std and just apply these estimates during validation and testing.<br>
The <code>momentum</code> term defines the update step of the running estimates using the current batch statistics.<br>
We could see some issues (bad testing accuracy while training looked good), when these estimates are off due to e.g. different data domains for the datasets, and might need to resample/shuffle the data or change the <code>momentum</code> term.</p>
<p>That being said, I don’t want to say that your approach is invalid in any sense.<br>
If you have valid assumptions, how you would like to apply the whitening during training and testing, I would recommend to run some tests and see, how it compares to e.g. standardization and batch norm. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>