Batch norm instability

derJaeger: <p>Hi everyone,<br>
I am having issues with batch norm for a while now. What happens is essentially that the exponential moving averages of mean and variance get corrupted at some point and do not represent the batch statistics anymore for whatever reason. This results in a stark increase in validation loss and bad predictions overall. When I run the validation with network.train() instead of eval() then everything stays normal (but is of course impractical for inference). (That’s how I know the moving averages must be at fault because that’s the only difference between train() and eval() in this model).<br>
Here is some additional detail to maybe help you understand what is going on:<br>
I am training a ResNet 50 on an image classification problem. The particular implementation of resnet is custom but I have had the same issues with the torchvision implementation. I am using Adam with an initial learning rate of 3e-4 and reduce the lr by a factor of 5 whenever the train loss has not improved in the last 30 epochs (ReduceLROnPlateau lr scheduler). The loss is the sum of BCE and F1 loss. BatchNorm parameters are all on default.<br>
I have attached a typical training curve down below. blue = train loss, red = val loss (both use the y axis on the left). green is evaluation metric (f1 score in this case; uses right axis).<br>
Interestingly the problem seems to solve itself as the learning rate decreases (see around epoch 850). Also interestingly, it is not easy to reproduce this problem. Sometimes it happens, other times it doesn’t.<br>
Do you have any idea what could cause this kind of behavior?<br>
Thank you so much!<br>
Fabian</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d81854449172d5f26b524234c2b1af0bc68dafc5.png" data-download-href="https://discuss.pytorch.org/uploads/default/d81854449172d5f26b524234c2b1af0bc68dafc5" title="progress"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d81854449172d5f26b524234c2b1af0bc68dafc5_2_625x500.png" alt="progress" data-base62-sha1="uPFg29PdQ6MuAt2Fl1EipDtvBwV" width="625" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d81854449172d5f26b524234c2b1af0bc68dafc5_2_625x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d81854449172d5f26b524234c2b1af0bc68dafc5_2_937x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d81854449172d5f26b524234c2b1af0bc68dafc5_2_1250x1000.png 2x" data-dominant-color="F9F9F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">progress</span><span class="informations">3000×2400 666 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>That’s an interesting observation.<br>
Did you have a look at the running stats before the loss exploded and during this issue?<br>
Are you shuffling the <code>Dataset</code> in each epoch?<br>
Have you tried to freeze the <code>BatchNorm</code> layers after a while? Currently I cannot imagine why the running stats should get that worse, if your model has already been trained for more than 400 epochs successfully.</p>

derJaeger: <p>Hi ptrblck, thanks for getting back to me.<br>
I should say that my use of the term epoch is a little off. What I am effectively doing is defining an epoch as iterating over 250 training batches (batch size 40). I have about 80k training images total. Each batch is either sampled randomly or with some probability for each training example from those 80k (there is some class imbalance and I am tackling that by oversampling rare classes).<br>
I will definitely check the numerical values of the running stats before, at and after the collapse. Right now I am not at home though (will have to wait until tomorrow). I will post them asap.<br>
One of my most recent experiments started giving nan values for validation loss. This is typically an indication that some activations are pushed to inf. Interesting. Maybe eps is doing something it shouldn’t?<br>
Best,<br>
Fabian</p>

ptrblck: <p>Thanks for the info.<br>
The strange loss behavior might be also related to your sampling strategy or the input in general.<br>
Could you check all inputs for <code>NaNs</code> or <code>Infs</code> in your training loop or beforehand?<br>
<code>torch.isnan</code> or <code>torch.isinf</code> might be helpful here.</p>

derJaeger: <p>Hi,<br>
all my inputs are normalized. I compute the mean and standard deviation over the dataset and normalize all images with that. There are no nans or infs in the data.<br>
How could this issue be related to my sampling strategy? Oversampling some cases may skew the running stats but not to an extend where everything collapses and especially the models recovering back to normal towards the end of the training (when the lr gets lower) contradicts that.<br>
I don’t quite know how the learning rate interacts with batch norm layers. As far as I remember, the gradient is not stopped for mean and var computation (via something like detach()). Maybe there is something there?<br>
Best,<br>
Fabian</p>

ptrblck: <p>I don’t think the oversampling strategy is bad at all, sorry for the misunderstanding.<br>
Since your current issue occurs randomly, I thought that maybe some batches, which are sampled randomly at a later epoch, might have invalid or very high/low values, thus throwing your running stats off.<br>
If you check the running stats in your training loop, could you also print the min/max/mean/std of the current batch?<br>
Maybe it’s not related to the running stats at all, but to other parameters which might suffer from potentially bad input.</p>
<p>As you see, I’m currently trying to collect all your information and think about some useful debug approaches. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

derJaeger: <p>Hi, yes I will provide those stats as soon as I can. Thank you so much again! I still don’t believe the input has invalid numbers as these are basically uint8 images that are normalized with some mean and std computed over the whole dataset (no infs or nans there). But better safe than sorry, right?<br>
I will also look at the batch norm running stats from different epochs (before, t, after the breakdown). What layer would you recommend I look at for those (it’s a resnet 50)?<br>
Best,<br>
Fabian</p>

ptrblck: <p>Yeah, I think the same. Even if the inputs should be fine, it doesn’t hurt to check them too.</p>
<p>I would suggest to have a look at the BatchNorm layers close to the input. It’s just a feeling, but I guess if something goes wrong it should be somewhere at the beginning of the forward pass.</p>

derJaeger: <p>Hi there,<br>
I could finally look at the network. I used epoch 300, 700 and final for before, during and after the breakdown.</p>
<p>Interestingly I don’t see any weirdness in the batch norm moving stats at the first bn layer:</p>
<blockquote>
<p><strong>epoch 300:</strong></p>
<p>In [9]: self.network.bn1.running_mean<br>
Out[9]:<br>
tensor([ 0.0265, -0.1110, -0.1160,  0.0190,  0.1058, -0.0179, -0.0286,  0.0524,<br>
-0.0180, -0.1844,  0.0287,  0.2987,  0.0370, -0.6230,  0.0031,  0.0256,<br>
-0.0679,  0.1194, -0.2744,  0.0251, -0.0909,  0.0986, -0.0009, -0.2132,<br>
0.0537,  0.1174, -0.0971, -0.0125, -0.0445, -0.0024, -0.3101, -0.0073],<br>
device=‘cuda:0’)</p>
<p>In [10]: self.network.bn1.running_var<br>
Out[10]:<br>
tensor([ 1.2123,  1.7482,  2.6450,  1.9762,  1.9177,  4.5315,  0.4663,  1.5161,<br>
1.1491,  6.9815,  1.1021,  9.3386,  1.2492, 29.5786,  1.1680,  0.4657,<br>
4.0417,  2.0542, 16.2966,  2.7019,  2.1935,  1.9766,  0.9401,  5.6444,<br>
1.5645,  1.7632,  6.9528,  1.8997,  1.1977,  1.5128, 20.3052,  0.8678],<br>
device=‘cuda:0’)</p>
<p>In [11]:</p>
<p><strong>epoch 700:</strong><br>
In [12]: self.network.bn1.running_mean<br>
Out[12]:<br>
tensor([ 0.0273, -0.1171, -0.0941,  0.0393,  0.1386,  0.0285, -0.0054,  0.0831,<br>
-0.0080, -0.1630,  0.0332,  0.3122,  0.0628, -0.4963, -0.0061,  0.0065,<br>
-0.1043,  0.1000, -0.2297,  0.0593, -0.0857,  0.0967,  0.0244, -0.1920,<br>
0.0275,  0.0837, -0.2192,  0.0217, -0.0661, -0.0337, -0.4571, -0.0109],<br>
device=‘cuda:0’)</p>
<p>In [13]: self.network.bn1.running_var<br>
Out[13]:<br>
tensor([ 1.1938,  2.7972,  4.3059,  3.4096,  2.8702,  5.8498,  0.5527,  2.3203,<br>
1.7257, 10.2496,  1.3604, 15.1904,  2.0068, 42.5539,  1.6873,  0.5314,<br>
6.4726,  3.1628, 22.7749,  4.3518,  2.8136,  2.8452,  1.9811,  7.4680,<br>
2.3911,  2.3746, 18.5319,  3.0089,  1.7593,  2.4780, 40.4268,  1.4430],<br>
device=‘cuda:0’)</p>
<p>In [14]:</p>
<p><strong>final:</strong></p>
<p>In [15]: self.network.bn1.running_mean<br>
Out[15]:<br>
tensor([ 0.0027, -0.1487, -0.1678,  0.0462,  0.1889,  0.0641,  0.0002,  0.0834,<br>
-0.0166, -0.1836,  0.0333,  0.4439,  0.0763, -0.7683,  0.0203,  0.0281,<br>
-0.1092,  0.1382, -0.2974,  0.0031, -0.0821,  0.1381,  0.0201, -0.2344,<br>
0.0412,  0.1326, -0.2208,  0.0218, -0.0966, -0.0137, -0.5406, -0.0126],<br>
device=‘cuda:0’)</p>
<p>In [16]: self.network.bn1.running_var<br>
Out[16]:<br>
tensor([ 1.1598,  3.0503,  4.5197,  3.4007,  3.1393,  5.8271,  0.5361,  2.2878,<br>
1.7715, 10.5495,  1.2760, 15.2090,  1.9926, 43.3876,  1.8111,  0.5640,<br>
5.4958,  3.2575, 22.9858,  4.0293,  2.7336,  3.1246,  2.0181,  7.6566,<br>
2.3407,  2.4714, 17.4709,  2.9588,  1.6889,  2.3654, 37.0451,  1.4181],<br>
device=‘cuda:0’)</p>
</blockquote>
<p>Here are some descriptive statistics of the running mean and var of the last bn layer (the one that comes after the last convolution in the very last residual block. It has 1024 entries which is why I am providing stats only here)</p>
<blockquote>
<p><strong>epoch 300:</strong></p>
<p>In [33]: self.network.layers[-1][-1].bn2.running_mean.max()<br>
Out[33]: tensor(0.2174, device=‘cuda:0’)</p>
<p>In [34]: self.network.layers[-1][-1].bn2.running_mean.min()<br>
Out[34]: tensor(-0.6384, device=‘cuda:0’)</p>
<p>In [35]: self.network.layers[-1][-1].bn2.running_mean.mean()<br>
Out[35]: tensor(-0.1016, device=‘cuda:0’)</p>
<p>In [36]: self.network.layers[-1][-1].bn2.running_mean.var()<br>
Out[36]: tensor(0.0184, device=‘cuda:0’)</p>
<p>In [39]: self.network.layers[-1][-1].bn2.running_var.max()<br>
Out[39]: tensor(0.8339, device=‘cuda:0’)</p>
<p>In [40]: self.network.layers[-1][-1].bn2.running_var.min()<br>
Out[40]: tensor(4.8037e-34, device=‘cuda:0’)</p>
<p>In [41]: self.network.layers[-1][-1].bn2.running_var.mean()<br>
Out[41]: tensor(0.0686, device=‘cuda:0’)</p>
<p>In [42]: self.network.layers[-1][-1].bn2.running_var.var()<br>
Out[42]: tensor(0.0122, device=‘cuda:0’)</p>
<p><strong>epoch 700:</strong></p>
<p>In [48]: self.network.layers[-1][-1].bn2.running_mean.max()<br>
Out[48]: tensor(0.1805, device=‘cuda:0’)</p>
<p>In [49]: self.network.layers[-1][-1].bn2.running_mean.min()<br>
Out[49]: tensor(-0.4298, device=‘cuda:0’)</p>
<p>In [50]: self.network.layers[-1][-1].bn2.running_mean.mean()<br>
Out[50]: tensor(-0.0727, device=‘cuda:0’)</p>
<p>In [51]: self.network.layers[-1][-1].bn2.running_mean.var()<br>
Out[51]: tensor(0.0118, device=‘cuda:0’)</p>
<p>In [44]: self.network.layers[-1][-1].bn2.running_var.max()<br>
Out[44]: tensor(0.2591, device=‘cuda:0’)</p>
<p>In [45]: self.network.layers[-1][-1].bn2.running_var.min()<br>
Out[45]: tensor(5.6052e-45, device=‘cuda:0’)</p>
<p>In [46]: self.network.layers[-1][-1].bn2.running_var.mean()<br>
Out[46]: tensor(0.0242, device=‘cuda:0’)</p>
<p>In [47]: self.network.layers[-1][-1].bn2.running_var.var()<br>
Out[47]: tensor(0.0016, device=‘cuda:0’)</p>
<p><strong>final:</strong></p>
<p>In [53]: self.network.layers[-1][-1].bn2.running_mean.max()<br>
Out[53]: tensor(0.0899, device=‘cuda:0’)</p>
<p>In [54]: self.network.layers[-1][-1].bn2.running_mean.min()<br>
Out[54]: tensor(-0.4327, device=‘cuda:0’)</p>
<p>In [55]: self.network.layers[-1][-1].bn2.running_mean.mean()<br>
Out[55]: tensor(-0.0624, device=‘cuda:0’)</p>
<p>In [56]: self.network.layers[-1][-1].bn2.running_mean.var()<br>
Out[56]: tensor(0.0093, device=‘cuda:0’)</p>
<p>In [57]: self.network.layers[-1][-1].bn2.running_var.max()<br>
Out[57]: tensor(0.1923, device=‘cuda:0’)</p>
<p>In [58]: self.network.layers[-1][-1].bn2.running_var.min()<br>
Out[58]: tensor(5.6052e-45, device=‘cuda:0’)</p>
<p>In [59]: self.network.layers[-1][-1].bn2.running_var.mean()<br>
Out[59]: tensor(0.0170, device=‘cuda:0’)</p>
<p>In [60]: self.network.layers[-1][-1].bn2.running_var.var()<br>
Out[60]: tensor(0.0009, device=‘cuda:0’)</p>
</blockquote>
<p>Batch norm statistics look normal-ish to me although I think running variances of 1e-45 are dangerous (almost division by zero). How can that happen?</p>
<p>I will plot batch statistics in my next post</p>

derJaeger: <p>Here I am showing both training batch statistics as well as statistics of feature maps activations right before the classification layer. Again I use epochs 300, 700 and final (1015) for before, during and after the breakdown. For each epoch, I draw 5 batches and show the aforementioned statistics for training, validation (network.train = False) and validation with network.train = True (I am including the latter to demonstrate that the problem must be the running stats as feature map activations for train=False are off while they are OK for train=Tru). As you can see, input data statistics are consistent for all batches and epochs, so no surprise here. For epoch 300 and final, feature maps statistics are consistent-ish for train, validation (train=False) and validation (train=True). For epoch 700 however (which is where the network breaks down), the feature map activations match between train and val (train=True) but don’t match for validation (train=False; this is where the running stats are applied).</p>
<p>I am also printing the loss for all epochs and phases (train. val, …). Don’t worry about the loss being negative. That is because the F1 loss ranges from 0 to -1 while the BCE is [0, inf] (total loss is the sum and thus ranges from [-1, inf]). In epochs 300 and final the loss between train and validations is basically identical (with some overfitting) while in epoch 700 the loss for validation (train=False) is way too high.</p>
<blockquote>
<p><strong>epoch:  300</strong></p>
<p><strong>training: (network is train=True and samples are drawn from training DataLoader)</strong><br>
Input data: max: 8.606085 min: -0.7775649 mean: 0.11011692 var: 1.1633884<br>
Features before classification: max: 17.142632 min: 0.0 mean: 0.24466729 var: 0.29341334</p>
<p>Input data: max: 8.5557785 min: -0.7629295 mean: 0.04259184 var: 1.0552053<br>
Features before classification: max: 18.69158 min: 0.0 mean: 0.24318185 var: 0.29009783</p>
<p>Input data: max: 8.638606 min: -0.7287697 mean: 0.10248642 var: 1.160278<br>
Features before classification: max: 22.366526 min: 0.0 mean: 0.24295413 var: 0.29546708</p>
<p>Input data: max: 8.53746 min: -0.76369715 mean: 0.14483288 var: 1.2324796<br>
Features before classification: max: 15.086981 min: 0.0 mean: 0.24739826 var: 0.28974423</p>
<p>Input data: max: 8.563448 min: -0.7089811 mean: 0.08948848 var: 1.133431<br>
Features before classification: max: 32.820637 min: 0.0 mean: 0.24512042 var: 0.28797802<br>
train loss : -0.5370</p>
<p><strong>validation: (network is train=False and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.07451663 var: 1.0982525<br>
Features before classification: max: 19.99752 min: 0.0 mean: 0.24418607 var: 0.2950879</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.06573128 var: 1.0647457<br>
Features before classification: max: 21.51592 min: 0.0 mean: 0.23669797 var: 0.2616221</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.07230965 var: 1.1099536<br>
Features before classification: max: 34.280563 min: 0.0 mean: 0.26418424 var: 0.3719333</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.16783251 var: 1.3045272<br>
Features before classification: max: 26.123634 min: 0.0 mean: 0.25989175 var: 0.37081867</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.110146575 var: 1.2274413<br>
Features before classification: max: 26.462603 min: 0.0 mean: 0.22915748 var: 0.25221625<br>
val loss (train=False): -0.4176</p>
<p><strong>validation (train=True): (network is train=True and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.041268155 var: 1.1999401<br>
Features before classification: max: 21.246727 min: 0.0 mean: 0.24235177 var: 0.2917685</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.1290773 var: 1.2299877<br>
Features before classification: max: 23.499233 min: 0.0 mean: 0.2495341 var: 0.28842986</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.077322766 var: 1.0789642<br>
Features before classification: max: 20.976784 min: 0.0 mean: 0.24999543 var: 0.28809312</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.12412989 var: 1.2118222<br>
Features before classification: max: 20.250025 min: 0.0 mean: 0.25311542 var: 0.28560916</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.058781143 var: 1.1097133<br>
Features before classification: max: 16.501434 min: 0.0 mean: 0.2545616 var: 0.28281626<br>
val loss (train=True): -0.5057</p>
<p><strong>epoch:  700</strong></p>
<p><strong>training: (network is train=True and samples are drawn from training DataLoader)</strong><br>
Input data: max: 8.55883 min: -0.77725536 mean: 0.11748674 var: 1.1631266<br>
Features before classification: max: 15.908981 min: 0.0 mean: 0.2101385 var: 0.21815181</p>
<p>Input data: max: 8.53746 min: -0.7591846 mean: 0.16579734 var: 1.1916564<br>
Features before classification: max: 16.09063 min: 0.0 mean: 0.21404302 var: 0.21558203</p>
<p>Input data: max: 8.555547 min: -0.6906217 mean: 0.101644285 var: 1.1464278<br>
Features before classification: max: 17.098934 min: 0.0 mean: 0.2126852 var: 0.21343498</p>
<p>Input data: max: 8.594714 min: -0.74816304 mean: 0.10817674 var: 1.1627903<br>
Features before classification: max: 17.051395 min: 0.0 mean: 0.20846856 var: 0.21932985</p>
<p>Input data: max: 8.563169 min: -0.78391445 mean: 0.051834863 var: 1.0531709<br>
Features before classification: max: 18.298471 min: 0.0 mean: 0.20677677 var: 0.21607172<br>
train loss : -0.6972</p>
<p><strong>validation: (network is train=False and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.13776894 var: 1.1895192<br>
Features before classification: max: 9.838846 min: 0.0 mean: 0.14135005 var: 0.07449843</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.16435458 var: 1.2554524<br>
Features before classification: max: 10.988285 min: 0.0 mean: 0.1407173 var: 0.0791312</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: -0.014078559 var: 0.9748203<br>
Features before classification: max: 11.078625 min: 0.0 mean: 0.13703124 var: 0.0729197</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.07416497 var: 1.1208515<br>
Features before classification: max: 26.292133 min: 0.0 mean: 0.14437278 var: 0.093446925</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.06462406 var: 1.1092615<br>
Features before classification: max: 10.157297 min: 0.0 mean: 0.14090487 var: 0.08402736<br>
val loss (train=False): 0.1353</p>
<p><strong>validation (train=True): (network is train=True and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.10422288 var: 1.2318301<br>
Features before classification: max: 16.510902 min: 0.0 mean: 0.20996732 var: 0.21453643</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.058634315 var: 1.1580375<br>
Features before classification: max: 20.746746 min: 0.0 mean: 0.20406458 var: 0.21710423</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.096261665 var: 1.1046003<br>
Features before classification: max: 20.254076 min: 0.0 mean: 0.20849131 var: 0.21499783</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.124039836 var: 1.2028973<br>
Features before classification: max: 19.05787 min: 0.0 mean: 0.2078148 var: 0.2171791</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.13756868 var: 1.1901097<br>
Features before classification: max: 17.98934 min: 0.0 mean: 0.21233286 var: 0.21371831<br>
val loss (train=True): -0.4974</p>
<p><strong>epoch:  1015</strong></p>
<p><strong>training: (network is train=True and samples are drawn from training DataLoader)</strong><br>
Input data: max: 8.591203 min: -0.744614 mean: 0.12869895 var: 1.2715969<br>
Features before classification: max: 15.38456 min: 0.0 mean: 0.22904472 var: 0.23167339</p>
<p>Input data: max: 8.543597 min: -0.75908107 mean: 0.04424698 var: 1.0194602<br>
Features before classification: max: 20.93504 min: 0.0 mean: 0.22539279 var: 0.22878553</p>
<p>Input data: max: 8.600206 min: -0.7582485 mean: 0.08429426 var: 1.2217238<br>
Features before classification: max: 16.953554 min: 0.0 mean: 0.22585449 var: 0.23169553</p>
<p>Input data: max: 8.589985 min: -0.74655396 mean: 0.009551339 var: 0.94260657<br>
Features before classification: max: 15.600396 min: 0.0 mean: 0.22332579 var: 0.22921476</p>
<p>Input data: max: 8.6038065 min: -0.75667745 mean: 0.11769733 var: 1.1169093<br>
Features before classification: max: 21.059433 min: 0.0 mean: 0.22816601 var: 0.23191863<br>
train loss : -0.7191</p>
<p><strong>validation: (network is train=False and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.057770584 var: 1.0422962<br>
Features before classification: max: 18.795538 min: 0.0 mean: 0.21927223 var: 0.20996067</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.03980498 var: 1.0491408<br>
Features before classification: max: 19.233257 min: 0.0 mean: 0.2139626 var: 0.1971896</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.05019422 var: 1.0788214<br>
Features before classification: max: 22.418144 min: 0.0 mean: 0.23339844 var: 0.25270167</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.12727928 var: 1.2005315<br>
Features before classification: max: 18.214798 min: 0.0 mean: 0.23262215 var: 0.25595525</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.09293718 var: 1.1846416<br>
Features before classification: max: 18.957647 min: 0.0 mean: 0.22007352 var: 0.21793382<br>
val loss (train=False): -0.5545</p>
<p><strong>validation (train=True): (network is train=True and samples are drawn from validation DataLoader)</strong><br>
Input data: max: 8.53746 min: -0.6521306 mean: 0.121076666 var: 1.3058403<br>
Features before classification: max: 19.245342 min: 0.0 mean: 0.23451522 var: 0.22419384</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.15040742 var: 1.227032<br>
Features before classification: max: 19.016422 min: 0.0 mean: 0.23178016 var: 0.2272836</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.18541424 var: 1.3306359<br>
Features before classification: max: 18.303911 min: 0.0 mean: 0.2370149 var: 0.22376394</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.036099356 var: 0.971277<br>
Features before classification: max: 21.868843 min: 0.0 mean: 0.23125803 var: 0.22653475</p>
<p>Input data: max: 8.53746 min: -0.6521306 mean: 0.07943766 var: 1.2029198<br>
Features before classification: max: 23.589802 min: 0.0 mean: 0.23089185 var: 0.22577839<br>
val loss (train=True): -0.5044</p>
</blockquote>
<p>If you need anything else please let me know!<br>
Best,<br>
Fabian</p>

ptrblck: <p>Thanks for the detailed debugging!<br>
I’m currently not sure what’s going on and try to thing about a sane way to debug this issue.</p>

derJaeger: <p>No worries. I very much appreciate that you are willing to spend some of your valuable time to figure this out!<br>
I found out that by setting the initial learning rate to 6e-5 instead of 3e-4 (reduced by factor 5) will make this problem go away. So I am not in a hurry to figure this out right now. Still I think it would be interesting to understand what is going on, because I am definitely not getting divergence (training loss is very smooth and behaves as expected).</p>
<p>Here are some more observations that may help understand the issue:</p>
<ul>
<li>Training with instance norm (track_running_stats=False) or group norm will never exhibit the behavior shown above.</li>
<li>Deeper architectures may be more robust to the problem mentioned here (this is a Resnet 50, Resnet 152 is stable, but interestingly Resnet 101 breaks completely and will not recover. Note that I am reducing the batch size when training larger models to accommodate everything on my GPU (patch size is 512x512))</li>
<li>The issue might be related to the loss function. My gut feeling is that running BCE loss only (no F1 loss) is less likely to run into this issue. F1 loss is not as nicely defined as BCE and I believe it may send very high gradients from time to time (I would need to check this though)</li>
</ul>
<p>Best,<br>
Fabian</p>

ptrblck: <p>Thanks for the detailed information!<br>
I think I was a bit mislead by the strange behavior and assumed some invalid inputs/updates/running estimates etc. Based on your description it seems the training setup is just creating this issue. I’m not sure if it’s possible or worth “debugging” it, as you already found some workarounds. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

deJQK: <p>I am facing the same problem, and I guess that the reason is due to for some batches and some layers, the inputs are all negative before relu and thus become zero, for depthwise layers. These will make the running stats gradually become very small values, and thus making the weights very large. Large weights typically will cause overfitting issue, and this is the reason why the training loss is small but validation loss is huge. I checked it and as the running_var is 5.6052e-45, the ratio between weight and bn std (which will become 3.162e-3, i.e. sqrt(1e-5) as bn.eps is 1e-5) will be around thousands. I wonder what will be the proper way if for some batch all inputs are zero. I tried to set bn.eps to 0.1 but although the weight are not huge, there is still strong overfitting (because in this case the BN might not work properly as the running stats are not collected correctly). BTW, I did not apply weight decay for depthwise conv weights.</p>