Repository: pytorch/vision

Issue:
Change default value of eps in FrozenBatchNorm to match BatchNorm
## ❓ Questions and Help
Hello
Loss is nan error occurs when I learn fast rcnn with resnext101 backbone
My code is as follows
```python
backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
```

error message
```
Epoch: [0]  [   0/7208]  eta: 1:27:42  lr: 0.000040  loss: 40613806080.0000 (40613806080.0000)  loss_box_reg: 7979147264.0000 (7979147264.0000)  loss_classifier: 11993160704.0000 (11993160704.0000)  loss_objectness: 9486380032.0000 (9486380032.0000)  loss_rpn_box_reg: 11155118080.0000 (11155118080.0000)  time: 0.7301  data: 0.4106  max mem: 1241
Loss is nan, stopping training
```

When i change the backbone to resnet50 and resnet152, no error occrus.
### Please note that this issue tracker is not a help form and this issue will be closed.

We have a set of [listed resources available on the website](https://pytorch.org/resources). Our primary means of support is our discussion forum:

- [Discussion Forum](https://discuss.pytorch.org/)



Discussion:
Hi @juyunsang 

as our template states:

> Please note that this issue tracker is not a help form and this issue will be closed. [...] Our primary means of support is our [discussion forum](https://discuss.pytorch.org).

---

Without knowing your data its hard to tell what is going wrong. I'm assuming your data is not corrupt since the other models are working. Thus, this might be a hyper-parameter problem. The loss in the first step seems fairly large (~ `40e9`). In a first step I would reduce the learning rate and see if this already solves the problem.
@juyunsang note that we do not provide pre-trained weights for detection models with the resnext101 backbone, which might explain the issue you are facing. You might be finetuning a detection model with ResNet50 pre-trained on COCO, while training it from scratch with ResNeXt101
@fmassa 
Thank you for reply
I don't understand your mention
```You might be finetuning a detection model with ResNet50 pre-trained on COCO, while training it from scratch with ResNeXt101```
Are you saying that I should use the code below?
```
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
```
Can you show me the code for how I can use the resnext101 backbone?


@juyunsang My understanding was that you were using
```python
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
```
for ResNet50, and 
```python
backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
```
for ResNeXt101. Is it the case?
@fmassa 
Yes! 
It works well with the code below.
```
backbone = resnet_fpn_backbone('resnet50', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
```
However, changing the parameter of resnet_fpn_backbone function to resnext101_32x8d results in a *Loss is Nan error.*

Thanks for confirming.

My first bet would be that it's an issue with the FPN, because we forgot to run the weight initialization, as discussed in https://github.com/pytorch/vision/issues/2326

Could you check if implementing that fix could solve the issue for you?
@fmassa 
Thank you for reply
I change code self.children to self.modules
but error still occued..
https://github.com/pytorch/vision/blob/c2e8a00885e68ae1200eb6440f540e181d9125de/torchvision/ops/feature_pyramid_network.py#L59-L63


Ok, I think I know what's going on. The resnext implementation might have weights which are zero in the batch norm, and we might need to set `eps` in https://github.com/pytorch/vision/blob/497744b9d510ff2df756f479ee5a19fce0d579b6/torchvision/ops/misc.py#L54 to be `1e-5` to match the default values in PyTorch.

This can be done by changing the `norm_layer` in https://github.com/pytorch/vision/blob/497744b9d510ff2df756f479ee5a19fce0d579b6/torchvision/models/detection/backbone_utils.py#L44 to be a lambda with the eps set out to be 1e-5.

Can you try this out and report back?

The same thing happened to @szagoruyko with WideResNets. Changing the default value of eps in FrozenBatchNorm should be taken into account, but it will change the results for pre-trained models, and this should be checked to see how much it affects performance
I was having a somewhat similar issue and this seems to fix it.
I was trying to load the weights from a pre-trained `torchvision.models.resnet50` model into a `torchvision.models.detection.backbone_utils.resnet_fpn_backbone` for use in a FasterRCNN model. Renaming the state dict keys (`dict([('.'.join(['body'] + k.split('.')[1:]),v) for k,v in checkpoint["state_dict"].items()])`) and loading it with `strict=False` worked, but I was getting `Loss is nan, stopping training` messages during training.

Using `functools.partial` as below created a model that got through training.
```
from functools import partial
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
from torchvision.ops.misc import FrozenBatchNorm2d

FBN = partial(FrozenBatchNorm2d, eps=1E-5)
backbone = resnet_fpn_backbone('resnet50', pretrained=True, norm_layer=FBN, trainable_layers=3).cuda()
```
I think it might be time to think about changing the default in `FrozenBatchNorm2d` to more closely align with what BatchNorm in PyTorch does, we should just check that this change doesn't affect the performance of the currently trained models in any way.
Hi @fmassa, I just opened #2852 to tackle this!
Hi @frgfm 

We still need to make sure that the current pre-trained models still give correct results with the new value.

@datumbox is going to be working on ensuring that this is the case.
No worries @fmassa ! Should I split the PR (one added eps to `__repr__` to avoid silent differences, and the other changing the default eps value)?
Yes please, if you could only make the `__repr__` changes in your PR it would be great.

@datumbox will be taking care of switching the default eps value in a follow-up PR
@fmassa done!

Resolving patch:
diff --git a/torchvision/ops/misc.py b/torchvision/ops/misc.py
--- a/torchvision/ops/misc.py
+++ b/torchvision/ops/misc.py
@@ -51,7 +51,7 @@ class FrozenBatchNorm2d(torch.nn.Module):
     def __init__(
         self,
         num_features: int,
-        eps: float = 0.,
+        eps: float = 1e-5,
         n: Optional[int] = None,
     ):
         # n=None for backward-compatibility


Test patch:
diff --git a/test/test_models.py b/test/test_models.py
--- a/test/test_models.py
+++ b/test/test_models.py
@@ -6,9 +6,10 @@
 import numpy as np
 from torchvision import models
 import unittest
-import traceback
 import random
 
+from torchvision.ops.misc import FrozenBatchNorm2d
+
 
 def set_rng_seed(seed):
     torch.manual_seed(seed)
@@ -149,6 +150,10 @@ def _test_detection_model(self, name, dev):
         if "retinanet" in name:
             kwargs["score_thresh"] = 0.013
         model = models.detection.__dict__[name](num_classes=50, pretrained_backbone=False, **kwargs)
+        if "keypointrcnn" in name or "retinanet" in name:
+            for module in model.modules():
+                if isinstance(module, FrozenBatchNorm2d):
+                    module.eps = 0
         model.eval().to(device=dev)
         input_shape = (3, 300, 300)
         # RNG always on CPU, to ensure x in cuda tests is bitwise identical to x in cpu tests
diff --git a/test/test_ops.py b/test/test_ops.py
--- a/test/test_ops.py
+++ b/test/test_ops.py
@@ -623,10 +623,10 @@ def test_frozenbatchnorm2d_eps(self):
                           running_var=torch.rand(sample_size[1]),
                           num_batches_tracked=torch.tensor(100))
 
-        # Check that default eps is zero for backward-compatibility
+        # Check that default eps is equal to the one of BN
         fbn = ops.misc.FrozenBatchNorm2d(sample_size[1])
         fbn.load_state_dict(state_dict, strict=False)
-        bn = torch.nn.BatchNorm2d(sample_size[1], eps=0).eval()
+        bn = torch.nn.BatchNorm2d(sample_size[1]).eval()
         bn.load_state_dict(state_dict)
         # Difference is expected to fall in an acceptable range
         self.assertTrue(torch.allclose(fbn(x), bn(x), atol=1e-6))