Repository: open-mmlab/mmengine

Issue:
[bug] error when using `python tools/train.py` to train `ReDet` in MMRotate.
when the `self.distributed` equals to False, `revert_sync_batchnorm` function cannot work normally. 
https://github.com/open-mmlab/mmengine/blob/bda92b49e751c721c11bb15623dabc7f7a9517e7/mmengine/runner/runner.py#L848

We argue this is caused by [ReDet](https://github.com/open-mmlab/mmrotate/blob/dev-1.x/configs/redet/redet_re50_refpn_1x_dota_le90.py) replacing ``torch.nn`` with ``e2cnn.nn``.

https://github.com/open-mmlab/mmrotate/blob/dev-1.x/mmrotate/models/backbones/re_resnet.py




Discussion:
Can you provide a more detailed description of the error? Does `revert_sync_batchnorm` raise an error or not convert the `SyncBN` to `BN`?
![1667186839276](https://user-images.githubusercontent.com/10410257/198924981-6b78d56c-49ee-4507-8206-76dc1cfddcd4.png)

Yes, `revert_sync_batchnorm` raises an error when converting the `SyncBN` to `BN`.  However,  there is no `SyncBatchNorm` module in e2cnn. The error occurs by the `add_module` function. In e2cnn, [R2Conv](https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/nn/modules/r2_conv/r2convolution.py) is the smallest unit, which cannot be split.  When we try to recreate `R2Conv` by `add_module`,  this [line](https://github.com/QUVA-Lab/e2cnn/blob/d8128366f07630059b669bdcba90ce625de06c8f/e2cnn/nn/modules/sequential_module.py#L97) raise an error.  It looks like `R2Conv` don't support being created by his child modules.


Maybe,  we need to allow the user to close `revert_sync_batchnorm` in the `runner`.



It makes sense to me. Will it be more proper to fallback not revert `SyncBN` if `revert_sync_batchnorm` failed?
> It makes sense to me. Will it be more proper to fallback not revert `SyncBN` if `revert_sync_batchnorm` failed?

That sounds like a nice idea. 

Resolving patch:
diff --git a/mmengine/model/utils.py b/mmengine/model/utils.py
--- a/mmengine/model/utils.py
+++ b/mmengine/model/utils.py
@@ -7,6 +7,7 @@
 import torch.nn as nn
 import torch.nn.functional as F
 
+from mmengine.logging import print_log
 from mmengine.utils.dl_utils import mmcv_full_available
 
 
@@ -192,7 +193,17 @@ def revert_sync_batchnorm(module: nn.Module) -> nn.Module:
         if hasattr(module, 'qconfig'):
             module_output.qconfig = module.qconfig
     for name, child in module.named_children():
-        module_output.add_module(name, revert_sync_batchnorm(child))
+        # Some custom modules or 3rd party implemented modules may raise an
+        # error when calling `add_module`. Therefore, try to catch the error
+        # and do not raise it. See https://github.com/open-mmlab/mmengine/issues/638 # noqa: E501
+        # for more details.
+        try:
+            module_output.add_module(name, revert_sync_batchnorm(child))
+        except Exception:
+            print_log(
+                F'Failed to convert {child} from SyncBN to BN!',
+                logger='current',
+                level=logging.WARNING)
     del module
     return module_output
 


Test patch:
diff --git a/tests/test_model/test_model_utils.py b/tests/test_model/test_model_utils.py
--- a/tests/test_model/test_model_utils.py
+++ b/tests/test_model/test_model_utils.py
@@ -15,6 +15,16 @@
 from mmengine.utils import is_installed
 
 
+class ToyModule(nn.Module):
+
+    def __init__(self):
+        super().__init__()
+        self.layer1 = nn.Linear(1, 1)
+
+    def add_module(self, name, module):
+        raise ValueError()
+
+
 @pytest.mark.skipif(
     torch.__version__ == 'parrots', reason='not supported in parrots now')
 def test_revert_syncbn():
@@ -28,6 +38,12 @@ def test_revert_syncbn():
     y = conv(x)
     assert y.shape == (1, 8, 9, 9)
 
+    # TODO, capsys provided by `pytest` cannot capture the error log produced
+    # by MMLogger. Test the error log after refactoring the unit test with
+    # `unittest`
+    conv = nn.Sequential(ToyModule(), nn.SyncBatchNorm(8))
+    revert_sync_batchnorm(conv)
+
 
 @pytest.mark.skipif(
     torch.__version__ == 'parrots', reason='not supported in parrots now')
@@ -41,10 +57,12 @@ def test_convert_syncbn():
     # Test convert to mmcv SyncBatchNorm
     if is_installed('mmcv'):
         # MMCV SyncBatchNorm is only supported on distributed training.
+        # torch 1.6 will throw an AssertionError, and higher version will
+        # throw an RuntimeError
         with pytest.raises((RuntimeError, AssertionError)):
             convert_sync_batchnorm(conv, implementation='mmcv')
 
-    # Test convert to Pytorch SyncBatchNorm
+    # Test convert BN to Pytorch SyncBatchNorm
     # Expect a ValueError prompting that SyncBN is not supported on CPU
     converted_conv = convert_sync_batchnorm(conv)
     assert isinstance(converted_conv[1], torch.nn.SyncBatchNorm)