Repository: open-mmlab/mmcv

Issue:
Unexpected behaviour `by_epoch` DvcliveLoggerHook/AnyLogger
Thanks for reporting the unexpected results and we appreciate it a lot.

See also the DVClive issue: https://github.com/iterative/dvclive/issues/267

**Describe the Issue**

I was using the `dvclive` hook for mmcv and expected the `by_epoch` variable to mean something else then it is doing right now. I expected to get a result per epoch. However, it seems to have no (or not the expected) effect. 

**Reproduction**

1. What command, code, or script did you run? I added the following to my config.

```bash
log_config = dict(
    hooks=[
        dict(
            type="DvcliveLoggerHook",
            path="{{ fileDirname }}/../live",
            interval=1,
            by_epoch=True,
        ),
    ],
)
```

2. Did you make any modifications on the code? Did you understand what you have modified?

I did not modify the code.

**Environment**

1. Please run `python -c "from mmcv.utils import collect_env; print(collect_env())"` to collect necessary environment information and paste it here.

<details><summary>Output</summary>

```none
{   
    'sys.platform': 'linux', 
    'Python': '3.8.8 (default, Feb 24 2021, 21: 46: 12) [GCC 7.3.0]',
    'CUDA available': True,
    'GPU 0, 1, 2, 3': 'NVIDIA TITAN RTX',
    'CUDA_HOME': '/usr/local/cuda',
    'NVCC': 'Cuda compilation tools, 
    release 11.2, V11.2.142',
    'GCC': 'gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0',
    'PyTorch': '1.9.0a0+df837d0', 
    'PyTorch compiling details': 'PyTorch built with:\n  - GCC 9.3\n  - C++ Version: 201402\n  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v1.7.0 (Git Hash N/A)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - NNPACK is enabled\n  - CPU capability usage: AVX2\n  - CUDA Runtime 11.2\n  - NVCC architecture flags: -gencode;arch=compute_52,
    code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,
    code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,
    code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,
    code=sm_86;-gencode;arch=compute_86,code=compute_86\n  - CuDNN 8.1.1\n  - Magma 2.5.2\n  - Build settings: BLAS_INFO=mkl,
    BUILD_TYPE=Release,
    CUDA_VERSION=11.2,
    CUDNN_VERSION=8.1.1,
    CXX_COMPILER=/usr/bin/c++,
    CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow,
    FORCE_FALLBACK_CUDA_MPI=1,
    LAPACK_INFO=mkl,
    PERF_WITH_AVX=1,
    PERF_WITH_AVX2=1,
    PERF_WITH_AVX512=1,
    TORCH_VERSION=1.9.0,
    USE_CUDA=ON,
    USE_CUDNN=ON,
    USE_EXCEPTION_PTR=1,
    USE_GFLAGS=OFF,
    USE_GLOG=OFF,
    USE_MKL=ON,
    USE_MKLDNN=ON,
    USE_MPI=ON,
    USE_NCCL=ON,
    USE_NNPACK=ON,
    USE_OPENMP=ON,
    \n',
    'TorchVision': '0.9.0a0',
    'OpenCV': '3.4.11',
    'MMCV': '1.5.0',
    'MMCV Compiler': 'GCC 9.3',
    'MMCV CUDA Compiler': '11.2'
}
```

</details>


2. You may add addition that may be helpful for locating the problem, such as
   - How you installed PyTorch \[e.g., pip, conda, source\]
   - Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)


**Error traceback**
If applicable, paste the error traceback here.

n/a

**Bug fix**
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

n/a



Discussion:
Hi, thanks for your feedback. 
Actually, except for `TextLoggerHook`, other logger hooks will only record training state by iterations, therefore your `by_epoch` will not influence your `dvclive` log.
Hi! I am saying that's why it is 'unexpected'. So the thought in the initial issue I created over at iterative/dvclive (https://github.com/iterative/dvclive/issues/267) was that they could update the behaviour of `by_epoch` themselves but that this would diverge from the default behaviour of `mmcv`. However, based on your comment now, it seems this is what is done in `TextLoggerHook` and would not be an inconsistency? 

Relevant also: What is the intended and actual use of `by_epoch`? Is `TextLoggerHook` the only one that is using it really?
TextLoggerHook Handle the `by_epoch` attribute in 
 https://github.com/open-mmlab/mmcv/blob/75ae2009545b086c8304d9d16150565ecbdc8565/mmcv/runner/hooks/logger/text.py#L163
Therefore you can control the iteration-based or epcoh-based logging format. 
Although this will work for the rest of the logger hooks as well,  it requires a lot of modification. Currently, we have no plans to add this part of the logic to the rest of the logger hooks.

Resolving patch:
diff --git a/mmcv/runner/hooks/logger/dvclive.py b/mmcv/runner/hooks/logger/dvclive.py
--- a/mmcv/runner/hooks/logger/dvclive.py
+++ b/mmcv/runner/hooks/logger/dvclive.py
@@ -21,7 +21,10 @@ class DvcliveLoggerHook(LoggerHook):
             if less than `interval`. Default: True.
         reset_flag (bool): Whether to clear the output buffer after logging.
             Default: False.
-        by_epoch (bool): Whether EpochBasedRunner is used. Default: True.
+        by_epoch (bool): Whether EpochBasedRunner is used.
+            Determines whether `log` is called `after_train_iter` or
+            `after_train_epoch`.
+            Default: True.
         dvclive (Live, optional): An instance of the `Live`_ logger to use
             instead of initializing a new one internally. Defaults to None.
         kwargs: Arguments for instantiating `Live`_ (ignored if `dvclive` is
@@ -58,16 +61,23 @@ def _import_dvclive(self, dvclive=None, **kwargs) -> None:
     def log(self, runner) -> None:
         tags = self.get_loggable_tags(runner)
         if tags:
-            self.dvclive.set_step(self.get_iter(runner))
+            step = self.get_epoch(runner) if self.by_epoch else self.get_iter(
+                runner)
+            self.dvclive.set_step(step)
+
             for k, v in tags.items():
                 self.dvclive.log(k, v)
 
-    @master_only
     def after_train_epoch(self, runner) -> None:
-        super().after_train_epoch(runner)
         if self.model_file is not None:
             runner.save_checkpoint(
                 Path(self.model_file).parent,
                 filename_tmpl=Path(self.model_file).name,
                 create_symlink=False,
             )
+        if self.by_epoch:
+            super().after_train_epoch(runner)
+
+    def after_train_iter(self, runner) -> None:
+        if not self.by_epoch:
+            super().after_train_iter(runner)


Test patch:
diff --git a/tests/test_runner/test_hooks.py b/tests/test_runner/test_hooks.py
--- a/tests/test_runner/test_hooks.py
+++ b/tests/test_runner/test_hooks.py
@@ -1644,11 +1644,12 @@ def test_neptune_hook():
     hook.run.stop.assert_called_with()
 
 
-def test_dvclive_hook():
+@pytest.mark.parametrize('by_epoch', [True, False])
+def test_dvclive_hook(by_epoch):
     sys.modules['dvclive'] = MagicMock()
     runner = _build_demo_runner()
 
-    hook = DvcliveLoggerHook()
+    hook = DvcliveLoggerHook(by_epoch=by_epoch)
     dvclive_mock = hook.dvclive
     loader = DataLoader(torch.ones((5, 2)))
 
@@ -1656,7 +1657,8 @@ def test_dvclive_hook():
     runner.run([loader, loader], [('train', 1), ('val', 1)])
     shutil.rmtree(runner.work_dir)
 
-    dvclive_mock.set_step.assert_called_with(6)
+    dvclive_mock.set_step.assert_called_with(1 if by_epoch else 6)
+    assert dvclive_mock.set_step.call_count == 1 if by_epoch else 5
     dvclive_mock.log.assert_called_with('momentum', 0.95)