Repository: open-mmlab/mmengine

Issue:
[Bug] Inconsistent usage of `file_client` in `CheckpointHook`. The previous best checkpoints are not removed for file backends other than `disk`.
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmengine/issues) and [Discussions](https://github.com/open-mmlab/mmengine/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmengine).

### Environment

OrderedDict([('sys.platform', 'linux'), ('Python', '3.8.16 (default, Mar 23 2023, 17:40:49) [GCC 10.2.1 20210110]'), ('CUDA available', True), ('numpy_random_seed', 2147483648), ('GPU 0', 'Tesla V100-SXM2-16GB'), ('CUDA_HOME', None), ('GCC', 'gcc (Debian 10.2.1-6) 10.2.1 20210110'), ('PyTorch', '2.0.0+cu117'), ('PyTorch compiling details', 'PyTorch built with:\n  - GCC 9.3\n  - C++ Version: 201703\n  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - LAPACK is enabled (usually provided by MKL)\n  - NNPACK is enabled\n  - CPU capability usage: AVX2\n  - CUDA Runtime 11.7\n  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n  - CuDNN 8.5\n  - Magma 2.6.1\n  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'), ('TorchVision', '0.15.1+cu117'), ('OpenCV', '4.7.0'), ('MMEngine', '0.7.2')])

### Reproduces the problem - code sample

Config example
The example uses aws backend implemeted in [this PR.](https://github.com/open-mmlab/mmengine/pull/984) One may test with Petrel backend as well to reproduce.

```python
# Client args
# ...
beargs = dict(
    backend="aws",
    path_mapping=dict(
        {
            "./work_dirs": f"aws://bucket_name/{run_id}",
            "/app/mmdetection3d/work_dirs": f"aws://bucket_name/{run_id}",
        }
    ),
)
# ...
# Define CheckpointHook
default_hooks = dict(
    checkpoint=dict(
        type="CheckpointHook",
        save_best='some_key',
        max_keep_ckpts=1,
        backend_args=beargs
    ),
)
```

### Reproduces the problem - command or script

```shell
python tools/train.py ../config.py
```

### Reproduces the problem - error message

 Unexpected behaviour.

### Additional information

1. Expected behavior:
The system should save only one best checkpoint at a time, and if a new checkpoint becomes better than the previous one, the old checkpoint should be removed. 

2. Current behavior:
The system saves a new checkpoint in addition to the previous one, which results in multiple checkpoints being saved.

3. The reason:
The issue arises due to the wrong simultaneous definition of `backend_args` and `file_client_args`. If `backend_args` is defined, the `file_client` is set to default to `disk`. The `file_backend` is used throughout the code, except in `_save_best_checkpoint`. Hence, when the system tries to locate the previous checkpoint [at this location](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L482-L488), it searches on the local drive.

4. A proposed solution:
The simultaneous definition of `backend_args` and `file_client_args` at this location should be changed [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L260-L267) 
to 
```python
        if self.file_client_args is None:
            self.file_backend = get_file_backend(
                self.out_dir, backend_args=self.backend_args)
            self.file_client = self.file_backend
        else:
            self.file_client = FileClient.infer_client(self.file_client_args,
                                                   self.out_dir)
            self.file_backend = self.file_client
```
Additionally, `self.file_client` should be replaced with `self.file_backend` at this location [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L481-L500) to ensure that the system locates the previous checkpoint correctly.



Discussion:

Resolving patch:
diff --git a/mmengine/hooks/checkpoint_hook.py b/mmengine/hooks/checkpoint_hook.py
--- a/mmengine/hooks/checkpoint_hook.py
+++ b/mmengine/hooks/checkpoint_hook.py
@@ -479,9 +479,9 @@ def _save_best_checkpoint(self, runner, metrics) -> None:
             runner.message_hub.update_info(best_score_key, best_score)
 
             if best_ckpt_path and \
-               self.file_client.isfile(best_ckpt_path) and \
+               self.file_backend.isfile(best_ckpt_path) and \
                is_main_process():
-                self.file_client.remove(best_ckpt_path)
+                self.file_backend.remove(best_ckpt_path)
                 runner.logger.info(
                     f'The previous best checkpoint {best_ckpt_path} '
                     'is removed')
@@ -490,13 +490,13 @@ def _save_best_checkpoint(self, runner, metrics) -> None:
             # Replace illegal characters for filename with `_`
             best_ckpt_name = best_ckpt_name.replace('/', '_')
             if len(self.key_indicators) == 1:
-                self.best_ckpt_path = self.file_client.join_path(  # type: ignore # noqa: E501
+                self.best_ckpt_path = self.file_backend.join_path(  # type: ignore # noqa: E501
                     self.out_dir, best_ckpt_name)
                 runner.message_hub.update_info(runtime_best_ckpt_key,
                                                self.best_ckpt_path)
             else:
                 self.best_ckpt_path_dict[
-                    key_indicator] = self.file_client.join_path(  # type: ignore # noqa: E501
+                    key_indicator] = self.file_backend.join_path(  # type: ignore # noqa: E501
                         self.out_dir, best_ckpt_name)
                 runner.message_hub.update_info(
                     runtime_best_ckpt_key,
diff --git a/mmengine/runner/runner.py b/mmengine/runner/runner.py
--- a/mmengine/runner/runner.py
+++ b/mmengine/runner/runner.py
@@ -2191,7 +2191,11 @@ def save_checkpoint(
                     checkpoint['param_schedulers'].append(state_dict)
 
         self.call_hook('before_save_checkpoint', checkpoint=checkpoint)
-        save_checkpoint(checkpoint, filepath)
+        save_checkpoint(
+            checkpoint,
+            filepath,
+            file_client_args=file_client_args,
+            backend_args=backend_args)
 
     @master_only
     def dump_config(self) -> None:


Test patch:
diff --git a/mmengine/testing/runner_test_case.py b/mmengine/testing/runner_test_case.py
--- a/mmengine/testing/runner_test_case.py
+++ b/mmengine/testing/runner_test_case.py
@@ -2,6 +2,7 @@
 import copy
 import logging
 import os
+import shutil
 import tempfile
 import time
 from unittest import TestCase
@@ -184,3 +185,12 @@ def setup_dist_env(self):
         os.environ['RANK'] = self.dist_cfg['RANK']
         os.environ['WORLD_SIZE'] = self.dist_cfg['WORLD_SIZE']
         os.environ['LOCAL_RANK'] = self.dist_cfg['LOCAL_RANK']
+
+    def clear_work_dir(self):
+        logging.shutdown()
+        for filename in os.listdir(self.temp_dir.name):
+            filepath = os.path.join(self.temp_dir.name, filename)
+            if os.path.isfile(filepath):
+                os.remove(filepath)
+            else:
+                shutil.rmtree(filepath)
diff --git a/tests/test_hooks/test_checkpoint_hook.py b/tests/test_hooks/test_checkpoint_hook.py
--- a/tests/test_hooks/test_checkpoint_hook.py
+++ b/tests/test_hooks/test_checkpoint_hook.py
@@ -3,6 +3,8 @@
 import os
 import os.path as osp
 import re
+import sys
+from unittest.mock import MagicMock, patch
 
 import torch
 from parameterized import parameterized
@@ -312,6 +314,54 @@ def test_after_val_epoch(self):
         self.assertFalse(
             osp.isfile(osp.join(runner.work_dir, 'last_checkpoint')))
 
+        # There should only one best checkpoint be reserved
+        # dist backend
+        for by_epoch, cfg in [(True, self.epoch_based_cfg),
+                              (False, self.iter_based_cfg)]:
+            self.clear_work_dir()
+            cfg = copy.deepcopy(cfg)
+            runner = self.build_runner(cfg)
+            checkpoint_hook = CheckpointHook(
+                interval=2, by_epoch=by_epoch, save_best='acc')
+            checkpoint_hook.before_train(runner)
+            checkpoint_hook.after_val_epoch(runner, metrics)
+            all_files = os.listdir(runner.work_dir)
+            best_ckpts = [
+                file for file in all_files if file.startswith('best')
+            ]
+            self.assertTrue(len(best_ckpts) == 1)
+
+        # petrel backend
+        # TODO use real petrel oss bucket to test
+        petrel_client = MagicMock()
+        for by_epoch, cfg in [(True, self.epoch_based_cfg),
+                              (False, self.iter_based_cfg)]:
+            isfile = MagicMock(return_value=True)
+            self.clear_work_dir()
+            with patch.dict(sys.modules, {'petrel_client': petrel_client}), \
+                 patch('mmengine.fileio.backends.PetrelBackend.put') as put_mock, \
+                 patch('mmengine.fileio.backends.PetrelBackend.remove') as remove_mock, \
+                 patch('mmengine.fileio.backends.PetrelBackend.isfile') as isfile:  # noqa: E501
+                cfg = copy.deepcopy(cfg)
+                runner = self.build_runner(cfg)
+                metrics = dict(acc=0.5)
+                petrel_client.client.Client = MagicMock(
+                    return_value=petrel_client)
+                checkpoint_hook = CheckpointHook(
+                    interval=2,
+                    by_epoch=by_epoch,
+                    save_best='acc',
+                    backend_args=dict(backend='petrel'))
+                checkpoint_hook.before_train(runner)
+                checkpoint_hook.after_val_epoch(runner, metrics)
+                put_mock.assert_called_once()
+                metrics['acc'] += 0.1
+                runner.train_loop._epoch += 1
+                runner.train_loop._iter += 1
+                checkpoint_hook.after_val_epoch(runner, metrics)
+                isfile.assert_called_once()
+                remove_mock.assert_called_once()
+
     def test_after_train_epoch(self):
         cfg = copy.deepcopy(self.epoch_based_cfg)
         runner = self.build_runner(cfg)
diff --git a/tests/test_runner/test_checkpoint.py b/tests/test_runner/test_checkpoint.py
--- a/tests/test_runner/test_checkpoint.py
+++ b/tests/test_runner/test_checkpoint.py
@@ -20,9 +20,6 @@
                                         load_from_local, load_from_pavi,
                                         save_checkpoint)
 
-sys.modules['petrel_client'] = MagicMock()
-sys.modules['petrel_client.client'] = MagicMock()
-
 
 @MODEL_WRAPPERS.register_module()
 class DDPWrapper:
@@ -150,9 +147,8 @@ def test_get_state_dict():
                         wrapped_model.module.conv.module.bias)
 
 
+@patch.dict(sys.modules, {'pavi': MagicMock()})
 def test_load_pavimodel_dist():
-    sys.modules['pavi'] = MagicMock()
-    sys.modules['pavi.modelcloud'] = MagicMock()
     pavimodel = Mockpavimodel()
     import pavi
     pavi.modelcloud.get = MagicMock(return_value=pavimodel)
@@ -296,6 +292,7 @@ def _load_from_state_dict(self, state_dict, prefix, local_metadata,
     assert torch.allclose(model_v2.conv1.weight, model_v2_conv1_weight)
 
 
+@patch.dict(sys.modules, {'petrel_client': MagicMock()})
 def test_checkpoint_loader():
     filenames = [
         'http://xx.xx/xx.pth', 'https://xx.xx/xx.pth',