Repository: open-mmlab/mmengine

Issue:
[Bug] `state_dict` of saved checkpoints no longer contains `_metadata` attribute
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmengine/issues) and [Discussions](https://github.com/open-mmlab/mmengine/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmengine).

### Environment

OrderedDict([('sys.platform', 'linux'), ('Python', '3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:08:06) [GCC 11.3.0]'), ('CUDA available', True), ('numpy_random_seed', 2147483648), ('GPU 0,1,2,3,4,5,6,7', 'NVIDIA GeForce RTX 3090'), ('CUDA_HOME', '/usr/local/cuda-11.8'), ('NVCC', 'Cuda compilation tools, release 11.8, V11.8.89'), ('GCC', 'gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0'), ('PyTorch', '2.0.0'), ('PyTorch compiling details', 'PyTorch built with:\n  - GCC 9.3\n  - C++ Version: 201703\n  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - LAPACK is enabled (usually provided by MKL)\n  - NNPACK is enabled\n  - CPU capability usage: AVX2\n  - CUDA Runtime 11.8\n  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n  - CuDNN 8.7\n  - Magma 2.6.1\n  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'), ('TorchVision', '0.15.0'), ('OpenCV', '4.7.0'), ('MMEngine', '0.7.3')])

### Reproduces the problem - code sample

save checkpoint while do any training with any `mmengine`-based framework, say `mmdetection3d`


### Reproduces the problem - command or script

in `mmdetection3d` folder
```shell
python tools/train.py mmdetection3d/configs/centerpoint/centerpoint_voxel01_second_secfpn_8xb4-cyclic-20e_nus-3d.py --cfg-options 'default_hooks.checkpoint.interval=1' --cfg-options 'default_hooks.checkpoint.by_epoch=False'
# This will save a checkpoint after one iter
```
trying to read the `metadata` of  `state_dict` saved checkpoint will  return an empty `OrderedDict()`
```shell
python -c 'import torch; ckpt=torch.load("work_dirs/centerpoint_voxel01_second_secfpn_8xb4-cyclic-20e_nus-3d/iter_1.pth"); print(ckpt["state_dict"]._metadata)'
```

### Reproduces the problem - error message

trying to read the `metadata` of  `state_dict` saved checkpoint will  return an empty `OrderedDict()`
```
OrderedDict()
```

### Additional information

1. Expected behavior:
checkpoints saved on previous versions will return a full metadata dict like:
```
OrderedDict([('', {'version': 1}), ('data_preprocessor', {'version': 1}), ('data_preprocessor.voxel_layer', {'version': 1}), ('pts_voxel_encoder', {'version': 1}), ('pts_middle_encoder', {'version': 1}), ('pts_middle_encoder.conv_input', {'version': 2}), ('pts_middle_encoder.conv_input.0', {'version': 2}), ('pts_middle_encoder.conv_input.1', {'version': 2}), ('pts_middle_encoder.conv_input.2', {'version': 1}), ('pts_middle_encoder.encoder_layers', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.0.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.1.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer1.2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.2.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.2.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer1.2.2', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.0.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.1.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer2.2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.2.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.2.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer2.2.2', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer3', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.0.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.1.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer3.2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.2.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.2.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer3.2.2', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer4', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.0.relu', {'version': 1}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1.conv1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1.bn1', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1.conv2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1.bn2', {'version': 2}), ('pts_middle_encoder.encoder_layers.encoder_layer4.1.relu', {'version': 1}), ('pts_middle_encoder.conv_out', {'version': 2}), ('pts_middle_encoder.conv_out.0', {'version': 2}), ('pts_middle_encoder.conv_out.1', {'version': 2}), ('pts_middle_encoder.conv_out.2', {'version': 1}), ('pts_backbone', {'version': 1}), ('pts_backbone.blocks', {'version': 1}), ('pts_backbone.blocks.0', {'version': 1}), ('pts_backbone.blocks.0.0', {'version': 1}), ('pts_backbone.blocks.0.1', {'version': 2}), ('pts_backbone.blocks.0.2', {'version': 1}), ('pts_backbone.blocks.0.3', {'version': 1}), ('pts_backbone.blocks.0.4', {'version': 2}), ('pts_backbone.blocks.0.5', {'version': 1}), ('pts_backbone.blocks.0.6', {'version': 1}), ('pts_backbone.blocks.0.7', {'version': 2}), ('pts_backbone.blocks.0.8', {'version': 1}), ('pts_backbone.blocks.0.9', {'version': 1}), ('pts_backbone.blocks.0.10', {'version': 2}), ('pts_backbone.blocks.0.11', {'version': 1}), ('pts_backbone.blocks.0.12', {'version': 1}), ('pts_backbone.blocks.0.13', {'version': 2}), ('pts_backbone.blocks.0.14', {'version': 1}), ('pts_backbone.blocks.0.15', {'version': 1}), ('pts_backbone.blocks.0.16', {'version': 2}), ('pts_backbone.blocks.0.17', {'version': 1}), ('pts_backbone.blocks.1', {'version': 1}), ('pts_backbone.blocks.1.0', {'version': 1}), ('pts_backbone.blocks.1.1', {'version': 2}), ('pts_backbone.blocks.1.2', {'version': 1}), ('pts_backbone.blocks.1.3', {'version': 1}), ('pts_backbone.blocks.1.4', {'version': 2}), ('pts_backbone.blocks.1.5', {'version': 1}), ('pts_backbone.blocks.1.6', {'version': 1}), ('pts_backbone.blocks.1.7', {'version': 2}), ('pts_backbone.blocks.1.8', {'version': 1}), ('pts_backbone.blocks.1.9', {'version': 1}), ('pts_backbone.blocks.1.10', {'version': 2}), ('pts_backbone.blocks.1.11', {'version': 1}), ('pts_backbone.blocks.1.12', {'version': 1}), ('pts_backbone.blocks.1.13', {'version': 2}), ('pts_backbone.blocks.1.14', {'version': 1}), ('pts_backbone.blocks.1.15', {'version': 1}), ('pts_backbone.blocks.1.16', {'version': 2}), ('pts_backbone.blocks.1.17', {'version': 1}), ('pts_neck', {'version': 1}), ('pts_neck.deblocks', {'version': 1}), ('pts_neck.deblocks.0', {'version': 1}), ('pts_neck.deblocks.0.0', {'version': 1}), ('pts_neck.deblocks.0.1', {'version': 2}), ('pts_neck.deblocks.0.2', {'version': 1}), ('pts_neck.deblocks.1', {'version': 1}), ('pts_neck.deblocks.1.0', {'version': 1}), ('pts_neck.deblocks.1.1', {'version': 2}), ('pts_neck.deblocks.1.2', {'version': 1}), ('pts_bbox_head', {'version': 1}), ('pts_bbox_head.loss_cls', {'version': 1}), ('pts_bbox_head.loss_bbox', {'version': 1}), ('pts_bbox_head.shared_conv', {'version': 1}), ('pts_bbox_head.shared_conv.conv', {'version': 1}), ('pts_bbox_head.shared_conv.bn', {'version': 2}), ('pts_bbox_head.shared_conv.activate', {'version': 1}), ('pts_bbox_head.task_heads', {'version': 1}), ('pts_bbox_head.task_heads.0', {'version': 1}), ('pts_bbox_head.task_heads.0.reg', {'version': 1}), ('pts_bbox_head.task_heads.0.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.0.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.0.height', {'version': 1}), ('pts_bbox_head.task_heads.0.height.0', {'version': 1}), ('pts_bbox_head.task_heads.0.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.height.1', {'version': 1}), ('pts_bbox_head.task_heads.0.dim', {'version': 1}), ('pts_bbox_head.task_heads.0.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.0.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.0.rot', {'version': 1}), ('pts_bbox_head.task_heads.0.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.0.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.0.vel', {'version': 1}), ('pts_bbox_head.task_heads.0.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.0.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.0.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.0.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.0.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.0.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.0.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.0.heatmap.1', {'version': 1}), ('pts_bbox_head.task_heads.1', {'version': 1}), ('pts_bbox_head.task_heads.1.reg', {'version': 1}), ('pts_bbox_head.task_heads.1.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.1.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.1.height', {'version': 1}), ('pts_bbox_head.task_heads.1.height.0', {'version': 1}), ('pts_bbox_head.task_heads.1.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.height.1', {'version': 1}), ('pts_bbox_head.task_heads.1.dim', {'version': 1}), ('pts_bbox_head.task_heads.1.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.1.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.1.rot', {'version': 1}), ('pts_bbox_head.task_heads.1.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.1.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.1.vel', {'version': 1}), ('pts_bbox_head.task_heads.1.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.1.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.1.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.1.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.1.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.1.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.1.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.1.heatmap.1', {'version': 1}), ('pts_bbox_head.task_heads.2', {'version': 1}), ('pts_bbox_head.task_heads.2.reg', {'version': 1}), ('pts_bbox_head.task_heads.2.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.2.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.2.height', {'version': 1}), ('pts_bbox_head.task_heads.2.height.0', {'version': 1}), ('pts_bbox_head.task_heads.2.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.height.1', {'version': 1}), ('pts_bbox_head.task_heads.2.dim', {'version': 1}), ('pts_bbox_head.task_heads.2.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.2.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.2.rot', {'version': 1}), ('pts_bbox_head.task_heads.2.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.2.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.2.vel', {'version': 1}), ('pts_bbox_head.task_heads.2.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.2.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.2.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.2.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.2.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.2.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.2.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.2.heatmap.1', {'version': 1}), ('pts_bbox_head.task_heads.3', {'version': 1}), ('pts_bbox_head.task_heads.3.reg', {'version': 1}), ('pts_bbox_head.task_heads.3.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.3.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.3.height', {'version': 1}), ('pts_bbox_head.task_heads.3.height.0', {'version': 1}), ('pts_bbox_head.task_heads.3.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.height.1', {'version': 1}), ('pts_bbox_head.task_heads.3.dim', {'version': 1}), ('pts_bbox_head.task_heads.3.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.3.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.3.rot', {'version': 1}), ('pts_bbox_head.task_heads.3.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.3.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.3.vel', {'version': 1}), ('pts_bbox_head.task_heads.3.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.3.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.3.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.3.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.3.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.3.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.3.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.3.heatmap.1', {'version': 1}), ('pts_bbox_head.task_heads.4', {'version': 1}), ('pts_bbox_head.task_heads.4.reg', {'version': 1}), ('pts_bbox_head.task_heads.4.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.4.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.4.height', {'version': 1}), ('pts_bbox_head.task_heads.4.height.0', {'version': 1}), ('pts_bbox_head.task_heads.4.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.height.1', {'version': 1}), ('pts_bbox_head.task_heads.4.dim', {'version': 1}), ('pts_bbox_head.task_heads.4.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.4.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.4.rot', {'version': 1}), ('pts_bbox_head.task_heads.4.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.4.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.4.vel', {'version': 1}), ('pts_bbox_head.task_heads.4.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.4.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.4.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.4.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.4.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.4.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.4.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.4.heatmap.1', {'version': 1}), ('pts_bbox_head.task_heads.5', {'version': 1}), ('pts_bbox_head.task_heads.5.reg', {'version': 1}), ('pts_bbox_head.task_heads.5.reg.0', {'version': 1}), ('pts_bbox_head.task_heads.5.reg.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.reg.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.reg.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.reg.1', {'version': 1}), ('pts_bbox_head.task_heads.5.height', {'version': 1}), ('pts_bbox_head.task_heads.5.height.0', {'version': 1}), ('pts_bbox_head.task_heads.5.height.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.height.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.height.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.height.1', {'version': 1}), ('pts_bbox_head.task_heads.5.dim', {'version': 1}), ('pts_bbox_head.task_heads.5.dim.0', {'version': 1}), ('pts_bbox_head.task_heads.5.dim.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.dim.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.dim.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.dim.1', {'version': 1}), ('pts_bbox_head.task_heads.5.rot', {'version': 1}), ('pts_bbox_head.task_heads.5.rot.0', {'version': 1}), ('pts_bbox_head.task_heads.5.rot.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.rot.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.rot.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.rot.1', {'version': 1}), ('pts_bbox_head.task_heads.5.vel', {'version': 1}), ('pts_bbox_head.task_heads.5.vel.0', {'version': 1}), ('pts_bbox_head.task_heads.5.vel.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.vel.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.vel.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.vel.1', {'version': 1}), ('pts_bbox_head.task_heads.5.heatmap', {'version': 1}), ('pts_bbox_head.task_heads.5.heatmap.0', {'version': 1}), ('pts_bbox_head.task_heads.5.heatmap.0.conv', {'version': 1}), ('pts_bbox_head.task_heads.5.heatmap.0.bn', {'version': 2}), ('pts_bbox_head.task_heads.5.heatmap.0.activate', {'version': 1}), ('pts_bbox_head.task_heads.5.heatmap.1', {'version': 1})])
```

2. Possible cause:
When saving checkpoints, `weights_to_cpu()` is called to move the `state_dict` of model to `cpu`. However, the function in its current implementation will clear the `_metadata` of the `state_dict`. 


Discussion:

Resolving patch:
diff --git a/mmengine/runner/checkpoint.py b/mmengine/runner/checkpoint.py
--- a/mmengine/runner/checkpoint.py
+++ b/mmengine/runner/checkpoint.py
@@ -652,10 +652,11 @@ def weights_to_cpu(state_dict):
     Returns:
         OrderedDict: Model weights on GPU.
     """
+    # stash metadata to put in state_dict later
+    metadata = getattr(state_dict, '_metadata', OrderedDict())
     state_dict = apply_to(state_dict, lambda x: hasattr(x, 'cpu'),
                           lambda x: x.cpu())
-    # Keep metadata in state_dict
-    state_dict._metadata = getattr(state_dict, '_metadata', OrderedDict())
+    state_dict._metadata = metadata
     return state_dict
 
 


Test patch: