Repository: open-mmlab/mmengine

Issue:
[Bug] File format checking when more than one dot in the path
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmengine/issues) and [Discussions](https://github.com/open-mmlab/mmengine/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmengine).

### Environment

OrderedDict([('sys.platform', 'linux'), ('Python', '3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0]'), ('CUDA available', True), ('numpy_random_seed', 2147483648), ('GPU 0,1,2,3', 'Tesla P100-SXM2-16GB'), ('CUDA_HOME', '/apps/cuda/11.6.2'), ('NVCC', 'Cuda compilation tools, release 11.6, V11.6.124'), ('GCC', 'gcc (SUSE Linux) 7.5.0'), ('PyTorch', '1.13.0'), ('PyTorch compiling details', 'PyTorch built with:\n  - GCC 9.3\n  - C++ Version: 201402\n  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - LAPACK is enabled (usually provided by MKL)\n  - NNPACK is enabled\n  - CPU capability usage: AVX2\n  - CUDA Runtime 11.6\n  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37\n  - CuDNN 8.3.2  (built against CUDA 11.5)\n  - Magma 2.6.1\n  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'), ('TorchVision', '0.14.0'), ('OpenCV', '4.6.0'), ('MMEngine', '0.3.2')])

### Reproduces the problem - code sample

`python demo/image_demo.py demo/demo.jpg configs/yolo/yolov3_mobilenetv2_8xb24-320-300e_coco.py yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth --device cpu --out-file result.jpg
`

### Reproduces the problem - command or script

`python demo/image_demo.py demo/demo.jpg configs/yolo/yolov3_mobilenetv2_8xb24-320-300e_coco.py yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth --device cpu --out-file result.jpg`


### Reproduces the problem - error message

Traceback (most recent call last):
  File "demo/image_demo.py", line 98, in <module>
    main(args)
  File "demo/image_demo.py", line 42, in main
    model = init_detector(
  File "/datasets/work/d61-visage-data/work/experiments/Yang/code/mmdetection/mmdet/apis/inference.py", line 47, in init_detector
    config = Config.fromfile(config)
  File "/scratch1/li309/miniconda3/envs/mmdet/lib/python3.8/site-packages/mmengine/config/config.py", line 172, in fromfile
    cfg_dict, cfg_text = Config._file2dict(filename,
  File "/scratch1/li309/miniconda3/envs/mmdet/lib/python3.8/site-packages/mmengine/config/config.py", line 401, in _file2dict
    for base_cfg_path in Config._get_base_files(temp_config_file.name):
  File "/scratch1/li309/miniconda3/envs/mmdet/lib/python3.8/site-packages/mmengine/config/config.py", line 576, in _get_base_files
    raise TypeError('The config type should be py, json, yaml or '
TypeError: The config type should be py, json, yaml or yml, but got 43629/tmpm37bjlje/tmpl5h1hmu0.py
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x15553a649940>
Traceback (most recent call last):
  File "/scratch1/li309/miniconda3/envs/mmdet/lib/python3.8/tempfile.py", line 440, in __del__
    self.close()
  File "/scratch1/li309/miniconda3/envs/mmdet/lib/python3.8/tempfile.py", line 436, in close
    unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/li309.43629/tmpm37bjlje/tmpl5h1hmu0.py'


### Additional information

Hi there,

I was running the demo script to test out mmdet framework on HPC. However, I got the issue above. When I tracked back the cause of the issue. I found out that the temp file written by mmengine cannot be detected when the temp file path has more than one dot. For example, on my HPC, my tmp file path is `/tmp/li309.43629/tmpm37bjlje/tmpl5h1hmu0.py`, the extracted file format becomes "43629/tmpm37bjlje/tmpl5h1hmu0.py" because the line 553 of the file mmengine/config/config.py uses partition() function, which only finds the first occurred dot. I fixed it by replacing it with split() function, so all "." are found and separated.

Please check if this is a bug that deserves to be fixed :)


Discussion:
Soooooo sorry for my late reply. If we use `rpartition` rather than `partition`, will this problem be solved? If it works, would mind posting a PR to fix this issue?
Thank you for the response. Yeah, I think `rpartition()` should solve the issue. And I am happy to post a PR for this issue soon ;)
Suggest using `os.path.splitext(filepath)`

Resolving patch:
diff --git a/mmengine/config/config.py b/mmengine/config/config.py
--- a/mmengine/config/config.py
+++ b/mmengine/config/config.py
@@ -550,8 +550,8 @@ def _get_base_files(filename: str) -> list:
         Returns:
             list: A list of base config.
         """
-        file_format = filename.partition('.')[-1]
-        if file_format == 'py':
+        file_format = osp.splitext(filename)[1]
+        if file_format == '.py':
             Config._validate_py_syntax(filename)
             with open(filename, encoding='utf-8') as f:
                 codes = ast.parse(f.read()).body
@@ -568,7 +568,7 @@ def is_base_line(c):
                     base_files = eval(compile(base_code, '', mode='eval'))
                 else:
                     base_files = []
-        elif file_format in ('yml', 'yaml', 'json'):
+        elif file_format in ('.yml', '.yaml', '.json'):
             import mmengine
             cfg_dict = mmengine.load(filename)
             base_files = cfg_dict.get(BASE_KEY, [])


Test patch:
diff --git a/tests/test_config/test_config.py b/tests/test_config/test_config.py
--- a/tests/test_config/test_config.py
+++ b/tests/test_config/test_config.py
@@ -5,8 +5,10 @@
 import os.path as osp
 import platform
 import sys
+import tempfile
 from importlib import import_module
 from pathlib import Path
+from unittest.mock import patch
 
 import pytest
 
@@ -715,6 +717,21 @@ def _base_variables(self):
         cfg = Config._file2dict(cfg_file)[0]
         assert cfg == dict(item1=dict(a=1))
 
+        # Simulate the case that the temporary directory includes `.`, etc.
+        # /tmp/test.axsgr12/. This patch is to check the issue
+        # https://github.com/open-mmlab/mmengine/issues/788 has been solved.
+        class PatchedTempDirectory(tempfile.TemporaryDirectory):
+
+            def __init__(self, *args, prefix='test.', **kwargs):
+                super().__init__(*args, prefix=prefix, **kwargs)
+
+        with patch('mmengine.config.config.tempfile.TemporaryDirectory',
+                   PatchedTempDirectory):
+            cfg_file = osp.join(self.data_path,
+                                'config/py_config/test_py_modify_key.py')
+            cfg = Config._file2dict(cfg_file)[0]
+            assert cfg == dict(item1=dict(a=1))
+
     def _merge_recursive_bases(self):
         cfg_file = osp.join(self.data_path,
                             'config/py_config/test_merge_recursive_bases.py')