Repository: ESMValGroup/ESMValCore

Issue:
~/.esmvaltool/config-user.yml is always read even when --config-file is specified
**Describe the bug**

If a user specified a custom user configuration file via `--config-file`, the default user configuration file at `~/.esmvaltool/config-user.yml` is [still read](https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/config/_config_object.py#L242). This can lead to very confusing results, e.g., if an invalid option is present in `~/.esmvaltool/config-user.yml` but not in the user-defined file, there will be an error which is very hard to solve for users without a deep knowledge of the tool.

It would be really nice if only the user-defined configuration file is read.

I guess this is a side effect of how the configuration is provided for the (experimental) API. @ESMValGroup/technical-lead-development-team does anyone have an idea how to solve this in a backwards-compatible way without too many changes? I can't think of an easy solution to this at the moment, so this might be something to consider when we rewrite our configuration file/code (see https://github.com/ESMValGroup/ESMValCore/issues/795).

Discussion:
schlunma: Here is my though process regarding this PR:

1. The first time in the code where we can properly access the `config-file` from the command line is in [`esmvalcore._main.ESMValTool.run`](https://github.com/ESMValGroup/ESMValCore/blob/989839f596a6b4433731c0efcb6b5cb4f3ca7045/esmvalcore/_main.py#L333). However, at that time, the `CFG` is already populated by an indirect import of `esmvalcore.config` earlier in [`esmvalcore._main.ESMValTool.__init__`](https://github.com/ESMValGroup/ESMValCore/blob/989839f596a6b4433731c0efcb6b5cb4f3ca7045/esmvalcore/_main.py#L303) (which actually happens when populating the entry points with the CMORizer CLI from ESMValTool).
2. Thus, I chose to read the CLI arguments directly via `sys.argv` in [`esmvalcore.config`](https://github.com/ESMValGroup/ESMValCore/blob/7d20d6e9cb64f2eedf16d0a5346088b5606109c6/esmvalcore/config/_config_object.py#L243). This feels very hacky, but ensures that the correct config file is used, regardless of when this module is imported (which might change in the future). This works well for the entire preprocessor.
3. However, when [running the diagnostics](https://github.com/ESMValGroup/ESMValCore/blob/7d20d6e9cb64f2eedf16d0a5346088b5606109c6/esmvalcore/_task.py#L523), no `--config-file` CLI argument is present, so this is not working. My thought was now to set an environment variable the first time `sys.argv` is parsed right at the beginning. That way, also the diagnostics have access to the correct config file.

All in all, these feels very brittle and not at all robust, but it's the best I can think of that the moment :sweat_smile: : 
schlunma: All right, this is working now. Need to add doc, tests, and clean a bit, then we should be ready to go.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) Report
Attention: `15 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`5171ee3`)](https://app.codecov.io/gh/ESMValGroup/ESMValCore/commit/5171ee3d7fd1a02b0bbc35ec737d81f718f4d60a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) 93.86% compared to head [(`cc9b708`)](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) 93.79%.

| [Files](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) | Patch % | Lines |
|---|---|---|
| [esmvalcore/config/\_config\_object.py](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9jb25maWcvX2NvbmZpZ19vYmplY3QucHk=) | 76.56% | [15 Missing :warning: ](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2309      +/-   ##
==========================================
- Coverage   93.86%   93.79%   -0.07%     
==========================================
  Files         240      240              
  Lines       13293    13321      +28     
==========================================
+ Hits        12477    12495      +18     
- Misses        816      826      +10     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2309?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup).

schlunma: > what happens when simply does `import esmvalcore` - that does not, on its own, go through `_main.py` and does not dynamically import the environment set up via esmvalcore's installation = so my feeling is that an environ variable exposed to the user at a mere import call needs be put in an eg activation script like they did with ESMF_MKFILE

If you just `import esmvalcore` (or more specifically `from esmvalcore.config import CFG`) in a script or a notebook the code tries to read the default user config at `~/.esmvaltool/config-user.yml`, and if that does not exist uses the default config. This is exactly the behavior we have right now (and the one we want).

We don't want anyone apart from the ESMValTool main process to set the `_ESMVALTOOL_USER_CONFIG_FILE_` manually. The only purpose of this is that child processes of the main process use the same configuration file as the main process (without that, child processes would again default to `~/.esmvaltool/config-user.yml`, which we don't want, see #2280). That's also the reason why there is no public documentation of this environment variable. 
schlunma: This is now ready to merge from my side
valeriupredoi: good man Manu! Let me have a testy-test done, then I'll approve then merge :beers: 
valeriupredoi: `esmvalcore`-only (dev installed on its own) tests pass beautifully! I am now running a set of tests with a dev installed `esmvaltool` and dev installed `esmvaltool` off this branch - if things go according to plan I will be rejoicing and praising Lord Manu :grin: 
valeriupredoi: hang tight, got distracted with S3 stuffs - on to it in a minute

Patch:
diff --git a/esmvalcore/_main.py b/esmvalcore/_main.py
index afb0cc047..1fa4e80ce 100755
--- a/esmvalcore/_main.py
+++ b/esmvalcore/_main.py
@@ -353,8 +353,10 @@ class ESMValTool():
             Recipe to run, as either the name of an installed recipe or the
             path to a non-installed one.
         config_file: str, optional
-            Configuration file to use. If not provided the file
-            ${HOME}/.esmvaltool/config-user.yml will be used.
+            Configuration file to use. Can be given as absolute or relative
+            path. In the latter case, search in the current working directory
+            and `${HOME}/.esmvaltool` (in that order). If not provided, the
+            file `${HOME}/.esmvaltool/config-user.yml` will be used.
         resume_from: list(str), optional
             Resume one or more previous runs by using preprocessor output files
             from these output directories.
@@ -383,9 +385,15 @@ class ESMValTool():
         """
         from .config import CFG
 
+        # At this point, --config_file is already parsed if a valid file has
+        # been given (see
+        # https://github.com/ESMValGroup/ESMValCore/issues/2280), but no error
+        # has been raised if the file does not exist. Thus, reload the file
+        # here with `load_from_file` to make sure a proper error is raised.
+        CFG.load_from_file(config_file)
+
         recipe = self._get_recipe(recipe)
 
-        CFG.load_from_file(config_file)
         session = CFG.start_session(recipe.stem)
         if check_level is not None:
             session['check_level'] = check_level
diff --git a/esmvalcore/config/__init__.py b/esmvalcore/config/__init__.py
index 74bcc486d..7c2b8e379 100644
--- a/esmvalcore/config/__init__.py
+++ b/esmvalcore/config/__init__.py
@@ -4,8 +4,10 @@
 
     ESMValCore configuration.
 
-    By default this will be loaded from the file
-    ``~/.esmvaltool/config-user.yml``.
+    By default, this will be loaded from the file
+    ``~/.esmvaltool/config-user.yml``. If used within the ``esmvaltool``
+    program, this will respect the ``--config_file`` argument.
+
 """
 
 from ._config_object import CFG, Config, Session
diff --git a/esmvalcore/config/_config_object.py b/esmvalcore/config/_config_object.py
index 0b9c59617..92bd409a0 100644
--- a/esmvalcore/config/_config_object.py
+++ b/esmvalcore/config/_config_object.py
@@ -1,15 +1,18 @@
 """Importable config object."""
+from __future__ import annotations
 
 import os
+import sys
 from datetime import datetime
 from pathlib import Path
 from types import MappingProxyType
-from typing import Optional, Union
+from typing import Optional
 
 import yaml
 
 import esmvalcore
 from esmvalcore.cmor.check import CheckLevels
+from esmvalcore.exceptions import InvalidConfigParameter
 
 from ._config_validators import (
     _deprecated_options_defaults,
@@ -27,7 +30,9 @@ class Config(ValidatedConfig):
 
     Do not instantiate this class directly, but use
     :obj:`esmvalcore.config.CFG` instead.
+
     """
+    _DEFAULT_USER_CONFIG_DIR = Path.home() / '.esmvaltool'
 
     _validate = _validators
     _deprecate = _deprecators
@@ -38,49 +43,65 @@ class Config(ValidatedConfig):
     )
 
     @classmethod
-    def _load_user_config(cls,
-                          filename: Union[os.PathLike, str],
-                          raise_exception: bool = True):
+    def _load_user_config(
+        cls,
+        filename: Optional[os.PathLike | str] = None,
+        raise_exception: bool = True,
+    ):
         """Load user configuration from the given file.
 
         The config is cleared and updated in-place.
 
         Parameters
         ----------
-        filename: pathlike
-            Name of the config file, must be yaml format
+        filename:
+            Name of the user configuration file (must be YAML format). If
+            `None`, use the rules given in `Config._get_config_user_path` to
+            determine the path.
         raise_exception : bool
-            Raise an exception if `filename` can not be found (default).
-            Otherwise, silently pass and use the default configuration. This
-            setting is necessary for the case where
-            `.esmvaltool/config-user.yml` has not been defined (i.e. first
-            start).
+            If ``True``, raise an exception if `filename` cannot be found.  If
+            ``False``, silently pass and use the default configuration. This
+            setting is necessary during the loading of this module when no
+            configuration file is given (relevant if used within a script or
+            notebook).
         """
         new = cls()
+        new.update(CFG_DEFAULT)
+
+        config_user_path = cls._get_config_user_path(filename)
 
         try:
-            mapping = _read_config_file(filename)
-            mapping['config_file'] = filename
-        except IOError:
+            mapping = cls._read_config_file(config_user_path)
+            mapping['config_file'] = config_user_path
+        except FileNotFoundError:
             if raise_exception:
                 raise
             mapping = {}
 
-        new.update(CFG_DEFAULT)
-        new.update(mapping)
-        new.check_missing()
+        try:
+            new.update(mapping)
+            new.check_missing()
+        except InvalidConfigParameter as exc:
+            raise InvalidConfigParameter(
+                f"Failed to parse user configuration file {config_user_path}: "
+                f"{str(exc)}"
+            ) from exc
 
         return new
 
     @classmethod
-    def _load_default_config(cls, filename: Union[os.PathLike, str]):
+    def _load_default_config(cls):
         """Load the default configuration."""
         new = cls()
 
-        mapping = _read_config_file(filename)
+        package_config_user_path = Path(
+            esmvalcore.__file__
+        ).parent / 'config-user.yml'
+        mapping = cls._read_config_file(package_config_user_path)
+
         # Add defaults that are not available in esmvalcore/config-user.yml
         mapping['check_level'] = CheckLevels.DEFAULT
-        mapping['config_file'] = filename
+        mapping['config_file'] = package_config_user_path
         mapping['diagnostics'] = None
         mapping['extra_facets_dir'] = tuple()
         mapping['max_datasets'] = None
@@ -93,29 +114,142 @@ class Config(ValidatedConfig):
 
         return new
 
+    @staticmethod
+    def _read_config_file(config_user_path: Path) -> dict:
+        """Read configuration file and store settings in a dictionary."""
+        if not config_user_path.is_file():
+            raise FileNotFoundError(
+                f"Config file '{config_user_path}' does not exist"
+            )
+
+        with open(config_user_path, 'r', encoding='utf-8') as file:
+            cfg = yaml.safe_load(file)
+
+        return cfg
+
+    @staticmethod
+    def _get_config_user_path(
+        filename: Optional[os.PathLike | str] = None
+    ) -> Path:
+        """Get path to user configuration file.
+
+        `filename` can be given as absolute or relative path. In the latter
+        case, search in the current working directory and `~/.esmvaltool` (in
+        that order).
+
+        If `filename` is not given, try to get user configuration file from the
+        following locations (sorted by descending priority):
+
+        1. Internal `_ESMVALTOOL_USER_CONFIG_FILE_` environment variable
+           (this ensures that any subprocess spawned by the esmvaltool program
+           will use the correct user configuration file).
+        2. Command line arguments `--config-file` or `--config_file` (both
+           variants are allowed by the fire module), but only if script name is
+           `esmvaltool`.
+        3. `config-user.yml` within default ESMValTool configuration directory
+           `~/.esmvaltool`.
+
+        Note
+        ----
+        This will NOT check if the returned file actually exists to allow
+        loading the module without any configuration file (this is relevant if
+        the module is used within a script or notebook). To check if the file
+        actually exists, use the method `load_from_file` (this is done when
+        using the `esmvaltool` CLI).
+
+        If used within the esmvaltool program, set the
+        _ESMVALTOOL_USER_CONFIG_FILE_ at the end of this method to make sure
+        that subsequent calls of this method (also in suprocesses) use the
+        correct user configuration file.
+
+        """
+        # (1) Try to get user configuration file from `filename` argument
+        config_user = filename
+
+        # (2) Try to get user configuration file from internal
+        # _ESMVALTOOL_USER_CONFIG_FILE_ environment variable
+        if (
+                config_user is None and
+                '_ESMVALTOOL_USER_CONFIG_FILE_' in os.environ
+        ):
+            config_user = os.environ['_ESMVALTOOL_USER_CONFIG_FILE_']
+
+        # (3) Try to get user configuration file from CLI arguments
+        if config_user is None:
+            config_user = Config._get_config_path_from_cli()
+
+        # (4) Default location
+        if config_user is None:
+            config_user = Config._DEFAULT_USER_CONFIG_DIR / 'config-user.yml'
+
+        config_user = Path(config_user).expanduser()
+
+        # Also search path relative to ~/.esmvaltool if necessary
+        if not (config_user.is_file() or config_user.is_absolute()):
+            config_user = Config._DEFAULT_USER_CONFIG_DIR / config_user
+        config_user = config_user.absolute()
+
+        # If used within the esmvaltool program, make sure that subsequent
+        # calls of this method (also in suprocesses) use the correct user
+        # configuration file
+        if Path(sys.argv[0]).name == 'esmvaltool':
+            os.environ['_ESMVALTOOL_USER_CONFIG_FILE_'] = str(config_user)
+
+        return config_user
+
+    @staticmethod
+    def _get_config_path_from_cli() -> None | str:
+        """Try to get configuration path from CLI arguments.
+
+        The hack of directly parsing the CLI arguments here (instead of using
+        the fire or argparser module) ensures that the correct user
+        configuration file is used. This will always work, regardless of when
+        this module has been imported in the code.
+
+        Note
+        ----
+        This only works if the script name is `esmvaltool`. Does not check if
+        file exists.
+
+        """
+        if Path(sys.argv[0]).name != 'esmvaltool':
+            return None
+
+        for arg in sys.argv:
+            for opt in ('--config-file', '--config_file'):
+                if opt in arg:
+                    # Parse '--config-file=/file.yml' or
+                    # '--config_file=/file.yml'
+                    partition = arg.partition('=')
+                    if partition[2]:
+                        return partition[2]
+
+                    # Parse '--config-file /file.yml' or
+                    # '--config_file /file.yml'
+                    config_idx = sys.argv.index(opt)
+                    if config_idx == len(sys.argv) - 1:  # no file given
+                        return None
+                    return sys.argv[config_idx + 1]
+
+        return None
+
     def load_from_file(
         self,
-        filename: Optional[Union[os.PathLike, str]] = None,
+        filename: Optional[os.PathLike | str] = None,
     ) -> None:
         """Load user configuration from the given file."""
-        if filename is None:
-            filename = USER_CONFIG
-        path = Path(filename).expanduser()
-        if not path.exists():
-            try_path = USER_CONFIG_DIR / filename
-            if try_path.exists():
-                path = try_path
-            else:
-                raise FileNotFoundError(f'Cannot find: `{filename}`'
-                                        f'locally or in `{try_path}`')
-
         self.clear()
-        self.update(Config._load_user_config(path))
+        self.update(Config._load_user_config(filename))
 
     def reload(self):
         """Reload the config file."""
-        filename = self.get('config_file', DEFAULT_CONFIG)
-        self.load_from_file(filename)
+        if 'config_file' not in self:
+            raise ValueError(
+                "Cannot reload configuration, option 'config_file' is "
+                "missing; make sure to only use the `CFG` object from the "
+                "`esmvalcore.config` module"
+            )
+        self.load_from_file(self['config_file'])
 
     def start_session(self, name: str):
         """Start a new session from this configuration object.
@@ -161,7 +295,7 @@ class Session(ValidatedConfig):
 
     def __init__(self, config: dict, name: str = 'session'):
         super().__init__(config)
-        self.session_name: Union[str, None] = None
+        self.session_name: str | None = None
         self.set_session_name(name)
 
     def set_session_name(self, name: str = 'session'):
@@ -201,7 +335,7 @@ class Session(ValidatedConfig):
     @property
     def config_dir(self):
         """Return user config directory."""
-        return USER_CONFIG_DIR
+        return Path(self['config_file']).parent
 
     @property
     def main_log(self):
@@ -219,24 +353,6 @@ class Session(ValidatedConfig):
         return self.session_dir / self._relative_fixed_file_dir
 
 
-def _read_config_file(config_file):
-    """Read config user file and store settings in a dictionary."""
-    config_file = Path(config_file)
-    if not config_file.exists():
-        raise IOError(f'Config file `{config_file}` does not exist.')
-
-    with open(config_file, 'r', encoding='utf-8') as file:
-        cfg = yaml.safe_load(file)
-
-    return cfg
-
-
-DEFAULT_CONFIG_DIR = Path(esmvalcore.__file__).parent
-DEFAULT_CONFIG = DEFAULT_CONFIG_DIR / 'config-user.yml'
-
-USER_CONFIG_DIR = Path.home() / '.esmvaltool'
-USER_CONFIG = USER_CONFIG_DIR / 'config-user.yml'
-
-# initialize placeholders
-CFG_DEFAULT = MappingProxyType(Config._load_default_config(DEFAULT_CONFIG))
-CFG = Config._load_user_config(USER_CONFIG, raise_exception=False)
+# Initialize configuration objects
+CFG_DEFAULT = MappingProxyType(Config._load_default_config())
+CFG = Config._load_user_config(raise_exception=False)


Test patch:
diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py
index 6372a1c04..c84002779 100644
--- a/tests/unit/config/test_config.py
+++ b/tests/unit/config/test_config.py
@@ -4,6 +4,7 @@ from pathlib import Path
 import pytest
 import yaml
 
+import esmvalcore
 from esmvalcore.cmor.check import CheckLevels
 from esmvalcore.config import CFG, _config
 from esmvalcore.config._config import (
@@ -193,6 +194,7 @@ def test_load_default_config(monkeypatch, default_config):
         'preproc_dir',
         'run_dir',
         'work_dir',
+        'config_dir',
     }
     # Check that only allowed keys are in it
     assert set(default_cfg) == set(cfg)
@@ -210,6 +212,7 @@ def test_load_default_config(monkeypatch, default_config):
     for path in ('preproc', 'work', 'run'):
         assert getattr(cfg, path + '_dir') == cfg.session_dir / path
     assert cfg.plot_dir == cfg.session_dir / 'plots'
+    assert cfg.config_dir == Path(esmvalcore.__file__).parent
 
     # Check that projects were configured
     assert project_cfg
diff --git a/tests/unit/config/test_config_object.py b/tests/unit/config/test_config_object.py
index 9e916ff95..16084effc 100644
--- a/tests/unit/config/test_config_object.py
+++ b/tests/unit/config/test_config_object.py
@@ -1,10 +1,26 @@
+import contextlib
+import os
+import sys
 from collections.abc import MutableMapping
+from copy import deepcopy
 from pathlib import Path
 
 import pytest
 
-from esmvalcore.config import Config, Session, _config_object
+import esmvalcore
+import esmvalcore.config._config_object
+from esmvalcore.config import Config, Session
 from esmvalcore.exceptions import InvalidConfigParameter
+from tests.integration.test_main import arguments
+
+
+@contextlib.contextmanager
+def environment(**kwargs):
+    """Temporary environment variables."""
+    backup = deepcopy(os.environ)
+    os.environ = kwargs
+    yield
+    os.environ = backup
 
 
 def test_config_class():
@@ -57,25 +73,66 @@ def test_config_init():
 
 
 def test_load_from_file(monkeypatch):
-    default_config_user_file = Path.home() / '.esmvaltool' / 'config-user.yml'
-    assert _config_object.USER_CONFIG == default_config_user_file
-    monkeypatch.setattr(
-        _config_object,
-        'USER_CONFIG',
-        _config_object.DEFAULT_CONFIG,
-    )
+    default_config_file = Path(esmvalcore.__file__).parent / 'config-user.yml'
     config = Config()
     assert not config
-    config.load_from_file()
+    config.load_from_file(default_config_file)
     assert config
 
 
+def test_load_from_file_filenotfound(monkeypatch):
+    """Test `Config.load_from_file`."""
+    config = Config()
+    assert not config
+
+    expected_path = Path.home() / '.esmvaltool' / 'not_existent_file.yml'
+    msg = f"Config file '{expected_path}' does not exist"
+    with pytest.raises(FileNotFoundError, match=msg):
+        config.load_from_file('not_existent_file.yml')
+
+
+def test_load_from_file_invalidconfigparameter(monkeypatch, tmp_path):
+    """Test `Config.load_from_file`."""
+    monkeypatch.chdir(tmp_path)
+    cfg_path = tmp_path / 'test.yml'
+    cfg_path.write_text('invalid_param: 42')
+
+    config = Config()
+    assert not config
+
+    msg = (
+        f"Failed to parse user configuration file {cfg_path}: `invalid_param` "
+        f"is not a valid config parameter."
+    )
+    with pytest.raises(InvalidConfigParameter, match=msg):
+        config.load_from_file(cfg_path)
+
+
 def test_config_key_error():
     config = Config()
     with pytest.raises(KeyError):
         config['invalid_key']
 
 
+def test_reload():
+    """Test `Config.reload`."""
+    cfg_path = Path(esmvalcore.__file__).parent / 'config-user.yml'
+    config = Config(config_file=cfg_path)
+    config.reload()
+    assert config['config_file'] == cfg_path
+
+
+def test_reload_fail():
+    """Test `Config.reload`."""
+    config = Config()
+    msg = (
+        "Cannot reload configuration, option 'config_file' is missing; make "
+        "sure to only use the `CFG` object from the `esmvalcore.config` module"
+    )
+    with pytest.raises(ValueError, match=msg):
+        config.reload()
+
+
 def test_session():
     config = Config({'output_dir': 'config'})
 
@@ -90,3 +147,174 @@ def test_session_key_error():
     session = Session({})
     with pytest.raises(KeyError):
         session['invalid_key']
+
+
+TEST_GET_CFG_PATH = [
+    (None, None, None, '~/.esmvaltool/config-user.yml', False),
+    (
+        None,
+        None,
+        ('any_other_module', '--config_file=cli.yml'),
+        '~/.esmvaltool/config-user.yml',
+        False,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--max-parallel-tasks=4'),
+        '~/.esmvaltool/config-user.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', '--config_file'),
+        '~/.esmvaltool/config-user.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config_file=/cli.yml'),
+        '/cli.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config_file=/cli.yml'),
+        '/cli.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config-file', '/cli.yml'),
+        '/cli.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        '/cli.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config-file=relative_cli.yml'),
+        '~/.esmvaltool/relative_cli.yml',
+        True,
+    ),
+    (
+        None,
+        None,
+        ('esmvaltool', 'run', '--config-file=existing_cfg.yml'),
+        'existing_cfg.yml',
+        True,
+    ),
+    (
+        None,
+        {'_ESMVALTOOL_USER_CONFIG_FILE_': '/env.yml'},
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        '/env.yml',
+        True,
+    ),
+    (
+        None,
+        {'_ESMVALTOOL_USER_CONFIG_FILE_': '/env.yml'},
+        None,
+        '/env.yml',
+        True,
+    ),
+    (
+        None,
+        {'_ESMVALTOOL_USER_CONFIG_FILE_': 'existing_cfg.yml'},
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        'existing_cfg.yml',
+        True,
+    ),
+    (
+        '/filename.yml',
+        {'_ESMVALTOOL_USER_CONFIG_FILE_': '/env.yml'},
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        '/filename.yml',
+        True,
+    ),
+    (
+        '/filename.yml',
+        None,
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        '/filename.yml',
+        True,
+    ),
+    ('/filename.yml', None, None, '/filename.yml', False),
+    (
+        'filename.yml',
+        None,
+        None,
+        '~/.esmvaltool/filename.yml',
+        False,
+    ),
+    (
+        'existing_cfg.yml',
+        {'_ESMVALTOOL_USER_CONFIG_FILE_': '/env.yml'},
+        ('esmvaltool', 'run', '--config-file=/cli.yml'),
+        'existing_cfg.yml',
+        True,
+    ),
+]
+
+
+@pytest.mark.parametrize(
+    'filename,env,cli_args,output,env_var_set', TEST_GET_CFG_PATH
+)
+def test_get_config_user_path(
+    filename, env, cli_args, output, env_var_set, monkeypatch, tmp_path
+):
+    """Test `Config._get_config_user_path`."""
+    # Create empty test file
+    monkeypatch.chdir(tmp_path)
+    (tmp_path / 'existing_cfg.yml').write_text('')
+
+    if env is None:
+        env = {}
+    if cli_args is None:
+        cli_args = sys.argv
+
+    if output == 'existing_cfg.yml':
+        output = tmp_path / 'existing_cfg.yml'
+    else:
+        output = Path(output).expanduser()
+
+    with environment(**env), arguments(*cli_args):
+        config_path = Config._get_config_user_path(filename)
+        if env_var_set:
+            assert os.environ['_ESMVALTOOL_USER_CONFIG_FILE_'] == str(output)
+        else:
+            assert '_ESMVALTOOL_USER_CONFIG_FILE_' not in os.environ
+    assert isinstance(config_path, Path)
+    assert config_path == output
+
+
+def test_load_user_config_filenotfound():
+    """Test `Config._load_user_config`."""
+    expected_path = Path.home() / '.esmvaltool' / 'not_existent_file.yml'
+    msg = f"Config file '{expected_path}' does not exist"
+    with pytest.raises(FileNotFoundError, match=msg):
+        Config._load_user_config('not_existent_file.yml')
+
+
+def test_load_user_config_invalidconfigparameter(monkeypatch, tmp_path):
+    """Test `Config._load_user_config`."""
+    monkeypatch.chdir(tmp_path)
+    cfg_path = tmp_path / 'test.yml'
+    cfg_path.write_text('invalid_param: 42')
+
+    msg = (
+        f"Failed to parse user configuration file {cfg_path}: `invalid_param` "
+        f"is not a valid config parameter."
+    )
+    with pytest.raises(InvalidConfigParameter, match=msg):
+        Config._load_user_config(cfg_path)