Repository: datalad/datalad-next

Issue:
Adjust `ConfigManager` to post overrides into the ENV
Overrides can come in via the CLI, or be directly set in a manager instance. Given that overrides are somewhat global, it makes sense to post them in the processes ENV too.

Here is a concrete example of where that would be useful https://github.com/psychoinformatics-de/knowledge-base/issues/15#issuecomment-1522907177 (instead of having to go git-native, a declaration via `datalad -c ...` would be sufficient).

This would also make a runner-specific change to pass on configuration, as proposed in https://github.com/datalad/datalad/pull/7344 (see issue https://github.com/datalad/datalad/issues/7352) unnecessary. While this would work as a solution for the clone issue linked above too, it would be limited to execution via the datalad runner. Simply posting overrides in the process ENV via `GIT_CONFIG_COUNT= GIT_CONFIG_KEY_ GIT_CONFIG_VALUE_` would be applicable to any means of executing subprocesses (e.g., plain `subprocess.run` -- something that is even done in the datalad codebase).

Issues this would address

- https://github.com/datalad/datalad/issues/3456 (closed but unresolved)
- https://github.com/datalad/datalad/issues/4119

### Challenges:

#### Interference with Git

If the Git-native variables are already present in the environment (see above), we need to ensure that any DataLad modification amends the existing setup and does not strip settings.


#### Override semantics

The nature of "overrides" in the datalad configuration is underspecified. Relevant statements from the documentation are:

- environment variables can be used to override any datalad configuration, and have precedence over any other configuration scope (any further elaboration is specific to `DATALAD_*` type variables.
- re `DATALAD_CONFIG_OVERRIDES_JSON`: f both individual configuration variables *and* JSON-overrides are used, the former take precedent over the latter, overriding the respective *individual* settings from configurations declared in the JSON-overrides
- the `configuration` command is not able to query overrides separately
- `ConfigManager` docs say: 'override' limits the modification to the `ConfigManager` instance, and the assigned value overrides any setting from any other source [remark: the use of "instance" is a concern here]
- `ConfigManager` declares: Each instance carries a public `overrides` attribute. This dictionary contains variables that override any setting read from a file. The overrides are persistent across reloads [remark: limitation to "from a file" is relevant here]
- CLI config overrides (`datalad -c`) become `ConfigManager` instance overrides


#### Potential inconsistencies

- on `ConfigManager.reload()` instance overrides are applied unconditionally to the "merged" config. But overrides from ENV only when `scope` is not `branch`.  `ConfigManager.get_from_source()` implemented the same logic.
- `ConfigManager` runs `git config`. If `GIT_CONFIG_COUNT= GIT_CONFIG_KEY_ GIT_CONFIG_VALUE_` are declared, this should apply these git-native overrides, but datalad "sees" them. It needs to be checked whether scope associations are inferred properly. Possibly, `ConfigManager` could afterwards simply replace these settings, because it did already "read them in".
  - No, that is not possible. The settings are "seen" (git-config reports them as origin='command line' , and the config manager umps them all into a `git` category. At this point they are no longer detectable as overrides. If the git-nativ declaration is then replaced, any sub-process will no longer get to see them, because they only live in the parent process's `ConfigManager`
- Overrides from ENV are not reported by `ConfigManager.repr()` (only instance overrides)


#### Threats

If the implementation is changed to post config overrides in git-native format into the ENV, we blur the distinction between ENV-overrides (cannot be manipulated via the `ConfigManager`, are applied differently than instance overrides), and instance/CLI overrides. Essentially, any instance override would affect the ENV. However, posting in Git-native format does not simply remove the distinction of instance and ENV overrides (although that may be desirable, given the complexity increase).

It remains to be evaluated, how git-native overrides interact with the attribution of origin of a configuration. Importantly, Git distinguished a dedicated scope `command` for this (separate from `system`, `global`, `local`, `worktree` (which would be most similar to our `branch`)). For this `command` scope, git-config says:

> These environment variables will override values in configuration files, but will be overridden by any explicit options passed via `git -c`

This is both similar and different from our notion of overrides. Yes, they are overrides, but CLI setting has the last word, whereras in datalad, we have ENV override CLI.

The latter could be aligned by the datalad CLI resorting to posting its `-c` setting via the Git-native mechanism into the ENV. This would give them the highest precedence for datalad-internal processing too. It would also allow for a declaration of any `datalad -c X`, in exactly the same way and with the same semantics as `git -c X`, and also result in any internal git call to behave as if called in that fashion.



Comments:

Patch:
diff --git a/.appveyor.yml b/.appveyor.yml
index 4f3e923..97005d7 100644
--- a/.appveyor.yml
+++ b/.appveyor.yml
@@ -94,7 +94,10 @@ environment:
         datalad.core
       # do not run tests that ensure behavior we intentionally changed
       # - test_gh1811: is included in next in an alternative implementation
-      KEYWORDS: not test_gh1811
+      # - test_librarymode: assumes that CLI config overrides end up in the
+      #   session `datalad.cfg.overrides`, but -next changes that behavior
+      #   to have `.overrides` be uniformly limited to instance overrides
+      KEYWORDS: not test_gh1811 and not test_librarymode
       APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu2004
       PY: 3.7
       INSTALL_SYSPKGS: python3-virtualenv
diff --git a/datalad_next/config/utils.py b/datalad_next/config/utils.py
new file mode 100644
index 0000000..053aa68
--- /dev/null
+++ b/datalad_next/config/utils.py
@@ -0,0 +1,104 @@
+from __future__ import annotations
+
+from os import environ
+from typing import (
+    Dict,
+    Mapping,
+    Tuple,
+)
+
+
+def get_gitconfig_items_from_env() -> Mapping[str, str | Tuple[str, ...]]:
+    """Parse git-config ENV (``GIT_CONFIG_COUNT|KEY|VALUE``) and return as dict
+
+    This implementation does not use ``git-config`` directly, but aims to
+    mimic its behavior with respect to parsing the environment as much
+    as possible.
+
+    Raises
+    ------
+    ValueError
+      Whenever ``git-config`` would also error out, and includes an
+      message in the respective exception that resembles ``git-config``'s
+      for that specific case.
+
+    Returns
+    -------
+    dict
+      Configuration key-value mappings. When a key is declared multiple
+      times, the respective values are aggregated in reported as a tuple
+      for that specific key.
+    """
+    items: Dict[str, str | Tuple[str, ...]] = {}
+    for k, v in ((_get_gitconfig_var_from_env(i, 'key'),
+                  _get_gitconfig_var_from_env(i, 'value'))
+                 for i in range(_get_gitconfig_itemcount())):
+        val = items.get(k)
+        if val is None:
+            items[k] = v
+        elif isinstance(val, tuple):
+            items[k] = val + (v,)
+        else:
+            items[k] = (val, v)
+    return items
+
+
+def _get_gitconfig_itemcount() -> int:
+    try:
+        return int(environ.get('GIT_CONFIG_COUNT', '0'))
+    except (TypeError, ValueError) as e:
+        raise ValueError("bogus count in GIT_CONFIG_COUNT") from e
+
+
+def _get_gitconfig_var_from_env(nid: int, kind: str) -> str:
+    envname = f'GIT_CONFIG_{kind.upper()}_{nid}'
+    var = environ.get(envname)
+    if var is None:
+        raise ValueError(f"missing config {kind} {envname}")
+    if kind != 'key':
+        return var
+    if not var:
+        raise ValueError(f"empty config key {envname}")
+    if '.' not in var:
+        raise ValueError(f"key {envname} does not contain a section: {var}")
+    return var
+
+
+def set_gitconfig_items_in_env(items: Mapping[str, str | Tuple[str, ...]]):
+    """Set git-config ENV (``GIT_CONFIG_COUNT|KEY|VALUE``) from a mapping
+
+    Any existing declaration of configuration items in the environment is
+    replaced. Any ENV variable of a *valid* existing declaration is removed,
+    before the set configuration items are posted in the ENV.
+
+    Multi-value configuration keys are supported (values provided as a tuple).
+
+    No verification (e.g., of syntax compliance) is performed.
+    """
+    _clean_env_from_gitconfig_items()
+
+    count = 0
+    for key, value in items.items():
+        # homogeneous processing of multiple value items, and single values
+        values = value if isinstance(value, tuple) else (value,)
+        for v in values:
+            environ[f'GIT_CONFIG_KEY_{count}'] = key
+            environ[f'GIT_CONFIG_VALUE_{count}'] = v
+            count += 1
+    if count:
+        environ['GIT_CONFIG_COUNT'] = str(count)
+
+
+def _clean_env_from_gitconfig_items():
+    # we only care about intact specifications here, if there was cruft
+    # to start with, we have no responsibilities
+    try:
+        count = _get_gitconfig_itemcount()
+    except ValueError:
+        return
+
+    for i in range(count):
+        environ.pop(f'GIT_CONFIG_KEY_{i}', None)
+        environ.pop(f'GIT_CONFIG_VALUE_{i}', None)
+
+    environ.pop('GIT_CONFIG_COUNT', None)
diff --git a/datalad_next/patches/cli_configoverrides.py b/datalad_next/patches/cli_configoverrides.py
new file mode 100644
index 0000000..59276b5
--- /dev/null
+++ b/datalad_next/patches/cli_configoverrides.py
@@ -0,0 +1,50 @@
+from datalad.config import _update_from_env as _update_from_datalad_env
+from datalad.cli.helpers import _parse_overrides_from_cmdline
+
+from datalad_next.config.utils import (
+    get_gitconfig_items_from_env,
+    set_gitconfig_items_in_env,
+)
+
+from . import apply_patch
+
+
+def parse_overrides_from_cmdline(cmdlineargs):
+    # read from cmdlineargs first to error on any syntax issues
+    # before any other processing
+    cli_overrides = _parse_overrides_from_cmdline(cmdlineargs)
+
+    # reuse datalad-core implementation of datalad-specific ENV parsing
+    # for config items
+    overrides = {}
+    _update_from_datalad_env(overrides)
+
+    # let CLI settings override any ENV -- in-line with the behavior of Git
+    overrides.update(cli_overrides)
+
+    # read any existing GIT_CONFIG ENV vars and superimpose our
+    # overrides on them, repost in ENV using git-native approach.
+    # This will apply the overrides to any git(-config) calls
+    # in this process and any subprocess
+    gc_overrides = get_gitconfig_items_from_env()
+    gc_overrides.update(overrides)
+    set_gitconfig_items_in_env(gc_overrides)
+
+    # we do not actually disclose any of these overrides.
+    # the CLI runs a `datalad.cfg.reload(force=True)`
+    # immediately after executing this function and thereby
+    # pulls in the overrides we just posted into the ENV
+    # here. This change reduced the scope of
+    # `datalad.cfg.overrides` to be mere instance overrides
+    # and no longer process overrides. This rectifies the mismatch
+    # between appearance and actual impact of this information
+    # in the ConfigManager
+    return {}
+
+
+apply_patch(
+    'datalad.cli.helpers', None, '_parse_overrides_from_cmdline',
+    parse_overrides_from_cmdline,
+    msg='Enable posting DataLad config overrides CLI/ENV as '
+    'GIT_CONFIG items in process ENV',
+)
diff --git a/datalad_next/patches/enabled.py b/datalad_next/patches/enabled.py
index 5b82ed4..84822c9 100644
--- a/datalad_next/patches/enabled.py
+++ b/datalad_next/patches/enabled.py
@@ -1,4 +1,5 @@
 from . import (
+    cli_configoverrides,
     commanderror,
     common_cfg,
     annexrepo,


Test patch:
diff --git a/datalad_next/config/tests/__init__.py b/datalad_next/config/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/datalad_next/config/tests/test_utils.py b/datalad_next/config/tests/test_utils.py
new file mode 100644
index 0000000..f084f82
--- /dev/null
+++ b/datalad_next/config/tests/test_utils.py
@@ -0,0 +1,146 @@
+
+import pytest
+
+from .. import utils  # for patching environ
+
+from ..utils import (
+    get_gitconfig_items_from_env,
+    set_gitconfig_items_in_env,
+)
+
+
+def test_get_gitconfig_items_from_env(monkeypatch):
+    with monkeypatch.context() as m:
+        # without the COUNT the rest does not matter and we always
+        # get an empty dict
+        m.delenv('GIT_CONFIG_COUNT', raising=False)
+        assert get_gitconfig_items_from_env() == {}
+
+    with monkeypatch.context() as m:
+        # setting zero items, also makes everything else irrelevant
+        m.setenv('GIT_CONFIG_COUNT', '0')
+        assert get_gitconfig_items_from_env() == {}
+
+    with monkeypatch.context() as m:
+        # predictable error for botched count
+        m.setenv('GIT_CONFIG_COUNT', 'rubbish')
+        with pytest.raises(ValueError) as e:
+            get_gitconfig_items_from_env()
+        assert 'bogus count in GIT_CONFIG_COUNT' in str(e)
+
+    # bunch of std error conditions
+    for env, excstr in (
+            ({'GIT_CONFIG_COUNT': 1,
+              'GIT_CONFIG_KEY_0': 'section.name'},
+             'missing config value'),
+            ({'GIT_CONFIG_COUNT': 1,
+              'GIT_CONFIG_VALUE_0': 'value'},
+             'missing config key'),
+            ({'GIT_CONFIG_COUNT': 1,
+              'GIT_CONFIG_KEY_0': '',
+              'GIT_CONFIG_VALUE_0': 'value'},
+             'empty config key'),
+            ({'GIT_CONFIG_COUNT': 1,
+              'GIT_CONFIG_KEY_0': 'nosection',
+              'GIT_CONFIG_VALUE_0': 'value'},
+             'does not contain a section'),
+    ):
+        with monkeypatch.context() as m:
+            m.setattr(utils, 'environ', env)
+            with pytest.raises(ValueError) as e:
+                get_gitconfig_items_from_env()
+            assert excstr in str(e)
+
+    # proper functioning
+    for env, target in (
+            ({'GIT_CONFIG_COUNT': 1,
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'value'},
+             {'section.name': 'value'}),
+            ({'GIT_CONFIG_COUNT': 2,
+              'GIT_CONFIG_KEY_0': 'section.name1',
+              'GIT_CONFIG_VALUE_0': 'value1',
+              'GIT_CONFIG_KEY_1': 'section.name2',
+              'GIT_CONFIG_VALUE_1': 'value2'},
+             {'section.name1': 'value1', 'section.name2': 'value2'}),
+            # double-specification appends
+            # ❯ GIT_CONFIG_COUNT=2 \
+            #   GIT_CONFIG_KEY_0=section.name \
+            #   GIT_CONFIG_VALUE_0=val1 \
+            # GIT_CONFIG_KEY_1=section.name \
+            #   GIT_CONFIG_VALUE_1=val2 \
+            #   git config --list --show-origin | grep 'command line:'
+            # command line:   section.name=val1
+            # command line:   section.name=val2
+            ({'GIT_CONFIG_COUNT': 3,
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'value0',
+              'GIT_CONFIG_KEY_1': 'section.name',
+              'GIT_CONFIG_VALUE_1': 'value1',
+              'GIT_CONFIG_KEY_2': 'section.name',
+              'GIT_CONFIG_VALUE_2': 'value2'},
+             {'section.name': ('value0', 'value1', 'value2')}),
+    ):
+        with monkeypatch.context() as m:
+            m.setattr(utils, 'environ', env)
+            assert get_gitconfig_items_from_env() == target
+
+
+def test_set_gitconfig_items_in_env(monkeypatch):
+    for start, items, target in (
+            # giving nothing preserves statusquo
+            ({}, {}, {}),
+            ({'DUMMY': 'value'}, {}, {'DUMMY': 'value'}),
+            # fixable specification is cleaned up
+            ({'GIT_CONFIG_COUNT': '526'}, {}, {}),
+            # but it has limits
+            ({'GIT_CONFIG_COUNT': 'nochance'}, {},
+             {'GIT_CONFIG_COUNT': 'nochance'}),
+            # and there is no exhaustive search
+            ({'GIT_CONFIG_KEY_3': 'dummy'}, {}, {'GIT_CONFIG_KEY_3': 'dummy'}),
+            # virgin territory
+            ({}, {'section.name': 'value'},
+             {'GIT_CONFIG_COUNT': '1',
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'value'}),
+            # "set" means "replace, not amend
+            ({'GIT_CONFIG_COUNT': '1',
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'value'},
+             {'altsection.name2': 'value2'},
+             {'GIT_CONFIG_COUNT': '1',
+              'GIT_CONFIG_KEY_0': 'altsection.name2',
+              'GIT_CONFIG_VALUE_0': 'value2'}),
+            # full cleanupage
+            ({'GIT_CONFIG_COUNT': '2',
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'value',
+              'GIT_CONFIG_KEY_1': 'altsection.name2',
+              'GIT_CONFIG_VALUE_1': 'value2'},
+             {}, {}),
+            # multi-value support, order preserved
+            ({}, {'section.name': ('c', 'a', 'b')},
+             {'GIT_CONFIG_COUNT': '3',
+              'GIT_CONFIG_KEY_0': 'section.name',
+              'GIT_CONFIG_VALUE_0': 'c',
+              'GIT_CONFIG_KEY_1': 'section.name',
+              'GIT_CONFIG_VALUE_1': 'a',
+              'GIT_CONFIG_KEY_2': 'section.name',
+              'GIT_CONFIG_VALUE_2': 'b'}),
+    ):
+        with monkeypatch.context() as m:
+            env = dict(start)
+            m.setattr(utils, 'environ', env)
+            set_gitconfig_items_in_env(items)
+            assert env == target
+
+
+def test_get_set_gitconfig_env_roundtrip(monkeypatch):
+    items = {'section.name': ('c', 'a', 'b'),
+             'space section.na me.so me': 'v al'}
+    with monkeypatch.context() as m:
+        env = {}
+        m.setattr(utils, 'environ', env)
+        # feed in copy to ensure validity of the test
+        set_gitconfig_items_in_env(dict(items))
+        assert get_gitconfig_items_from_env() == items
diff --git a/datalad_next/patches/tests/test_cli_configoverrides.py b/datalad_next/patches/tests/test_cli_configoverrides.py
new file mode 100644
index 0000000..b1656e3
--- /dev/null
+++ b/datalad_next/patches/tests/test_cli_configoverrides.py
@@ -0,0 +1,18 @@
+from datalad_next.utils import chpwd
+from datalad_next.tests.utils import run_main
+
+
+def test_cli_configoverrides(existing_dataset):
+    # test whether a `datalad -c ...` is effective within the
+    # execution environment of a subprocess (for a non-datalad
+    # configuration item
+    with chpwd(existing_dataset.path):
+        out, err = run_main(
+            [
+                '-c', 'bogusdataladtestsec.subsec=unique',
+                'run',
+                'git config bogusdataladtestsec.subsec',
+            ],
+            # git-config would fail, if the config item is unknown
+            exit_code=0,
+        )