Repository: dynaconf/dynaconf
Pull request #1001 (2023-09-09 19:24:08)

Feature: Dotted Set with Indexing
Related to: 
* https://github.com/dynaconf/dynaconf/issues/299#issuecomment-1677918707
* https://github.com/dynaconf/dynaconf/issues/299#issuecomment-1699855166
* https://github.com/dynaconf/dynaconf/issues/299#issuecomment-1705571274

In one of my use case, I would like a merge strategy via environment variables, like the following

```
DATA:
  a:
    - key:
        subkey: "value"
```
Then in bash, I can set an environment variable to replace `value` with "new value":

```
export DYNACONF_DATA__a__0__key__subkey="new value"
```

which uses the pattern `__(\d+)` to indicate index (presumably user can also set their own desired regex match pattern by overwriting the new variable `NESTED_INDEX_FOR_DYNACONF`.

# Checklist

* [x] Update doc to reflect this feature (added to `docs/configuration.md`)
* [x] Unit test for environment variable overwrite. (Please suggest a more appropriate place / file for this unit test. Currently under `tests/test_base.py: test_environ_dotted_set_with_index`).
* [x] Fixing functional tests.
* ~~Dotted get.~~ This PR has too many changes already. I will make another PR once this is merged.


# Capabilities added in this PR

* Support for `dunder-index setting. set("listy__1__foo", "bar")` (already there)
* Support for stringified-python-index `setting. set("listy[1].foo", "bar")`
* Add list-merge strategy param options to `object_merge` and `handle_metavalues`.
* Add a variable `NESTED_INDEX_FOR_DYNACONF` that allows users to define custom separator for indices when overwriting variables using global variables. Default to `__(\d+)` regex pattern.

netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *dynaconf* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 20b42aa5d207eded24808f9c9e055e799d19ce6c |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/dynaconf/deploys/64fcc65a81419e0008b50f40 |
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1001?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) Report
> Merging [#1001](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1001?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) (38a9236) into [master](https://app.codecov.io/gh/dynaconf/dynaconf/commit/835d5909a98b3097fbf1912205f9f847e887c245?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) (835d590) will **decrease** coverage by `0.25%`.
> Report is 1 commits behind head on master.
> The diff coverage is `95.45%`.

```diff
@@            Coverage Diff             @@
##           master    #1001      +/-   ##
==========================================
- Coverage   98.74%   98.50%   -0.25%     
==========================================
  Files          23       23              
  Lines        2239     2269      +30     
==========================================
+ Hits         2211     2235      +24     
- Misses         28       34       +6     
```


| [Files Changed](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) | Coverage Δ | |
|---|---|---|
| [dynaconf/base.py](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf#diff-ZHluYWNvbmYvYmFzZS5weQ==) | `99.45% <95.45%> (-0.18%)` | :arrow_down: |
| [dynaconf/utils/\_\_init\_\_.py](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf#diff-ZHluYWNvbmYvdXRpbHMvX19pbml0X18ucHk=) | `97.72% <95.45%> (-2.28%)` | :arrow_down: |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf)

EdwardCuiPeacock: @rochacbruno Hi, Bruno. Can you provide feedback on this PR? Thank you.
pedro-psb: As I've commented before, I was worried about performance regressions, so here is a dirty benchmark I ran. The difference seems small enough for me.

```bash
$ git checkout envvar_index2 # this PR
$ hyperfine "make tests_only"
Time (mean ± σ):     11.734 s ±  0.220 s    [User: 11.494 s, System: 0.217 s]
Range (min … max):   11.468 s … 12.162 s    10 runs

$ git checkout $(git merge-base upstream/master envvar_index2) # common ancestor with upstream/master
$ hyperfine "make tests_only"
Time (mean ± σ):     10.807 s ±  0.368 s    [User: 10.555 s, System: 0.230 s]
Range (min … max):   10.601 s … 11.816 s    10 runs
```
rochacbruno: Hi @EdwardCuiPeacock today me and @pedro-psb talked a bit about this.

## Problem

Avoid Ambiguity on object initialization.

To understand the source of the problem, open a terminal on the root of dynaconf and run the following:

```console
$ DYNACONF_LIST__0=Bruno DYNACONF_LIST__1=Rocha dynaconf list -k LIST
# Ignore the warnings
LIST<DynaBox> <Box: {'1': 'Rocha', '0': 'Bruno'}>
```

In the case above, there is no data previously on dynaconf and data is being initialized from the existing envvars and in that case right now dynaconf assumes `__` to be the nested separator for mapping data structure.

There is no way to tell dynaconf if the identifier after the `__` is an index or a key, and we want to avoid the ambiguity, it might be rare but there are configurations where numbers are used as keys.

## Proposal

To resolve the problem we suggest to keep the core of your implementation on `dotted_set` but change the format for exporting the variables, so:

1. add the `INDEX_SEPARATOR_FOR_DYNACONF` variable and its value will default to `___` 3 underscores.
2. on `base.py:Settings.set` detect if there are `___` sequential underlines followed by any number of digits and replace it with the syntax your `dotted_set` can handle.

Example:

`DATA__NESTED___0` becomes `DATA.NESTED[0]`  (the `.` is [already being resolved ](https://github.com/dynaconf/dynaconf/blob/f2bb032ea25a3110f3ba38891fc3307edc203a9a/dynaconf/base.py#L942-L946)in the `.set` method).

The regexp used must take `___0` fragment and replace with `[0]` 


Something like

```
import re

input_string1 = "DATA__NESTED___0"
input_string2 = "DATA__NESTED___0__FOO"

# Replace three underscores followed by digits with '[', the digits, and ']'
output_string1 = re.sub(r'___(\d+)', r'[\1]', input_string1)
output_string2 = re.sub(r'___(\d+)', r'[\1]', input_string2)

print(output_string1)
print(output_string2)
```

"DATA__NESTED___0" becomes "DATA_NESTED[0]".
"DATA__NESTED___0__FOO" becomes "DATA__NESTED[0]__FOO"

Which will lately have the `__` replaced by `.` before calling `dotted_set`







EdwardCuiPeacock: @rochacbruno I made the update based on your description above.
rochacbruno: This is great @EdwardCuiPeacock I tested and works.

As it is a new feature I am thinking about releasing it after the next bug fix release,
WDYT @pedro-psb ? we first release 3.2.4 with the current master state,
then we hold this feature until 3.3.0
pedro-psb: @rochacbruno Sure. I'll migrate the release automation for 3.2.4 this week and we can wait a week or so to ship 3.3.0 and see if we can include anything else.

Patch:
diff --git a/docs/configuration.md b/docs/configuration.md
index 0c45adb..ea008b6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -368,7 +368,7 @@ Otherwise it will be only what is specified in the latest loaded file. read more
 
 ---
 
-### **nested_separator**
+### **nested_separators**
 
 > type=`str`, default=`"__"` (double underescore) </br>
 > env-var=`NESTED_SEPARATOR_FOR_DYNACONF`
@@ -393,6 +393,34 @@ DATABASES = {
 }
 ```
 
+To access list by indexing
+
+> type=`str`, default=`"___"` (triple underescore) </br>
+> env-var=`INDEX_SEPARATOR_FOR_DYNACONF`
+
+ex:
+
+```bash
+export DYNACONF_DATABASES__default__WORKERS___0__Address="1.1.1.1"
+export DYNACONF_DATABASES__default__WORKERS___1__Address="2.2.2.2"
+```
+
+generates:
+
+```python
+DATABASES = {
+    "default": {
+        "ENGINE": {
+            "Address": "0.0.0.0"
+        },
+        "WORKERS": [
+            {"Address": "1.1.1.1"},
+            {"Address": "2.2.2.2"} 
+        ]
+    }
+}
+```
+
 !!! warning
     Choose something that is suitable for env vars, usually you don't need to change this variable.
 
diff --git a/dynaconf/base.py b/dynaconf/base.py
index b516a42..14f079c 100644
--- a/dynaconf/base.py
+++ b/dynaconf/base.py
@@ -4,6 +4,7 @@ import copy
 import importlib
 import inspect
 import os
+import re
 import warnings
 from collections import defaultdict
 from contextlib import contextmanager
@@ -887,15 +888,34 @@ class Settings:
                 "VALIDATE_ON_UPDATE_FOR_DYNACONF"
             )  # pragma: nocover
 
-        split_keys = dotted_key.split(".")
+        # Add a "." before "[" to help splitting
+        split_keys = dotted_key.replace("[", ".[").split(".")
         existing_data = self.get(split_keys[0], {})
         new_data = tree = DynaBox(box_settings=self)
+        value = parse_conf_data(value, tomlfy=tomlfy, box_settings=self)
 
-        for k in split_keys[:-1]:
-            tree = tree.setdefault(k, {})
+        for n, k in enumerate(split_keys):
+            is_not_end = n < (len(split_keys) - 1)
+            if is_not_end:
+                next_default = [] if "[" in split_keys[n + 1] else {}
 
-        value = parse_conf_data(value, tomlfy=tomlfy, box_settings=self)
-        tree[split_keys[-1]] = value
+            if "[" not in k:  # accessing field of a dict
+                if is_not_end:
+                    tree = tree.setdefault(k, next_default)  # get next
+                else:
+                    tree[k] = value  # assign value
+            elif k.startswith("[") and k.endswith(
+                "]"
+            ):  # accessing index of a list
+                index = int(k.replace("[", "").replace("]", ""))
+                # This makes sure we can assign any arbitrary index
+                tree.extend([next_default] * (index + 1))
+                if is_not_end:
+                    tree = tree[index]  # get at index
+                else:
+                    tree[index] = value  # assign value
+            else:  # odd cases like [2]0
+                raise (ValueError("Invalid field:", k))
 
         if existing_data:
             old_data = DynaBox(
@@ -905,6 +925,7 @@ class Settings:
                 old=old_data,
                 new=new_data,
                 full_path=split_keys,
+                list_merge="deep",  # when to use deep / shallow replace?
             )
         self.update(data=new_data, tomlfy=tomlfy, validate=validate, **kwargs)
 
@@ -945,13 +966,22 @@ class Settings:
             validate = self.get("VALIDATE_ON_UPDATE_FOR_DYNACONF")
         if dotted_lookup is empty:
             dotted_lookup = self.get("DOTTED_LOOKUP_FOR_DYNACONF")
+
+        # Do index replacement first
+        nested_ind = self.get("INDEX_SEPARATOR_FOR_DYNACONF")
+        # DYNACONF_DATA__a___0__key___2__subkey ->
+        # DYNACONF_DATA__a[0]__key[2]__subkey
+        if nested_ind and isinstance(key, str):
+            nested_ind = rf"{nested_ind}(\d+)"
+            key = re.sub(nested_ind, r"[\1]", key)
+
         nested_sep = self.get("NESTED_SEPARATOR_FOR_DYNACONF")
 
         if isinstance(key, str):
             if nested_sep and nested_sep in key:
                 key = key.replace(nested_sep, ".")  # FOO__bar -> FOO.bar
 
-            if "." in key and dotted_lookup is True:
+            if ("." in key or "[" in key) and dotted_lookup is True:
                 return self._dotted_set(
                     key,
                     value,
@@ -1003,7 +1033,15 @@ class Settings:
                 # `dynaconf_merge` may be used within the key structure
                 # Or merge_enabled is set to True
                 parsed, source_metadata = self._merge_before_set(
-                    existing, parsed, source_metadata, context_merge=merge
+                    existing,
+                    parsed,
+                    source_metadata,
+                    context_merge=merge,
+                    list_merge="replace"
+                    if source_metadata
+                    and source_metadata.loader == "setdefault"
+                    and source_metadata.env == "development"
+                    else "merge",  # fix 905
                 )
 
         if isinstance(parsed, dict) and not isinstance(parsed, DynaBox):
@@ -1070,6 +1108,7 @@ class Settings:
 
         data = data or {}
         data.update(kwargs)
+
         for key, value in data.items():
             # update() will handle validation later
             with suppress(ValidationError):
@@ -1095,6 +1134,7 @@ class Settings:
         value,
         identifier: SourceMetadata | None = None,
         context_merge=empty,
+        list_merge="merge",
     ):
         """
         Merge the new value being set with the existing value before set
@@ -1116,7 +1156,7 @@ class Settings:
                 identifier = (
                     identifier._replace(merged=True) if identifier else None
                 )
-                value = object_merge(existing, value)
+                value = object_merge(existing, value, list_merge=list_merge)
 
         if isinstance(value, (list, tuple)):
             value = list(value)
diff --git a/dynaconf/default_settings.py b/dynaconf/default_settings.py
index 8b86469..4126660 100644
--- a/dynaconf/default_settings.py
+++ b/dynaconf/default_settings.py
@@ -144,6 +144,17 @@ DOTTED_LOOKUP_FOR_DYNACONF = get("DOTTED_LOOKUP_FOR_DYNACONF", True)
 # To disable it one can set `NESTED_SEPARATOR_FOR_DYNACONF=false`
 NESTED_SEPARATOR_FOR_DYNACONF = get("NESTED_SEPARATOR_FOR_DYNACONF", "__")
 
+# By default `__(\d+)` (regexp) is the separated for nested list env vars
+# export `DYNACONF_DATABASE_servers__0=server.com`
+# export `DYNACONF_DATABASE_servers__1=server.org`
+# export `DYNACONF_DATABASE_servers_2_0=server.io`
+# Should result in settings.DATABASE == {
+#   "servers": ["server.com", "server.org"],
+#   "servers_2_0": "server.io"
+# }
+# To disable it one can set `INDEX_SEPARATOR_FOR_DYNACONF=false`
+INDEX_SEPARATOR_FOR_DYNACONF = get("INDEX_SEPARATOR_FOR_DYNACONF", "___")
+
 # The env var specifying settings module
 ENVVAR_FOR_DYNACONF = get("ENVVAR_FOR_DYNACONF", "SETTINGS_FILE_FOR_DYNACONF")
 
diff --git a/dynaconf/utils/__init__.py b/dynaconf/utils/__init__.py
index 339a4ef..b5d78d0 100644
--- a/dynaconf/utils/__init__.py
+++ b/dynaconf/utils/__init__.py
@@ -6,6 +6,7 @@ from collections import defaultdict
 from json import JSONDecoder
 from typing import Any
 from typing import Iterator
+from typing import Literal
 from typing import TYPE_CHECKING
 from typing import TypeVar
 
@@ -30,7 +31,11 @@ if os.name == "nt":  # pragma: no cover
 
 
 def object_merge(
-    old: Any, new: Any, unique: bool = False, full_path: list[str] = None
+    old: Any,
+    new: Any,
+    unique: bool = False,
+    full_path: list[str] = None,
+    list_merge: Literal["merge", "shallow", "deep"] = "merge",
 ) -> Any:
     """
     Recursively merge two data structures, new is mutated in-place.
@@ -39,6 +44,11 @@ def object_merge(
     :param new: The new data to get old values merged in to.
     :param unique: When set to True existing list items are not set.
     :param full_path: Indicates the elements of a tree.
+    :param list_merge: Methods to use to merge lists
+        - merge: default merge behavior, i.e. (unique) concatenation
+        - shallow: replace the top-most level list of the nested structure
+        - deep: iteratively traverse the nested structure and replace
+            the element in the list at the level specified by the full_path
     """
     if full_path is None:
         full_path = []
@@ -52,10 +62,29 @@ def object_merge(
             new.remove("dynaconf_merge_unique")
             unique = True
 
-        for item in old[::-1]:
-            if unique and item in new:
-                continue
-            new.insert(0, item)
+        if list_merge == "merge" or unique:
+            for item in old[::-1]:
+                if unique and item in new:
+                    continue
+                new.insert(0, item)
+        # replace mode
+        # elif list_merge == "replace": pass
+        elif len(full_path) > 0:  # element-wise merge
+            new.extend([[]] * max(len(old) - len(new), 0))
+            for ii, item in enumerate(old):
+                # replace at corresponding positions
+                if list_merge == "shallow":
+                    new[ii] = new[ii] or item
+                else:  # deep replace
+                    if not new[ii]:  # copy over the older values
+                        new[ii] = item
+                    elif item:  # old[ii] is not None
+                        object_merge(
+                            old[ii],
+                            new[ii],
+                            full_path=full_path[1:],
+                            list_merge="deep",
+                        )
 
     if isinstance(old, dict) and isinstance(new, dict):
         existing_value = recursive_get(old, full_path)  # doesn't handle None
@@ -96,13 +125,13 @@ def object_merge(
                 if old_key not in new:
                     new[old_key] = value
                 else:
-                    object_merge(
+                    new[old_key] = object_merge(
                         value,
                         new[old_key],
                         full_path=full_path[1:] if full_path else None,
+                        list_merge=list_merge,
                     )
-
-        handle_metavalues(old, new)
+        handle_metavalues(old, new, list_merge=list_merge)
 
     return new
 
@@ -111,22 +140,39 @@ def recursive_get(
     obj: DynaBox | dict[str, int] | dict[str, str | int],
     names: list[str] | None,
 ) -> Any:
-    """Given a dot accessible object and a list of names `foo.bar.zaz`
-    gets recursively all names one by one obj.foo.bar.zaz.
+    """Given a dot accessible object and a list of names `foo.bar.[1].zaz`
+    gets recursively all names one by one obj.foo.bar.[1].zaz.
     """
-    if not names:
+    if not names or obj is None:
         return
     head, *tail = names
-    result = getattr(obj, head, None)
+    if "[" not in head:
+        result = getattr(obj, head, None)
+    else:
+        index = int(head.replace("[", "").replace("]", ""))
+        result = obj[index] if index < len(obj) else []
+
     if not tail:
         return result
+
     return recursive_get(result, tail)
 
 
 def handle_metavalues(
-    old: DynaBox | dict[str, int] | dict[str, str | int], new: Any
+    old: DynaBox | dict[str, int] | dict[str, str | int],
+    new: Any,
+    list_merge: Literal["merge", "shallow", "deep"] = "merge",
 ) -> None:
-    """Cleanup of MetaValues on new dict"""
+    """
+    Cleanup of MetaValues on new dict
+    :param old: old values
+    :param new: new values
+    :param list_merge: Methods to use to merge lists
+        - merge: default merge behavior, i.e. (unique) concatenation
+        - shallow: replace the top-most level list of the nested structure
+        - deep: iteratively traverse the nested structure and replace
+            the element in the list at the level specified by the full_path
+    """
 
     for key in list(new.keys()):
         # MetaValue instances
@@ -140,7 +186,9 @@ def handle_metavalues(
         elif getattr(new[key], "_dynaconf_merge", False):
             # a Merge on `new` triggers merge with existing data
             new[key] = object_merge(
-                old.get(key), new[key].unwrap(), unique=new[key].unique
+                old.get(key),
+                new[key].unwrap(),
+                unique=new[key].unique,
             )
 
         # Data structures containing merge tokens
@@ -173,7 +221,9 @@ def handle_metavalues(
                 new[key] = local_merge
 
             if local_merge:
-                new[key] = object_merge(old.get(key), new[key])
+                new[key] = object_merge(
+                    old.get(key), new[key], list_merge=list_merge
+                )
 
 
 class DynaconfDict(dict):


Test patch:
diff --git a/tests/test_base.py b/tests/test_base.py
index 16419ea..d03d12e 100644
--- a/tests/test_base.py
+++ b/tests/test_base.py
@@ -700,6 +700,40 @@ def test_dotted_set(settings):
     }
 
 
+def test_dotted_set_with_indexing(settings):
+    settings.set("MERGE_ENABLED_FOR_DYNACONF", False)
+
+    # Dotted set with index
+    settings.set("nested_a.nested_b[2][1].nested_c.nested_d[3]", "old_conf")
+    settings.set(
+        "nested_a.nested_b[2][1].nested_c.nested_d[3]", "new_conf1"
+    )  # overwrite
+    settings.set(
+        "nested_a.nested_b[2][1].nested_c.nested_d[2]", "new_conf2"
+    )  # insert
+    assert settings.NESTED_A.NESTED_B[2][1].NESTED_C.NESTED_D[3] == "new_conf1"
+    assert settings.NESTED_A.NESTED_B[2][1].NESTED_C.NESTED_D[2] == "new_conf2"
+    assert len(settings.NESTED_A.NESTED_B[0]) < 1
+    settings.set(
+        "nested_a.nested_b[0][2].nested_c.nested_d[0]", "extra_conf"
+    )  # add more
+    assert len(settings.NESTED_A.NESTED_B[0]) > 0
+    settings.set(
+        "nested_a.nested_b[2][1].nested_c.nested_d",
+        ["conf1", "conf2", "conf3"],
+    )  # overwrite list
+    assert settings.NESTED_A.NESTED_B[2][1].NESTED_C.NESTED_D == [
+        "conf1",
+        "conf2",
+        "conf3",
+    ]
+
+    # This test case is the reason why choosing
+    # __(\d+) pattern instead of _(\d+)_
+    settings.set("nested_5.nested_6_0", "World")
+    assert settings.NESTED_5.NESTED_6_0 == "World"
+
+
 def test_dotted_set_with_merge(settings):
     settings.set("MERGE_ENABLED_FOR_DYNACONF", False)
 
@@ -1508,3 +1542,21 @@ def test_no_extra_values_in_nested_structure():
     settings = Dynaconf()
     settings.set("key", [{"d": "v"}])
     assert settings.key == [{"d": "v"}]
+
+
+def test_environ_dotted_set_with_index():
+    os.environ["DYNACONF_NESTED_A__nested_1__nested_2"] = "new_conf"
+    os.environ[
+        "DYNACONF_NESTED_A__nested_b___2___1__nested_c__nested_d___3"
+    ] = "old_conf"
+    settings = Dynaconf(envvar_prefix="DYANCONF")
+    assert isinstance(settings.NESTED_A.NESTED_B, list)
+    assert isinstance(settings.NESTED_A.NESTED_B[2], list)
+    assert isinstance(settings.NESTED_A.NESTED_B[2][1], dict)
+    assert settings.NESTED_A.NESTED_B[2][1].NESTED_C.NESTED_D[3] == "old_conf"
+    assert settings.NESTED_A.NESTED_1.NESTED_2 == "new_conf"
+    # remove environment variables after testing
+    del os.environ["DYNACONF_NESTED_A__nested_1__nested_2"]
+    del os.environ[
+        "DYNACONF_NESTED_A__nested_b___2___1__nested_c__nested_d___3"
+    ]