Repository: meltano/meltano

Issue:
bug: nested configs are cleared when using space as delimiter and when using `--interactive`
### Update (@aaronsteers)

Per the issue thread below, we've identified that the issue is present in the behavior of the CLI syntax with space as delimiter but not when dot is used. The proposal is to make these two function identically:

```bash
meltano config testy-mctest set file_configs.format json       # the expected behavior (does not reset sibling config)
meltano config testy-mctest set file_configs format json       # bug: clears sibling nested config
```



---------

### Original Issue description (@pnadolny13)

When using `--interactive` to update nested settings it overwrites the entry config set. For example:

```yaml
  - name: tap-file
    config:
      file_configs:
        dataset_name: test_file
        format: csv
        path: /my/path/
```

Then I run `meltano config tap-file set --interactive`, navigating to the setting `file_configs.format` (in dot notation) as defined in the plugin definition lock file, updating to `json`, meltano overwrites the whole nested config with only the new value:

```yaml
  - name: tap-file
    config:
      file_configs:
        format: json
```

Slack thread with video showing bug https://meltano.slack.com/archives/C03GKHWS0HM/p1671746236351709

meltano, version 2.12.0


Discussion:
<details>
<summary>meltano.yml</summary>

```yaml
version: 1
default_environment: dev
project_id: c80de8e4-2d61-4115-a24b-b44460a8d35b
environments:
- name: dev
- name: staging
- name: prod
plugins:
  utilities:
  - name: testy-mctest
    namespace: test
    settings:
    - name: file_configs.dataset_name
    - name: file_configs.format
    - name: file_configs.path
    config:
      file_configs:
        format: json
```

</details>

This correctly changes only the wanted setting:

```console
$ meltano config testy-mctest set file_configs.format json
```

though it sets it as `file_configs.format: json` (which is correct, though perhaps unexpected)

While this results in removing the other settings, just like interactive:

```console
$ meltano config testy-mctest set file_configs format json
```
@edgarrmondragon - Thanks for this additional info - it is very helpful and important context, I think.

Based on your additional info, one could presume then that the `--interactive` mode is likely using the code path from:

```
meltano config testy-mctest set file_configs format json
```

and not:

```
meltano config testy-mctest set file_configs.format json
```

@tayloramurphy and @pnadolny13 - Is there any reason from a UX perspective why these should behave differently?

```
meltano config testy-mctest set file_configs format json
meltano config testy-mctest set file_configs.format json
````

If not, I'd propose we try to update the above 2 invocations to be synonyms for each other. If this fixes the issue, awesome, and if not, we'd continue iterating in the PR to ensure all three are equivalent invocations.

@pnadolny13 - Can you also confirm for your case that using the undotted notation above also reproduces the `--interactive` behavior?

cc @kgpayne 
> Is there any reason from a UX perspective why these should behave differently?

@aaronsteers I don't think there should be a difference. As I understand it currently the structure with the `.` will literally put `file_configs.format` in your meltano.yml, where as the other will to the nesting. My preference would be for consistency, so having everything nest appropriately would be great.
> > Is there any reason from a UX perspective why these should behave differently?
>
> @aaronsteers I don't think there should be a difference.

Agreed. 👍 

In that case, I have updated the issue description to focus on aligning these behaviors across the three different invocation patterns: dotted delimiter, space delimiter, and --interactive.
@aaronsteers and @edgarrmondragon I have repro'd both the 'flattening' behaviour (where `set file_configs.format json` creates a flattened key called `file_configs.format`) and the unexpected unset of sibling keys and will work fixing both in the morning 👍 I have already resolved the flattening issue, so I expect to be done by the time you are online 🤞

Resolving patch:
diff --git a/src/meltano/cli/config.py b/src/meltano/cli/config.py
--- a/src/meltano/cli/config.py
+++ b/src/meltano/cli/config.py
@@ -348,8 +348,17 @@ def reset(ctx, store):
 )
 @click.pass_context
 @_use_meltano_env
-def set_(ctx, setting_name, value, store, interactive):
+def set_(
+    ctx: click.core.Context,
+    setting_name: tuple[str, ...],
+    value: Any,
+    store: str,
+    interactive: bool,
+):
     """Set the configurations' setting `<name>` to `<value>`."""
+    if len(setting_name) == 1:
+        setting_name = tuple(setting_name[0].split("."))
+
     interaction = InteractiveConfig(ctx=ctx, store=store, extras=False)
     if interactive:
         interaction.configure_all()
diff --git a/src/meltano/core/utils/__init__.py b/src/meltano/core/utils/__init__.py
--- a/src/meltano/core/utils/__init__.py
+++ b/src/meltano/core/utils/__init__.py
@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import asyncio
+import collections
 import functools
 import hashlib
 import logging
@@ -11,7 +12,6 @@
 import re
 import sys
 import traceback
-from collections import OrderedDict
 from copy import deepcopy
 from datetime import date, datetime, time
 from enum import IntEnum
@@ -170,6 +170,10 @@ def merge(src, dest):
     return dest
 
 
+def are_similar_types(left, right):
+    return isinstance(left, type(right)) or isinstance(right, type(left))
+
+
 def nest(d: dict, path: str, value=None, maxsplit=-1, force=False):  # noqa: WPS210
     """Create a hierarchical dictionary path and return the leaf dict.
 
@@ -215,7 +219,7 @@ def nest(d: dict, path: str, value=None, maxsplit=-1, force=False):  # noqa: WPS
         cursor = cursor[key]
 
     if tail not in cursor or (
-        type(cursor[tail]) is not type(value) and force  # noqa: WPS516
+        (not are_similar_types(cursor[tail], value)) and force  # noqa: WPS516
     ):
         # We need to copy the value to make sure
         # the `value` parameter is not mutated.
@@ -580,7 +584,7 @@ def uniques_in(original: Sequence[T]) -> list[T]:
     Returns:
         A list of unique values from the provided sequence in the order they appeared.
     """
-    return list(OrderedDict.fromkeys(original))
+    return list(collections.OrderedDict.fromkeys(original))
 
 
 # https://gist.github.com/cbwar/d2dfbc19b140bd599daccbe0fe925597#gistcomment-2845059


Test patch:
diff --git a/tests/meltano/core/test_utils.py b/tests/meltano/core/test_utils.py
--- a/tests/meltano/core/test_utils.py
+++ b/tests/meltano/core/test_utils.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from collections import OrderedDict
+
 import pytest  # noqa: F401
 
 from meltano.core.utils import (
@@ -34,6 +36,21 @@ def test_nest():
     # make sure it is a copy
     assert val == start_value and val is not start_value
 
+    new_b = nest(subject, "a.b", "not_a_dict", force=True)
+    assert new_b == "not_a_dict"
+    assert subject == {"a": {"b": "not_a_dict", "list": [], "value": {"value": 1}}}
+
+    # make sure existing values aren't cleared when `value=None` and `force=True`
+    _ = nest(subject, "a.b", OrderedDict({"d": "d_value"}), force=True)  # noqa: WPS122
+    assert subject == {
+        "a": {"b": OrderedDict({"d": "d_value"}), "list": [], "value": {"value": 1}}
+    }
+    similar_b = nest(subject, "a.b", force=True)
+    assert similar_b == OrderedDict({"d": "d_value"})
+    assert subject == {
+        "a": {"b": OrderedDict({"d": "d_value"}), "list": [], "value": {"value": 1}}
+    }
+
 
 def test_pop_at_path():
     subject = {}