Repository: conan-io/conan
Pull request: conf dict updates
Changelog: Feature: Introduce the ``conf`` dictionary update semantics with ``*=`` operator.
Docs: https://github.com/conan-io/docs/pull/3141

Close https://github.com/conan-io/conan/issues/13535

Problem statement:
[feature] Add support for updating values of a dictionary type configuration in profile (e.g. tools.build:compiler_executables)
### What is your suggestion?

Hi there!

I am proposing a new feature for Conan package manager that would allow updating values of a dictionary configuration like `tools.build:compiler_executables` in the same way we can already append or prepend values of a list configuration. This would make it easier to compose profiles and install compilers via `tool_requires`.

Currently, we can append or prepend values to an existing list configuration in Conan like so:

```less
# Value defined somewhere in global.conf or a main profile
tools.build:cxxflags=["--flag value"]
...
# Appending values into the existing list in some profile
tools.build:cxxflags+=["--suffix-flag suffix-value"] # -> tools.build:cxxflags=["--flag value", "--suffix-flag suffix-value"]
...
# Prepending values into the existing list in some other profile
tools.build:cxxflags=+["--prefix-flag prefix-value"] # -> tools.build:cxxflags=["--prefix-flag prefix-value", "--flag value", "--suffix-flag suffix-value"]
```

However, there is currently no way to update values of a dictionary configuration. With the proposed feature, we could update the `tools.build:compiler_executables` configuration in the same way:

```less
# Value defined somewhere in global.conf or a main profile
tools.build:compiler_executables={"c": "cl", "cpp": "cl"}
...
# Updating values by overriding into the existing dictionary in some profile
tools.build:compiler_executables**={"fortran": "ifort"} # -> tools.build:compiler_executables={"c": "cl", "cpp": "cl", "fortran": "ifort"}
...
# Updating values without overriding into the existing dictionary in some other profile (fortran is not updated because it already exists)
tools.build:compiler_executables=**{"fortran": "gfortran"} # -> tools.build:compiler_executables={"c'" "cl", "cpp": "cl", "fortran": "ifort"}
...
# Updating values without overriding into the existing dictionary in some other profile (cuda is added because it does not already exists)
tools.build:compiler_executables=**{"cuda": "nvcc"} # -> tools.build:compiler_executables={"c'" "cl", "cpp": "cl", "fortran": "ifort", "cuda": "nvcc"}
```

This feature would make it easier to compose a main profile with secondary profiles that install compilers via `tool_requires`. For example:

```less
# Main msvc profile
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.runtime=dynamic
compiler.version=192
compiler.cppstd=17
os=Windows
[conf]
tools.build:compiler_executables={'c': 'cl', 'cpp': 'cl'},
...
# Secondary ifort profile
[tool_requires]
ifort/version@company/channel
[conf]
tools.build:compiler_executables**={'fortran': 'ifort'}
...
# Secondary nvcc profile
[tool_requires]
nvcc/version@company/channel
[conf]
tools.build:compiler_executables**={'cuda': 'nvcc'}
```

Then composing the profiles would give use the desired output:

```shell
$ conan profile show -pr msvc -pr ifort -pr nvcc
Host profile:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=17
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=192
os=Windows
[tool_requires]
*: ifort/version@company/channel, nvcc/version@company/channel
[conf]
tools.build:compiler_executables={'c': 'cl', 'cpp': 'cl', 'fortran': 'ifort', 'cuda': 'nvcc'}
```

It would make my life much easier for creating usable and composable secondary compiler profiles. Whatever syntax is used for updating the dictionary values.

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide

Hints from issue discussion:
HI @todorico 

I think this might be possible, lets give it a try.

Commit message: conf dict updates

Patch to review:
diff --git a/conans/model/conf.py b/conans/model/conf.py
index d7f56d62f47..6773bf06218 100644
--- a/conans/model/conf.py
+++ b/conans/model/conf.py
@@ -122,13 +122,14 @@ class _ConfVarPlaceHolder:
 
 class _ConfValue(object):
 
-    def __init__(self, name, value, path=False):
+    def __init__(self, name, value, path=False, update=None):
         if name != name.lower():
             raise ConanException("Conf '{}' must be lowercase".format(name))
         self._name = name
         self._value = value
         self._value_type = type(value)
         self._path = path
+        self._update = update
 
     def __repr__(self):
         return repr(self._value)
@@ -166,8 +167,9 @@ def serialize(self):
         return {self._name: _value}
 
     def update(self, value):
-        if self._value_type is dict:
-            self._value.update(value)
+        assert self._value_type is dict, "Only dicts can be updated"
+        assert isinstance(value, dict), "Only dicts can update"
+        self._value.update(value)
 
     def remove(self, value):
         if self._value_type is list:
@@ -210,6 +212,13 @@ def compose_conf_value(self, other):
                 new_value = self._value[:]  # do a copy
                 new_value[index:index + 1] = other._value  # replace the placeholder
                 self._value = new_value
+        elif v_type is dict and o_type is dict:
+            if self._update:
+                # only if the current one is marked as "*=" update, otherwise it remains
+                # as this is a "compose" operation, self has priority, it is the one updating
+                new_value = other._value.copy()
+                new_value.update(self._value)
+                self._value = new_value
         elif self._value is None or other._value is None:
             # It means any of those values were an "unset" so doing nothing because we don't
             # really know the original value type
@@ -361,11 +370,12 @@ def update(self, name, value):
         :param name: Name of the configuration.
         :param value: Value of the configuration.
         """
-        conf_value = _ConfValue(name, {})
+        # Placeholder trick is not good for dict update, so we need to explicitly update=True
+        conf_value = _ConfValue(name, {}, update=True)
         self._values.setdefault(name, conf_value).update(value)
 
     def update_path(self, name, value):
-        conf_value = _ConfValue(name, {}, path=True)
+        conf_value = _ConfValue(name, {}, path=True, update=True)
         self._values.setdefault(name, conf_value).update(value)
 
     def append(self, name, value):
@@ -469,8 +479,9 @@ def set_relative_base_folder(self, folder):
 
 class ConfDefinition:
 
+    # Order is important, "define" must be latest
     actions = (("+=", "append"), ("=+", "prepend"),
-               ("=!", "unset"), ("=", "define"))
+               ("=!", "unset"), ("*=", "update"), ("=", "define"))
 
     def __init__(self):
         self._pattern_confs = OrderedDict()
diff --git a/conans/test/integration/configuration/test_profile_priority.py b/conans/test/integration/configuration/test_profile_priority.py
index b34881d0df6..0fac53b1792 100644
--- a/conans/test/integration/configuration/test_profile_priority.py
+++ b/conans/test/integration/configuration/test_profile_priority.py
@@ -1,4 +1,5 @@
 import os
+import textwrap
 
 from conans.test.utils.tools import TestClient
 from conans.util.files import save
@@ -43,3 +44,30 @@ def test_profile_cache_folder_priority():
 
     c.run("install . -pr=default")
     assert "os=AIX" in c.out
+
+
+def test_profile_cli_priority():
+    c = TestClient()
+    profile1 = textwrap.dedent("""\
+        [settings]
+        os=AIX
+        [conf]
+        user.myconf:myvalue1=1
+        user.myconf:myvalue2=[2]
+        user.myconf:myvalue3={"3": "4"}
+        """)
+    profile2 = textwrap.dedent("""\
+        [settings]
+        os=FreeBSD
+        [conf]
+        user.myconf:myvalue1=2
+        user.myconf:myvalue2+=4
+        user.myconf:myvalue3*={"3": "5"}
+        """)
+    c.save({"profile1": profile1,
+            "profile2": profile2})
+    c.run("profile show -pr=./profile1 -pr=./profile2")
+    assert "os=FreeBSD" in c.out
+    assert "user.myconf:myvalue1=2" in c.out
+    assert "user.myconf:myvalue2=[2, 4]" in c.out
+    assert "user.myconf:myvalue3={'3': '5'}" in c.out
diff --git a/conans/test/unittests/model/test_conf.py b/conans/test/unittests/model/test_conf.py
index 602d4fad164..58bd9989434 100644
--- a/conans/test/unittests/model/test_conf.py
+++ b/conans/test/unittests/model/test_conf.py
@@ -1,4 +1,3 @@
-import sys
 import textwrap
 
 import pytest
@@ -196,11 +195,19 @@ def test_compose_conf_complex():
         zlib:user.company.check:shared_str="False"
     """)
 
-    if sys.version_info.major == 2:  # problems with the order in Python 2.x
-        text = c.dumps()
-        assert all([line in text for line in expected_text.splitlines()])
-    else:
-        assert c.dumps() == expected_text
+    assert c.dumps() == expected_text
+
+
+def test_compose_conf_dict_updates():
+    c = ConfDefinition()
+    c.loads("user.company:mydict={'1': 'a'}\n"
+            "user.company:mydict2={'1': 'a'}")
+    c2 = ConfDefinition()
+    c2.loads("user.company:mydict={'2': 'b'}\n"
+             "user.company:mydict2*={'2': 'b'}")
+    c.update_conf_definition(c2)
+    assert c.dumps() == ("user.company:mydict={'2': 'b'}\n"
+                         "user.company:mydict2={'1': 'a', '2': 'b'}\n")
 
 
 def test_conf_get_check_type_and_default():


Code review comments:
diff_hunk: @@ -43,3 +44,30 @@ def test_profile_cache_folder_priority():
 
     c.run("install . -pr=default")
     assert "os=AIX" in c.out
+
+
+def test_profile_cli_priority():
+    c = TestClient()
+    profile1 = textwrap.dedent("""\
+        [settings]
+        os=AIX
+        [conf]
+        user.myconf:myvalue1=1
+        user.myconf:myvalue2=[2]
+        user.myconf:myvalue3={"3": "4"}
path: conans/test/integration/configuration/test_profile_priority.py
text: @user1:
Can you add another key to this dict? Just to ensure the update won't mess with that value

@author:
Added, fixed a bug with it! Good catch!