Repository: conan-io/conan

Issue:
[feature] Recipe definition of options with medium precedence over profiles
### What is your suggestion?

So some recipes can define ``hwloc*:shared=True``, that has higher precedence than profile ``*:shared=False``, but lower than profile ``hwloc*:shared=False``.

cc @uilianries 

### Have you read the CONTRIBUTING guide?

- [ ] I've read the CONTRIBUTING guide
[question] Force option values on requirements
### What is your question?

The Documentation of the `configure()` method (https://docs.conan.io/2/reference/conanfile/methods/configure.html) states the following: 

> The definition of options values is only a “suggestion" [...]

Is there any chance to force an option onto a dependency. Or at least automatically check if the set option was actually used?

In our case we have several libraries, all depending on Qt. Some of them need specific modules of Qt, so they configure those in their recipe. But this only seems to have an effect in some cases, totally not obvious when.

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
We agreed to give this a try for a POC, and see how it looks

Hi @frogarian 

Thanks for  your question.

> Is there any chance to force an option onto a dependency. 
To clarify, this is forcing the dependency option, except in some cases:

- If some other more downstream consumer decides to pass a different option value, the more downstream always have precedence (imaging your consumer ``conanfile.py`` setting up dependencies options to some values and just being ignored because some intermediate dependency is deciding otherwise)
- If the user profile or command line options define otherwise (same principle, the most downstream always have precedence)


> Or at least automatically check if the set option was actually used?

Yes, packages can validate that the options of their dependencies are the correct ones in their ``validate()`` methods, and raise if the values are not valid

In general the recommendation are:
- Define that ``default_options`` in the source package (in this case qt), to be what you need
- Define the options that you need in your ``profile`` files. This is the recommendation in most cases.
- Define the options that you need in conanfile files. This can be messy, because there can be multiple conanfiles setting different values, conflicting, etc, so not recommended in most cases
> If some other more downstream consumer decides to pass a different option value, the more downstream always have precedence (imaging your consumer conanfile.py setting up dependencies options to some values and just being ignored because some intermediate dependency is deciding otherwise)

I understand this point when the more downstream consumer actually sets a different value for the option.
But in our case it seems like more downstream consumer also override the value with the default value when they do NOT explicitly set it.
This behavior seems a little strange to me. Wouldn't it be nice to at least have the chance to force some option into the graph?
> But in our case it seems like more downstream consumer also override the value with the default value when they do NOT explicitly set it.

Yes, it is true, the first one requiring the package will define its set of options.
We kind of had that in Conan 1.X and it was a neverending stream of issues related to settings options, bugs, confusions... really complicated. So we decided in 2.0 to simplify, and recommend to avoid defining options in recipes as much as possible, and approach it by defining the right default in the dependency or in profiles.

> This behavior seems a little strange to me. Wouldn't it be nice to at least have the chance to force some option into the graph?

We have this ticket to investigate this: https://github.com/conan-io/conan/issues/14764. I think it would be mostly what you are asking for here, so maybe we can close this ticket and follow up there.

The main issue is that it is not evident. While it seems that it is desired to be able that some recipe "forces" some option of the dependency, this also brings extreme confusion and frustration when users try to define the value of that option in their command line or in their ``conanfile`` and it seems that Conan is failing at that, and they will report a bug against Conan.


I get your point but to me the way it is now in Conan 2 might also lead to confusion. To the user it is not evident which package is the first to require some other package and thereby define its options. And since defining them is just a suggestion, it seems more or less useless in those cases where a dependency is injected from multiple packages.

If on the other hand a user is able to force the option into the graph you might be able to output this to the user and it is at the very least visible in the recipe.

Regarding your feature request I'm not sure if this actually covers my problem.

In our case for example we have, amongst others, two packages `A` and `B`, which both have a dependency on `Qt`. Where package `B` has a strict requirement for the qt websockets submodule, which in conan is modeled as option. But if now package `A` for some reason is the 'first' to require Qt in the graph, package `B` will be left with a Qt package it cannot use due to the missing qt websockets module. And since we provide only those libraries `A` and `B` as part of a bigger bundle of packages we don't have a toplevel recipe to pin down some options and the users of our stack must define lot's of options we could have provided them.

Hope this example demonstrates our dilemma a little better. I'm not sure if this is unique for our use case or why this didn't come up already, but sadly for our case this is really a step back compared to Conan v1.
> Hope this example demonstrates our dilemma a little better. I'm not sure if this is unique for our use case or why this didn't come up already, but sadly for our case this is really a step back compared to Conan v1.

The problem is that Conan 1.X had some important bugs in this regard with the graph expansion. Maybe they were not hitting you, as it worked well for some cases, but it was a nightmare in many others, it was mostly flawed. Any package that had conditional requirements based on options and these options would be defined from different consumer packages would easily lead to broken dependency graphs, but in so subtle ways that it could takes tons of time to debug and understand what was happening.

Let me put a real example in form of a **failing** test in Conan 1.X to demonstrate one of the issues:
```python
def test_qt():
    c = TestClient()
    qt = textwrap.dedent("""
        from conan import ConanFile
        class Qt(ConanFile):
            name = "qt"
            version = "0.1"
            options = {"ssl": [False, True]}
            default_options = {"ssl": False}
            def requirements(self):
                if self.options.ssl:
                    self.requires("openssl/0.1")
            """)
    c.save({"openssl/conanfile.py": GenConanfile("openssl", "0.1"),
            "qt/conanfile.py": qt,
            "sub1/conanfile.py": GenConanfile("sub1", "0.1").with_requires("qt/0.1"),
            "sub2/conanfile.py": GenConanfile("sub2", "0.1").with_requires("qt/0.1")
                                                            .with_default_option("qt*:ssl", True),
            "app/conanfile.py": GenConanfile("app", "0.1").with_requires("sub1/0.1", "sub2/0.1")
            })
    c.run("create openssl")
    c.run("export qt")
    c.run("export sub1")
    c.run("export sub2")
    c.run("install app --build=missing")
    print(c.out)
    assert "openssl" in c.out  # This fails, openssl not there!
```

Even if ``sub2`` package is explicitly defining ``qt:ssl=True`` option, the graph is being resolved without ``openssl``. Conan 2 is not really a step back, it was not working in Conan 1.X either. 

The only approach that we found that can be managed easily and consistently is defining the optionality in one place. Either in the upstream recipe, in this case ``Qt`` or in the downstream common ``profile`` file. The correctness of optionality for specific packages can be easily enforced checking in ``validate()``.
But isn't this test case failing in Conan v2 as well?
Sure. The recommendation for defining options is to centralize it in the dependency recipe or in profiles, both in 1.X and 2.0. Trying to distribute options definitions among multiple recipes in the graph is problematic both in 1.X and 2.0, it is an intrinsic problem of decentralization of information in something that doesn't evaluate linearly, like a graph. 
Ok, so it seems that until now with Conan v1 we just got lucky that it somehow worked out for us and with Conan v2 we finally have to stick to those rules.

Thanks for clarifying this.



Patch:
diff --git a/conans/model/options.py b/conans/model/options.py
index 9384fb08e0e..62aedac2132 100644
--- a/conans/model/options.py
+++ b/conans/model/options.py
@@ -17,6 +17,7 @@ class _PackageOption:
     def __init__(self, name, value, possible_values=None):
         self._name = name
         self._value = value  # Value None = not defined
+        self.important = False
         # possible_values only possible origin is recipes
         if possible_values is None:
             self._possible_values = None
@@ -27,10 +28,11 @@ def __init__(self, name, value, possible_values=None):
     def dumps(self, scope=None):
         if self._value is None:
             return None
+        important = "!" if self.important else ""
         if scope:
-            return "%s:%s=%s" % (scope, self._name, self._value)
+            return "%s:%s%s=%s" % (scope, self._name, important, self._value)
         else:
-            return "%s=%s" % (self._name, self._value)
+            return "%s%s=%s" % (self._name, important, self._value)
 
     def copy_conaninfo_option(self):
         # To generate a copy without validation, for package_id info.options value
@@ -182,12 +184,19 @@ def __setitem__(self, item, value):
 
     def _set(self, item, value):
         # programmatic way to define values, for Conan codebase
+        important = item[-1] == "!"
+        item = item[:-1] if important else item
+
         current_value = self._data.get(item)
         if self._freeze and current_value.value is not None and current_value != value:
             raise ConanException(f"Incorrect attempt to modify option '{item}' "
                                  f"from '{current_value}' to '{value}'")
         self._ensure_exists(item)
-        self._data.setdefault(item, _PackageOption(item, None)).value = value
+        v = self._data.setdefault(item, _PackageOption(item, None))
+        new_value_important = important or (isinstance(value, _PackageOption) and value.important)
+        if new_value_important or not v.important:
+            v.value = value
+            v.important = new_value_important
 
     def items(self):
         result = []


Test patch:
diff --git a/conans/test/integration/options/options_test.py b/conans/test/integration/options/options_test.py
index 520131896d4..94bca69346f 100644
--- a/conans/test/integration/options/options_test.py
+++ b/conans/test/integration/options/options_test.py
@@ -666,3 +666,69 @@ def test_transitive_options_conanfile_py_create(self, client):
         client.save({"app/conanfile.py": conanfile})
         client.run("create app --build=missing")
         self.check(client, False)
+
+
+class TestImportantOptions:
+    @pytest.mark.parametrize("pkg", ["liba", "libb", "app"])
+    def test_important_options(self, pkg):
+        c = TestClient()
+
+        liba = GenConanfile("liba", "0.1").with_option("myoption", [1, 2, 3])
+        libb = GenConanfile("libb", "0.1").with_requires("liba/0.1")
+        app = GenConanfile().with_requires("libb/0.1")
+        if pkg == "liba":
+            liba.with_default_option("myoption!", 2)
+        elif pkg == "libb":
+            libb.with_default_option("*:myoption!", 2)
+        elif pkg == "app":
+            app.with_default_option("*:myoption!", 2)
+        package_id = textwrap.dedent("""
+            def package_id(self):
+                self.output.info(f"MYOPTION: {self.info.options.myoption}")
+            """)
+        liba = str(liba) + textwrap.indent(package_id, "    ")
+
+        c.save({"liba/conanfile.py": liba,
+                "libb/conanfile.py": libb,
+                "app/conanfile.py": app})
+        c.run("export liba")
+        c.run("export libb")
+
+        c.run("graph info app -o *:myoption=3")
+        assert "liba/0.1: MYOPTION: 2" in c.out
+
+        # And the profile can always important-override the option
+        c.run("graph info app -o *:myoption!=3")
+        assert "liba/0.1: MYOPTION: 3" in c.out
+
+    def test_profile_shows_important(self):
+        c = TestClient()
+        c.run("profile show  -o *:myoption!=3")
+        assert "*:myoption!=3" in c.out
+
+    def test_important_options_recipe_priority(self):
+        c = TestClient()
+
+        liba = GenConanfile("liba", "0.1").with_option("myoption", [1, 2, 3, 4])\
+                                          .with_default_option("myoption!", 1)
+        libb = GenConanfile("libb", "0.1").with_requires("liba/0.1")\
+                                          .with_default_option("*:myoption!", 2)
+        app = GenConanfile().with_requires("libb/0.1").with_default_option("*:myoption!", 3)
+
+        package_id = textwrap.dedent("""
+            def package_id(self):
+                self.output.info(f"MYOPTION: {self.info.options.myoption}")
+            """)
+        liba = str(liba) + textwrap.indent(package_id, "    ")
+
+        c.save({"liba/conanfile.py": liba,
+                "libb/conanfile.py": libb,
+                "app/conanfile.py": app})
+        c.run("export liba")
+        c.run("export libb")
+
+        c.run("graph info app")
+        assert "liba/0.1: MYOPTION: 3" in c.out
+
+        c.run("graph info app -o *:myoption!=4")
+        assert "liba/0.1: MYOPTION: 4" in c.out