Repository: conan-io/conan

Issue:
[bug] Workspaces: No propagation of options from profile
### Describe the bug

When trying the new workspace feature I was comparing how well this would fit to streamline our current workflow.

The majority of our package's conanfiles is using custom `generate()` methods assigning cache variables based on the selected options, for example:
```python
def generate(self):
  ...
  tc = CMakeToolchain(self)
  tc.cache_variables["BUILD_CLIENT_TOOLS"] = self.options.build_client_tools
  tc.cache_variables["BUILD_DOCS"] = self.options.get_safe('build_docs', False)
```

For daily development we maintain a collection of profiles defining toolchains and options for the packages as required for the respective platforms.

```plain
[settings]
os=Linux
arch=x86_64
build_type=Release

[options]
pkg_a/*:build_client_tools=True
..
```

I tried to create a [monolithic build](https://docs.conan.io/2.18/tutorial/developing_packages/workspaces.html#monolithic-build) style workspace using several of our packages and a `conanws.py` as suggested in the docs:

```python

from conan import Workspace
from conan import ConanFile
from conan.tools.cmake import CMakeDeps, CMakeToolchain, cmake_layout
import os

class MyWs(ConanFile):
   """ This is a special conanfile, used only for workspace definition of layout
   and generators. It shouldn't have requirements, tool_requirements. It shouldn't have
   build() or package() methods
   """
   settings = "os", "compiler", "build_type", "arch"

   def generate(self):
      deps = CMakeDeps(self)
      deps.generate()
      tc = CMakeToolchain(self)
      tc.generate()

   def layout(self):
      cmake_layout(self)

class Ws(Workspace):
   def root_conanfile(self):
      return MyWs  # Note this is the class name

   def packages(self):
      result = []
      for f in ['pkg_a', 'pkg_b']:
        conanfile = self.load_conanfile(os.path.join(self.folder, f))
        result.append({"path": f, "ref": f"{conanfile.name}/{conanfile.version}"})
      self.output.info(f"Workspace packages: {result}")
      return result
```

Supplying one of our profiles while invoking `conan workspace super-install ..` configures the projects but upon inspecting the generated presets file I found the cache variable definitions from packages be missing in the workspace' CMake presets.

Is that a defect indeed or just me holding it wrong?

Thanks.


Discussion:
Hi @mariuszwicker-cnhind 

Thanks for your feedback.

When using the ``super-install``, the individual packages ``generate()`` are not called at all. Only the ``generate()`` of the ``conanws.py`` is called, because you are defining just 1 super-build for all of them, not an individual build for every one.

That means that you would need to put the:
```cmake
tc.cache_variables["BUILD_CLIENT_TOOLS"] = self.options.build_client_tools
tc.cache_variables["BUILD_DOCS"] = self.options.get_safe('build_docs', False)
```
lines in the ``MyWs(ConanFile)`` workspace super-build conanfile.
I think this one cannot receive ``options`` at the moment, but the idea is that it could be simple, just hardcode the values you want to use, at the end of the day, the workspace is your current definition of workspace, you might be able to do that, so please try:
```cmake
tc.cache_variables["BUILD_CLIENT_TOOLS"] = True
tc.cache_variables["BUILD_DOCS"] = False
```

And see if it works.
I will have a look if options are being processed in that conanfile, and if this would be possible or not.
Thanks I appreciate you looking into how this could be improved. I expect this to be rather difficult to be implemented but an ideal solution (for us) would be if `generate()` for all individual packages would simply get transparently invoked even if in a workspace making this truly transparent. Can `workspace super-install` maybe invoke `workspace install` and merge the output?

If I need to manually put the cache variables into `MyWs` this means I kinda need to replicate everything that is in the individual packages `generate()` method into the workspace making this rather hard to maintain.
> Thanks I appreciate you looking into how this could be improved. I expect this to be rather difficult to be implemented but an ideal solution (for us) would be if generate() for all individual packages would simply get transparently invoked even if in a workspace making this truly transparent. Can workspace super-install maybe invoke workspace install and merge the output?

I am afraid that would be extremely challenging, the individual ``generate()`` methods generate individual ``conan_toolchain.cmake`` files for individual packages. These files might be even very incompatible, like it is perfectly possible that one package ``generate()`` activates its generation of docs, and another package in the workspace disables that generation of docs. Doing a quick check on ``conan-center-recipes`` I find some other few things with a high potential to create conflicts and issues if aggregated:
- Changes to ``CMAKE_POLICY_`` for individual projects
- Definition of ``BUILD_TESTING`` hardcoded or conditional to ``conf``
- Hardcoded ``BUILD_SHARED_LIBS``
- Projects that define a ``WITH_SSL`` with different values, like the path to openssl or True/False in different packages

While this is ok for individual packages ``generate()``, it is impossible to take them and aggregate them in a single toolchain, because of such incompatibilities. And the implementation of ``generate()`` methods can be arbitrarily complex, with conditionals on different things, so not something that Conan could analyze and aggregate at the method/source level.


> If I need to manually put the cache variables into MyWs this means I kinda need to replicate everything that is in the individual packages generate() method into the workspace making this rather hard to maintain.

I think it is possible, if you have some convention over your options configurations across packages to do something like:

```python
def generate(self):
  ...
  tc = CMakeToolchain(self)
  for k, v in self.dependencies.host.items():
      for myvar in ("BUILD_CLIENT_TOOLS", "BUILD_DOCS", ...):
           value = v.options.get_safe(myvar)
           if value is not None:
                   tc.cache_variables[myvar.upper()] = value 
```
And it will automatically collect those options values. That way you only need to maintain the list of toggles like ``"BUILD_CLIENT_TOOLS", "BUILD_DOCS"`` that you want to manage form the super-build

Yeah I agree, finding a generic solution is hard and even harder with an environment as complex as C++...

I think your proposal could be a direction to workaround this, however a brief test showed that `self.dependencies` does not contain the subpackages when using `workspace super-install` only the transitive packages outside the workspace. Is that expected or should I file a separate issue on this?
> I think your proposal could be a direction to workaround this, however a brief test showed that self.dependencies does not contain the subpackages when using workspace super-install only the transitive packages outside the workspace. Is that expected or should I file a separate issue on this?

No, you are right. In the ``super-install``, the workspace packages are actually collapsed in the graph, so they won't be part of ``self.dependencies`` in that way... I'll need to think about something else. I have just done a proof of concept, and it seems possible to add processing of ``options`` values in the workspace super-conanfile too, maybe that would be another approach to check, the list of configs is moved to ``options``, and then:
```python
options = {"build_client_tools": [True, False],
                 "build_docs": [True, False]}

def generate(self):
  ...
  tc = CMakeToolchain(self)
  for option in self.options.possible_values:
     value = v.options.get_safe(myvar)
     if value is not None:
         tc.cache_variables[myvar.upper()] = value
```
(This doesn't work at the moment, we'd need to provide support for that, this is the proof-of-concept I have in a branch)
If the workspace packages are collapsed could you still think of a way to access their defined options and assigned values? From looking at [`super_build_graph`](https://github.com/conan-io/conan/blob/develop2/conan/api/subapi/workspace.py#L241) it seems to ignore options altogether currently?

The elegance of your idea above would be that profiles could continue to assign options per individual package and only the generated presets would collapse them all into one for the `super-install`. 
I am investigating in https://github.com/conan-io/conan/pull/18608 a way to collect and give access to the super-build to the workspace packages options. This is very experimental, just an idea, but I'll check this idea with the team too.



Patch:
diff --git a/conan/api/subapi/workspace.py b/conan/api/subapi/workspace.py
index ddfec264a8c..65f2f0fff7c 100644
--- a/conan/api/subapi/workspace.py
+++ b/conan/api/subapi/workspace.py
@@ -11,13 +11,17 @@
 from conan.cli.printers.graph import print_graph_basic, print_graph_packages
 from conan.errors import ConanException
 from conan.internal.conan_app import ConanApp
+from conan.internal.errors import conanfile_exception_formatter
 from conan.internal.graph.install_graph import ProfileArgs
+from conan.internal.methods import auto_language, auto_shared_fpic_config_options, \
+    auto_shared_fpic_configure
+from conan.internal.model.options import Options
 from conan.internal.model.workspace import Workspace, WORKSPACE_YML, WORKSPACE_PY, WORKSPACE_FOLDER
 from conan.tools.scm import Git
 from conan.internal.graph.graph import RECIPE_EDITABLE, DepsGraph, CONTEXT_HOST, RECIPE_VIRTUAL, Node, \
     RECIPE_CONSUMER
 from conan.internal.graph.graph import TransitiveRequirement
-from conan.internal.graph.profile_node_definer import consumer_definer
+from conan.internal.graph.profile_node_definer import consumer_definer, initialize_conanfile_profile
 from conan.internal.loader import load_python_file
 from conan.internal.source import retrieve_exports_sources
 from conan.internal.util.files import merge_directories, save
@@ -238,13 +242,35 @@ def info(self):
                 "folder": self._folder,
                 "packages": self._ws.packages()}
 
+    @staticmethod
+    def _init_options(conanfile, options):
+        if hasattr(conanfile, "config_options"):
+            with conanfile_exception_formatter(conanfile, "config_options"):
+                conanfile.config_options()
+        elif "auto_shared_fpic" in conanfile.implements:
+            auto_shared_fpic_config_options(conanfile)
+
+        auto_language(conanfile)  # default implementation removes `compiler.cstd`
+
+        # Assign only the current package options values, but none of the dependencies
+        conanfile.options.apply_downstream(Options(), options, None, True)
+
+        if hasattr(conanfile, "configure"):
+            with conanfile_exception_formatter(conanfile, "configure"):
+                conanfile.configure()
+        elif "auto_shared_fpic" in conanfile.implements:
+            auto_shared_fpic_configure(conanfile)
+
     def super_build_graph(self, deps_graph, profile_host, profile_build):
         ConanOutput().title("Collapsing workspace packages")
 
         root_class = self._ws.root_conanfile()
         if root_class is not None:
             conanfile = root_class(f"{WORKSPACE_PY} base project Conanfile")
-            consumer_definer(conanfile, profile_host, profile_build)
+            initialize_conanfile_profile(conanfile, profile_build, profile_host, CONTEXT_HOST,
+                                         is_build_require=False)
+            # consumer_definer(conanfile, profile_host, profile_build)
+            self._init_options(conanfile, profile_host.options)
             root = Node(None, conanfile, context=CONTEXT_HOST, recipe=RECIPE_CONSUMER,
                         path=self._folder)  # path lets use the conanws.py folder
             root.should_build = True  # It is a consumer, this is something we are building
@@ -261,10 +287,12 @@ def super_build_graph(self, deps_graph, profile_host, profile_build):
 
         result = DepsGraph()  # TODO: We might need to copy more information from the original graph
         result.add_node(root)
+        conanfile.workspace_packages_options = {}
         for node in deps_graph.nodes[1:]:  # Exclude the current root
             if node.recipe != RECIPE_EDITABLE:
                 result.add_node(node)
                 continue
+            conanfile.workspace_packages_options[node.ref] = node.conanfile.options.serialize()
             for r, t in node.transitive_deps.items():
                 if t.node.recipe == RECIPE_EDITABLE:
                     continue
@@ -293,7 +321,6 @@ def export(self, lockfile=None, remotes=None):
             exported.append(ref)
         return exported
 
-
     def select_packages(self, packages):
         self._check_ws()
         editable = self.packages()


Test patch:
diff --git a/test/integration/workspace/test_workspace.py b/test/integration/workspace/test_workspace.py
index 1ae1863b50a..e9d3b10cff2 100644
--- a/test/integration/workspace/test_workspace.py
+++ b/test/integration/workspace/test_workspace.py
@@ -567,6 +567,94 @@ def test_install_partial(self):
             assert "dep1-config.cmake" in files
             assert "dep2-config.cmake" not in files
 
+    def test_workspace_options(self):
+        c = TestClient()
+        conanfilews = textwrap.dedent("""
+            from conan import ConanFile
+            from conan import Workspace
+            class MyWs(ConanFile):
+                settings = "arch", "build_type"
+                options = {"myoption": [1, 2, 3]}
+                def generate(self):
+                    self.output.info(f"Generating with my option {self.options.myoption}!!!!")
+
+            class Ws(Workspace):
+                def root_conanfile(self):
+                    return MyWs
+            """)
+
+        c.save({"dep/conanfile.py": GenConanfile("dep", "0.1"),
+                "conanws.py": conanfilews})
+        c.run("workspace add dep")
+        c.run("workspace super-install -of=build")
+        assert "conanws.py base project Conanfile: Generating with my option None!!!!" in c.out
+        c.run("workspace super-install -of=build -o *:myoption=1")
+        assert "conanws.py base project Conanfile: Generating with my option 1!!!!" in c.out
+
+    def test_workspace_common_shared_options(self):
+        c = TestClient()
+        conanfilews = textwrap.dedent("""
+            from conan import ConanFile
+            from conan import Workspace
+            class MyWs(ConanFile):
+                settings = "os"
+                options = {"shared": [True, False],
+                           "fPIC": [True, False]}
+                default_options = {"shared": False, "fPIC": True}
+                implements = "auto_shared_fpic"
+
+                def generate(self):
+                    self.output.info(f"OS={self.settings.os}!!!!")
+                    self.output.info(f"shared={self.options.shared}!!!!")
+                    self.output.info(f"fPIC={self.options.get_safe('fPIC')}!!!!")
+
+            class Ws(Workspace):
+                def root_conanfile(self):
+                    return MyWs
+            """)
+
+        c.save({"dep/conanfile.py": GenConanfile("dep", "0.1"),
+                "conanws.py": conanfilews})
+        c.run("workspace add dep")
+        c.run("workspace super-install -of=build -s os=Windows")
+        assert "conanws.py base project Conanfile: OS=Windows!!!!" in c.out
+        assert "conanws.py base project Conanfile: shared=False!!!!" in c.out
+        assert "conanws.py base project Conanfile: fPIC=None!!!!" in c.out
+        c.run("workspace super-install -of=build -s os=Linux -o *:shared=True")
+        assert "conanws.py base project Conanfile: OS=Linux!!!!" in c.out
+        assert "conanws.py base project Conanfile: shared=True!!!!" in c.out
+        assert "conanws.py base project Conanfile: fPIC=None!!!!" in c.out
+        c.run("workspace super-install -of=build -s os=Linux -o *:shared=False")
+        assert "conanws.py base project Conanfile: OS=Linux!!!!" in c.out
+        assert "conanws.py base project Conanfile: shared=False!!!!" in c.out
+        assert "conanws.py base project Conanfile: fPIC=True!!!!" in c.out
+
+    def test_workspace_pkg_options(self):
+        c = TestClient()
+        conanfilews = textwrap.dedent("""
+            from conan import ConanFile
+            from conan import Workspace
+
+            class MyWs(ConanFile):
+                settings = "arch", "build_type"
+                def generate(self):
+                    for pkg, options in self.workspace_packages_options.items():
+                        for k, v in options.items():
+                            self.output.info(f"Generating with opt {pkg}:{k}={v}!!!!")
+
+            class Ws(Workspace):
+                def root_conanfile(self):
+                    return MyWs
+            """)
+
+        c.save({"dep/conanfile.py": GenConanfile("dep", "0.1").with_option("myoption", [1, 2, 3]),
+                "conanws.py": conanfilews})
+        c.run("workspace add dep")
+        c.run("workspace super-install -of=build")
+        assert "project Conanfile: Generating with opt dep/0.1:myoption=None!!!!" in c.out
+        c.run("workspace super-install -of=build -o *:myoption=1")
+        assert "project Conanfile: Generating with opt dep/0.1:myoption=1!!!!" in c.out
+
 
 def test_workspace_with_local_recipes_index():
     c3i_folder = temp_folder()