Repository: conan-io/conan

Issue:
[question] Can conan sort out package dependency order when using workspace super-install?
### What is your question?

I'm using the workspace template (generated from conan new workspace) to explore workspaces with a super-install. When using this template, do I have ensure that add_project() calls must be ordered such that dependencies are declared before they are used?

For example when using the stock workspace template, if I move the add_project(app1) from below add_project(libb) to above it, it will fail to build. The "conan workspace super-install" will complete, however the following call to "cmake --preset conan-release" will fail because it can't find the libb package.

For a large project cmake does a reasonably good job of building the dependency tree such that add_subdirectory() calls do not have be in a specific order.  Is there a way to use conan workspaces with a super-project to accomplish this?

In the above setup I'm using Ubuntu 22.04, Conan 2.19.0 and cmake 3.25.2.

### Have you read the CONTRIBUTING guide?

- [x] I've read the CONTRIBUTING guide


Discussion:
Hi @ScottGTI 

Thanks for your question.

This is indeed a very good point. I also thought that the ``FetchContent`` would behave similar to the ``add_subdirectory()`` in that regard, and it would do a better job with the order, but apparently it doesn't.

Conan has the ``conan graph build-order`` command that can show you the build order for any graph (with json and html outputs too), so if you give it the path to one of your workspace consumer packages, it will give you the right order.

I have also done https://github.com/conan-io/conan/pull/18693, that will print this order directly in the ``conan workspace super-install`` command for convenience. 

It might also be possible to add CMakeLists.txt-editing capabilities to the ``workspace`` commands, like it could automatically re-order the ``add_project()`` calls, but at the moment it seems a bit too much, what do you think? If the command gives the order will be enough?
Thanks for your input. It is too bad that the `FetchContent` doesn't behave well, it's so close.

I looked at the `conan graph build-order` and it provides a lot of information, but would require a bit of tooling to make work. After looking at the graph output, I would see the following:

* Run `conan workspace super-install`
* Run a tool that would:
  * Search the local tree to find all of the `conanfile.py` files.  For each `conanfile.py`
    * Either load or parse the `conanfile.py` to find the name of the package. I think this step is necessary because the output json for the `conan graph` command always refers to package names and version, but not the `conanfile.py` it came from. 
    * Run `conan graph build-order --format=json`
    * Parse the json file to determine build order for that package.
  * Create a global build order based on all of the build orders of the individual graph outputs
  * Write or edit the `CMakeLists.txt` to add the various projects in the correct order.
* Run cmake commands
* Rejoice!

Does that sound like the necessary steps? Is there an easier way?

I would worry that such local tooling would be rather brittle and may not fit well within the developers workflow. My preference, of course, would be to have conan handle this complexity internally. Some possible approaches:
* Have conan workspace edit the CMakeLists.txt, as you suggested
* Is it possible for conan to export a cmake function that looks something like `conan_add_project( LIST OF DIRS )` that just does the right thing?
* If the above tooling is still required, would it be possible to have a `conan workspace graph` command that provides the build order for the entire super-build?
* Would a conan extension, with access to its rich APIs, be a better place for this tooling?

These are just wild guesses at possible approaches as I'm not familiar with conan internals. I'm sure there are other options as well.




Patch:
diff --git a/conan/api/subapi/workspace.py b/conan/api/subapi/workspace.py
index 65f2f0fff7c..b09f084fe8a 100644
--- a/conan/api/subapi/workspace.py
+++ b/conan/api/subapi/workspace.py
@@ -262,6 +262,19 @@ def _init_options(conanfile, options):
             auto_shared_fpic_configure(conanfile)
 
     def super_build_graph(self, deps_graph, profile_host, profile_build):
+        order = []
+        packages = self._ws.packages()
+
+        def find_folder(ref):
+            return next(p["path"] for p in packages if RecipeReference.loads(p["ref"]) == ref)
+
+        for level in deps_graph.by_levels():
+            items = [item for item in level if item.recipe == "Editable"]
+            items = [{"ref": item.ref, "folder": find_folder(item.ref)} for item in items]
+            if items:
+                order.append(items)
+        self._ws.build_order(order)
+
         ConanOutput().title("Collapsing workspace packages")
 
         root_class = self._ws.root_conanfile()
diff --git a/conan/cli/commands/workspace.py b/conan/cli/commands/workspace.py
index 8f85e1fc7bb..084b19a621d 100644
--- a/conan/cli/commands/workspace.py
+++ b/conan/cli/commands/workspace.py
@@ -86,7 +86,7 @@ def workspace_remove(conan_api: ConanAPI, parser, subparser, *args):
     ConanOutput().info(f"Removed from workspace: {removed}")
 
 
-def print_json(data):
+def _print_json(data):
     results = data["info"]
     myjson = json.dumps(results, indent=4)
     cli_out_write(myjson)
@@ -103,7 +103,7 @@ def _print_workspace_info(data):
     print_serial(data["info"])
 
 
-@conan_subcommand(formatters={"text": _print_workspace_info, "json": print_json})
+@conan_subcommand(formatters={"text": _print_workspace_info, "json": _print_json})
 def workspace_info(conan_api: ConanAPI, parser, subparser, *args):  # noqa
     """
     Display info for current workspace
@@ -217,7 +217,7 @@ def workspace_super_install(conan_api: ConanAPI, parser, subparser, *args):
     requires = conan_api.workspace.select_packages(args.pkg)
     deps_graph = conan_api.graph.load_graph_requires(requires, [],
                                                      profile_host, profile_build, lockfile,
-                                                     remotes, args.build, args.update)
+                                                     remotes, update=args.update)
     deps_graph.report_graph_error()
     print_graph_basic(deps_graph)
 
@@ -225,7 +225,6 @@ def workspace_super_install(conan_api: ConanAPI, parser, subparser, *args):
     ws_graph = conan_api.workspace.super_build_graph(deps_graph, profile_host, profile_build)
     ConanOutput().subtitle("Collapsed graph")
     print_graph_basic(ws_graph)
-
     conan_api.graph.analyze_binaries(ws_graph, args.build, remotes=remotes, update=args.update,
                                      lockfile=lockfile)
     print_graph_packages(ws_graph)
@@ -250,8 +249,8 @@ def workspace_init(conan_api: ConanAPI, parser, subparser, *args):
     Clean the temporary build folders when possible
     """
     subparser.add_argument("path", nargs="?", default=os.getcwd(),
-                        help="Path to a folder where the workspace will be initialized. "
-                             "If  does not exist")
+                           help="Path to a folder where the workspace will be initialized. "
+                                "If does not exist")
     args = parser.parse_args(*args)
     conan_api.workspace.init(args.path)
 
diff --git a/conan/internal/api/new/workspace.py b/conan/internal/api/new/workspace.py
index b82c753843f..0a400283495 100644
--- a/conan/internal/api/new/workspace.py
+++ b/conan/internal/api/new/workspace.py
@@ -19,13 +19,8 @@
 
 include(FetchContent)
 
-function(add_project PACKAGE_NAME)
-    if(ARGC GREATER 1)
-        set(SUBFOLDER "${ARGV1}")
-    else()
-        set(SUBFOLDER "${PACKAGE_NAME}")
-    endif()
-    message(STATUS "Adding project: ${PACKAGE_NAME}")
+function(add_project PACKAGE_NAME SUBFOLDER)
+    message(STATUS "Adding project: ${PACKAGE_NAME}. Folder: ${SUBFOLDER}")
     FetchContent_Declare(
         ${PACKAGE_NAME}
         SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/${SUBFOLDER}
@@ -35,17 +30,28 @@
     FetchContent_MakeAvailable(${PACKAGE_NAME})
 endfunction()
 
-add_project(liba)
-# They should be defined in the liba/CMakeLists.txt, but we can fix it here
-add_library(liba::liba ALIAS liba)
-add_project(libb)
-add_library(libb::libb ALIAS libb)
-add_project(app1)
+include(build/conanws_build_order.cmake)
+
+foreach(pair ${CONAN_WS_BUILD_ORDER})
+    string(FIND "${pair}" ":" pos)
+    string(SUBSTRING "${pair}" 0 "${pos}" pkg)
+    math(EXPR pos "${pos} + 1")  # Skip the separator
+    string(SUBSTRING "${pair}" "${pos}" -1 folder)
+
+    add_project(${pkg} ${folder})
+    # This target should be defined in the liba/CMakeLists.txt, but we can fix it here
+    get_target_property(target_type ${pkg} TYPE)
+    if (NOT target_type STREQUAL "EXECUTABLE")
+        add_library(${pkg}::${pkg} ALIAS ${pkg})
+    endif()
+endforeach()
 """
 
-conanfile = '''\
+conanfile = r'''\
+import json
 from conan import Workspace
 from conan import ConanFile
+from conan.tools.files import save
 from conan.tools.cmake import CMakeDeps, CMakeToolchain, cmake_layout
 
 
@@ -69,6 +75,11 @@ def layout(self):
 class Ws(Workspace):
     def root_conanfile(self):
         return MyWs
+
+    def build_order(self, order):
+        super().build_order(order)  # default behavior prints the build order
+        pkglist = " ".join([f'{it["ref"].name}:{it["folder"]}' for level in order for it in level])
+        save(self, "build/conanws_build_order.cmake", f"set(CONAN_WS_BUILD_ORDER {pkglist})")
 '''
 
 workspace_files = {"conanws.yml": conanws_yml,
diff --git a/conan/internal/model/workspace.py b/conan/internal/model/workspace.py
index 38c12a34821..3e04ce0b958 100644
--- a/conan/internal/model/workspace.py
+++ b/conan/internal/model/workspace.py
@@ -23,7 +23,7 @@ def __init__(self, folder, conan_api):
         self.folder = folder
         self.conan_data = self._conan_load_data()
         self._conan_api = conan_api
-        self.output = ConanOutput(scope=self.name())
+        self.output = ConanOutput(scope=f"Workspace '{self.name()}'")
 
     def name(self):
         return self.conan_data.get("name") or os.path.basename(self.folder)
@@ -104,3 +104,10 @@ def load_conanfile(self, conanfile_path):
 
     def root_conanfile(self):  # noqa
         return None
+
+    def build_order(self, order):  # noqa
+        msg = ["Packages build order:"]
+        for level in order:
+            for item in level:
+                msg.append(f"    {item['ref']}")
+        self.output.info("\n".join(msg))


Test patch:
diff --git a/test/functional/workspace/test_workspace.py b/test/functional/workspace/test_workspace.py
index e1099618e48..f2207071810 100644
--- a/test/functional/workspace/test_workspace.py
+++ b/test/functional/workspace/test_workspace.py
@@ -63,9 +63,6 @@ def test_new_template_and_different_folder():
     replace_in_file(ConanFileMock(), os.path.join(c.current_folder, "conanws.yml"),
                     "  - path: liba",
                     "  - path: libX")
-    replace_in_file(ConanFileMock(), os.path.join(c.current_folder, "CMakeLists.txt"),
-                    "add_project(liba)",
-                    "add_project(liba libX)")
     c.run("workspace super-install")
     config_preset = "conan-default" if platform.system() == "Windows" else "conan-release"
     c.run_command(f"cmake --preset {config_preset}")  # it does not fail
diff --git a/test/integration/workspace/test_workspace.py b/test/integration/workspace/test_workspace.py
index e9d3b10cff2..fa4ff4a8c82 100644
--- a/test/integration/workspace/test_workspace.py
+++ b/test/integration/workspace/test_workspace.py
@@ -279,11 +279,11 @@ def remove(self, path, *args, **kwargs):
         c.save({"conanws.py": workspace,
                 "dep/conanfile.py": GenConanfile("dep", "0.1")})
         c.run("workspace add dep")
-        assert "myws: Adding dep/0.1" in c.out
+        assert "Workspace 'myws': Adding dep/0.1" in c.out
         c.run("workspace info")
         assert "dep/0.1" in c.out
         c.run("workspace remove dep")
-        assert "myws: Removing" in c.out
+        assert "Workspace 'myws': Removing" in c.out
         c.run("workspace info")
         assert "dep/0.1" not in c.out
 
@@ -488,7 +488,9 @@ def test_install(self):
                clean_first=True)
         c.run("workspace add liba")
         c.run("workspace add libb")
-        c.run("workspace super-install -g CMakeDeps -g CMakeToolchain -of=build --envs-generation=false")
+        c.run("workspace super-install -g CMakeDeps -g CMakeToolchain -of=build "
+              "--envs-generation=false")
+        assert "Packages build order:\n    liba/0.1\n    libb/0.1" in c.out
         assert "Workspace conanws.py not found in the workspace folder, using default" in c.out
         files = os.listdir(os.path.join(c.current_folder, "build"))
         assert "conan_toolchain.cmake" in files
@@ -695,8 +697,8 @@ def test_clean(self):
         assert os.path.exists(os.path.join(c.current_folder, "build", "pkga"))
         assert os.path.exists(os.path.join(c.current_folder, "build", "pkgb"))
         c.run("workspace clean")
-        assert "my_workspace: Removing pkga/0.1 output folder" in c.out
-        assert "my_workspace: Removing pkgb/0.1 output folder" in c.out
+        assert "Workspace 'my_workspace': Removing pkga/0.1 output folder" in c.out
+        assert "Workspace 'my_workspace': Removing pkgb/0.1 output folder" in c.out
         assert "Editable pkgc/0.1 doesn't have an output_folder defined" in c.out
         assert not os.path.exists(os.path.join(c.current_folder, "build", "pkga"))
         assert not os.path.exists(os.path.join(c.current_folder, "build", "pkgb"))
@@ -714,7 +716,7 @@ def clean(self):
         c = TestClient()
         c.save({"conanws.py": conanfilews})
         c.run("workspace clean")
-        assert "my_workspace: MY CLEAN!!!" in c.out
+        assert "Workspace 'my_workspace': MY CLEAN!!!" in c.out
 
 
 def test_relative_paths():