Repository: conan-io/conan

Issue:
[feature] conan workspace build command
### What is your suggestion?

Hello!

As part of the workspace feature (#15992), I would like to have a `conan workspace build` command. The idea is that it would resolve the dependency graph and build missing editable packages in the cache (I expect that here, you would need to build all and rely on the underlying build system caching the build artifacts correctly).

In my [example repo](https://github.com/aander80/conan-workspace-test) where I am testing workspaces, I am thinking that I should be able to run `conan workspace build` and it will first build `liba` and `libb`, followed by `app` (`app` is an executable depending on static libraries `liba` and `libb`).

This would allow users to mimic a monorepo with multiple repos, and have a nice CLI similar to what you have with `cargo build` with workspaces in Rust.

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Hi @aander80 

Thanks for your suggestion.

I agree that this could be useful, and it is something we will discuss in next steps

At the moment, you can do:
```
conan build app --build=editable
```
This will build ``liba``, ``libb`` and ``app`` in the right order, because it resolves the dependency graph (you can use profile and settings arguments, of course).

The ``conan workspace build`` would be a bit of a higher level syntax over than, where mainly the "root" package of the workspace (in this case the ``app``) is defined in the workspace files. This is one of the aspects that we need to think, as it is possible to have different "roots" in workspaces.
> At the moment, you can do:
>
> ```conan build app --build=editable```

Thanks, I was looking for a command like that! :)

> The `conan workspace build` would be a bit of a higher level syntax over than, where mainly the "root" package of the workspace (in this case the `app`) is defined in the workspace files. This is one of the aspects that we need to think, as it is possible to have different "roots" in workspaces.

Would it make sense to check
* If there is a single root, build that
* If there are multiple roots and none is specified in the command:
  * Error
  * Build all roots
  * Something else?

I expect that it would be useful to specify `conan workspace build <root>` to a specific node in the workspace as root; this could be any of `liba`, `libb` and `app`, and is similar to `cargo build --bin <name>`.
> If there are multiple roots and none is specified in the command:

Yeah, this is exactly the kind of thing that needs to be considered. I think that it might be possible to iterate the builds of several roots. We also have the ``conan graph build-order`` capabilities, including merging build-orders, so this is definitely doable in an efficient way, but we also have to consider the complexities of implementation, the UX, etc.

> I expect that it would be useful to specify conan workspace build <root> to a specific node in the workspace as root; this could be any of liba, libb and app, and is similar to cargo build --bin <name>.

If that maps to ``conan build app`` then probably it is more boilerplate than helpful. 
But maybe what makes sense is to configure something in the workspace itself that can default the ``--build=editable`` if desired.
> But maybe what makes sense is to configure something in the workspace itself that can default the `--build=editable` if desired.

Yeah I expect that for the *most* part you would want that behavior by default, but I can also see cases where you would only want to build a single package but still use the editables from the workspace; a very simple example of this is if you have 10 libraries and building, even with caching, takes as little as 3 seconds, then you need to wait for 30 seconds for the application to even start building every time.

I am thinking that *if* default behavior is to build a single node in the workspace graph (e.g. `conan workspace build app` builds only `app` but not `liba` or `libb`), then you could run `conan workspace build app --build=workspace` which would build all nodes in the workspace that are included in the dependency tree from `app`.

For the most part, I guess that `--build=workspace` would be the same as `--build=editable`, but there are of course edge cases where you have statically configured editables outside of your workspace that you may not want to build.´, and `--build=workspace` is perhaps more explicit and intuitive.
I have started to work on this in https://github.com/conan-io/conan/pull/17538



Patch:
diff --git a/conan/api/subapi/new.py b/conan/api/subapi/new.py
index 8b3089a0c19..a9a99ef149b 100644
--- a/conan/api/subapi/new.py
+++ b/conan/api/subapi/new.py
@@ -32,6 +32,7 @@ def get_builtin_template(template_name):
         from conan.internal.api.new.autoools_exe import autotools_exe_files
         from conan.internal.api.new.local_recipes_index import local_recipes_index_files
         from conan.internal.api.new.qbs_lib import qbs_lib_files
+        from conan.internal.api.new.workspace import workspace_files
         new_templates = {"basic": basic_file,
                          "cmake_lib": cmake_lib_files,
                          "cmake_exe": cmake_exe_files,
@@ -48,7 +49,8 @@ def get_builtin_template(template_name):
                          "autotools_exe": autotools_exe_files,
                          "alias": alias_file,
                          "local_recipes_index": local_recipes_index_files,
-                         "qbs_lib": qbs_lib_files}
+                         "qbs_lib": qbs_lib_files,
+                         "workspace": workspace_files}
         template_files = new_templates.get(template_name)
         return template_files
 
diff --git a/conan/api/subapi/workspace.py b/conan/api/subapi/workspace.py
index 9485e769e44..02e01231480 100644
--- a/conan/api/subapi/workspace.py
+++ b/conan/api/subapi/workspace.py
@@ -19,9 +19,15 @@ def __init__(self, conan_api):
         self._workspace = Workspace(conan_api)
 
     def home_folder(self):
+        """
+        @return: The custom defined Conan home/cache folder if defined, else None
+        """
         return self._workspace.home_folder()
 
     def folder(self):
+        """
+        @return: the current workspace folder where the conanws.yml or conanws.py is located
+        """
         return self._workspace.folder
 
     def config_folder(self):
@@ -31,6 +37,10 @@ def config_folder(self):
     def editable_packages(self):
         return self._workspace.editables()
 
+    @property
+    def products(self):
+        return self._workspace.products
+
     def open(self, require, remotes, cwd=None):
         app = ConanApp(self._conan_api)
         ref = RecipeReference.loads(require)
@@ -63,21 +73,37 @@ def open(self, require, remotes, cwd=None):
         return dst_path
 
     def add(self, path, name=None, version=None, user=None, channel=None, cwd=None,
-            output_folder=None, remotes=None):
-        path = self._conan_api.local.get_conanfile_path(path, cwd, py=True)
+            output_folder=None, remotes=None, product=False):
+        """
+        Add a new editable package to the current workspace (the current workspace must exist)
+        @param path: The path to the folder containing the conanfile.py that defines the package
+        @param name: (optional) The name of the package to be added if not defined in recipe
+        @param version:
+        @param user:
+        @param channel:
+        @param cwd:
+        @param output_folder:
+        @param remotes:
+        @param product:
+        @return: The reference of the added package
+        """
+        full_path = self._conan_api.local.get_conanfile_path(path, cwd, py=True)
         app = ConanApp(self._conan_api)
-        conanfile = app.loader.load_named(path, name, version, user, channel, remotes=remotes)
+        conanfile = app.loader.load_named(full_path, name, version, user, channel, remotes=remotes)
         if conanfile.name is None or conanfile.version is None:
             raise ConanException("Editable package recipe should declare its name and version")
         ref = RecipeReference(conanfile.name, conanfile.version, conanfile.user, conanfile.channel)
         ref.validate_ref()
         output_folder = make_abs_path(output_folder) if output_folder else None
         # Check the conanfile is there, and name/version matches
-        self._workspace.add(ref, path, output_folder=output_folder)
+        self._workspace.add(ref, full_path, output_folder=output_folder,  product=product)
         return ref
 
-    def remove(self, path):
+    def remove(self, path, cwd=None):
         return self._workspace.remove(path)
 
     def info(self):
         return self._workspace.serialize()
+
+    def editable_from_path(self, path):
+        return self._workspace.editable_from_path(path)
diff --git a/conan/cli/commands/new.py b/conan/cli/commands/new.py
index ecc9615893a..374f9548e3b 100644
--- a/conan/cli/commands/new.py
+++ b/conan/cli/commands/new.py
@@ -16,7 +16,7 @@ def new(conan_api, parser, *args):
                         "either a predefined built-in or a user-provided one. "
                         "Available built-in templates: basic, cmake_lib, cmake_exe, "
                         "meson_lib, meson_exe, msbuild_lib, msbuild_exe, bazel_lib, bazel_exe, "
-                        "autotools_lib, autotools_exe, local_recipes_index. "
+                        "autotools_lib, autotools_exe, local_recipes_index, workspace. "
                         "E.g. 'conan new cmake_lib -d name=hello -d version=0.1'. "
                         "You can define your own templates too by inputting an absolute path "
                         "as your template, or a path relative to your conan home folder."
diff --git a/conan/cli/commands/workspace.py b/conan/cli/commands/workspace.py
index bbf19b607d8..3f3bbdcc32f 100644
--- a/conan/cli/commands/workspace.py
+++ b/conan/cli/commands/workspace.py
@@ -4,9 +4,11 @@
 from conan.api.conan_api import ConanAPI
 from conan.api.output import ConanOutput, cli_out_write
 from conan.cli import make_abs_path
-from conan.cli.args import add_reference_args
+from conan.cli.args import add_reference_args, add_common_install_arguments, add_lockfile_args
 from conan.cli.command import conan_command, conan_subcommand
 from conan.cli.commands.list import print_serial
+from conan.cli.printers import print_profiles
+from conan.cli.printers.graph import print_graph_packages, print_graph_basic
 from conan.errors import ConanException
 
 
@@ -56,6 +58,7 @@ def workspace_add(conan_api: ConanAPI, parser, subparser, *args):
                        help='Look in the specified remote or remotes server')
     group.add_argument("-nr", "--no-remote", action="store_true",
                        help='Do not use remote, resolve exclusively in the cache')
+    subparser.add_argument("--product", action="store_true", help="Add the package as a product")
     args = parser.parse_args(*args)
     if args.path and args.ref:
         raise ConanException("Do not use both 'path' and '--ref' argument")
@@ -67,7 +70,7 @@ def workspace_add(conan_api: ConanAPI, parser, subparser, *args):
         path = conan_api.workspace.open(args.ref, remotes, cwd=cwd)
     ref = conan_api.workspace.add(path,
                                   args.name, args.version, args.user, args.channel,
-                                  cwd, args.output_folder, remotes=remotes)
+                                  cwd, args.output_folder, remotes=remotes, product=args.product)
     ConanOutput().success("Reference '{}' added to workspace".format(ref))
 
 
@@ -101,6 +104,66 @@ def workspace_info(conan_api: ConanAPI, parser, subparser, *args):
     return {"info": conan_api.workspace.info()}
 
 
+@conan_subcommand()
+def workspace_build(conan_api: ConanAPI, parser, subparser, *args):
+    """
+    Build the current workspace, starting from the "products"
+    """
+    subparser.add_argument("path", nargs="?",
+                           help='Path to a package folder in the user workspace')
+    add_common_install_arguments(subparser)
+    add_lockfile_args(subparser)
+    args = parser.parse_args(*args)
+    # Basic collaborators: remotes, lockfile, profiles
+    remotes = conan_api.remotes.list(args.remote) if not args.no_remote else []
+    overrides = eval(args.lockfile_overrides) if args.lockfile_overrides else None
+    # The lockfile by default if not defined will be read from the root workspace folder
+    ws_folder = conan_api.workspace.folder()
+    lockfile = conan_api.lockfile.get_lockfile(lockfile=args.lockfile, conanfile_path=ws_folder,
+                                               cwd=None,
+                                               partial=args.lockfile_partial, overrides=overrides)
+    profile_host, profile_build = conan_api.profiles.get_profiles_from_args(args)
+    print_profiles(profile_host, profile_build)
+
+    build_mode = args.build or []
+    if "editable" not in build_mode:
+        ConanOutput().info("Adding '--build=editable' as build mode")
+        build_mode.append("editable")
+
+    if args.path:
+        products = [args.path]
+    else:  # all products
+        products = conan_api.workspace.products
+        if products is None:
+            raise ConanException("There are no products defined in the workspace, can't build\n"
+                                 "You can use 'conan build <path> --build=editable' to build")
+        ConanOutput().title(f"Building workspace products {products}")
+
+    editables = conan_api.workspace.editable_packages
+    # TODO: This has to be improved to avoid repetition when there are multiple products
+    for product in products:
+        ConanOutput().subtitle(f"Building workspace product: {product}")
+        product_ref = conan_api.workspace.editable_from_path(product)
+        if product_ref is None:
+            raise ConanException(f"Product '{product}' not defined in the workspace as editable")
+        editable = editables[product_ref]
+        editable_path = editable["path"]
+        deps_graph = conan_api.graph.load_graph_consumer(editable_path, None, None, None, None,
+                                                         profile_host, profile_build, lockfile,
+                                                         remotes, args.update)
+        deps_graph.report_graph_error()
+        print_graph_basic(deps_graph)
+        conan_api.graph.analyze_binaries(deps_graph, build_mode, remotes=remotes, update=args.update,
+                                         lockfile=lockfile)
+        print_graph_packages(deps_graph)
+        conan_api.install.install_binaries(deps_graph=deps_graph, remotes=remotes)
+        conan_api.install.install_consumer(deps_graph, None, os.path.dirname(editable_path),
+                                           editable.get("output_folder"))
+        ConanOutput().title(f"Calling build() for the product {product_ref}")
+        conanfile = deps_graph.root.conanfile
+        conan_api.local.build(conanfile)
+
+
 @conan_command(group="Consumer")
 def workspace(conan_api, parser, *args):
     """
diff --git a/conan/internal/api/new/workspace.py b/conan/internal/api/new/workspace.py
new file mode 100644
index 00000000000..7a56617eb94
--- /dev/null
+++ b/conan/internal/api/new/workspace.py
@@ -0,0 +1,33 @@
+from conan.api.subapi.new import NewAPI
+from conan.internal.api.new.cmake_exe import cmake_exe_files
+from conan.internal.api.new.cmake_lib import cmake_lib_files
+
+
+conanws_yml = """\
+editables:
+  liba/0.1:
+    path: liba
+  libb/0.1:
+    path: libb
+  app1/0.1:
+    path: app1
+products:
+- app1
+"""
+
+workspace_files = {"conanws.yml": conanws_yml}
+
+for lib in ("liba", "libb"):
+    definitions = {"name": lib}
+    if lib == "libb":
+        definitions["requires"] = ["liba/0.1"]
+    elif lib == "libc":
+        definitions["requires"] = ["libb/0.1"]
+    files = NewAPI.render(cmake_lib_files, definitions)
+    files = {f"{lib}/{k}": v for k, v in files.items()}
+    workspace_files.update(files)
+
+
+files = NewAPI.render(cmake_exe_files, definitions={"name": "app1", "requires": ["libb/0.1"]})
+files = {f"app1/{k}": v for k, v in files.items()}
+workspace_files.update(files)
diff --git a/conan/internal/workspace.py b/conan/internal/workspace.py
index 1abd54ebe1c..ae076e6dbcc 100644
--- a/conan/internal/workspace.py
+++ b/conan/internal/workspace.py
@@ -72,6 +72,10 @@ def __init__(self, conan_api):
     def name(self):
         return self._attr("name") or os.path.basename(self._folder)
 
+    @property
+    def products(self):
+        return self._attr("products")
+
     @property
     def folder(self):
         return self._folder
@@ -102,17 +106,21 @@ def _check_ws(self):
             raise ConanException("Workspace not defined, please create a "
                                  "'conanws.py' or 'conanws.yml' file")
 
-    def add(self, ref, path, output_folder):
+    def add(self, ref, path, output_folder, product=False):
         """
         Add a new editable to the current workspace 'conanws.yml' file.
         If existing, the 'conanws.py' must use this via 'conanws_data' attribute
         """
         self._check_ws()
         self._yml = self._yml or {}
-        editable = {"path": self._rel_path(path)}
+        assert os.path.isfile(path)
+        path = self._rel_path(os.path.dirname(path))
+        editable = {"path": path}
         if output_folder:
             editable["output_folder"] = self._rel_path(output_folder)
         self._yml.setdefault("editables", {})[str(ref)] = editable
+        if product:
+            self._yml.setdefault("products", []).append(path)
         save(self._yml_file, yaml.dump(self._yml))
 
     def _rel_path(self, path):
@@ -126,29 +134,40 @@ def _rel_path(self, path):
                                  f"{self._folder}")
         return path.replace("\\", "/")  # Normalize to unix path
 
+    def editable_from_path(self, path):
+        editables = self._attr("editables")
+        for ref, info in editables.items():
+            if info["path"].replace("\\", "/") == path:
+                return RecipeReference.loads(ref)
+
     def remove(self, path):
         self._check_ws()
         self._yml = self._yml or {}
         found_ref = None
         path = self._rel_path(path)
         for ref, info in self._yml.get("editables", {}).items():
-            if os.path.dirname(info["path"]).replace("\\", "/") == path:
+            if info["path"].replace("\\", "/") == path:
                 found_ref = ref
                 break
         if not found_ref:
             raise ConanException(f"No editable package to remove from this path: {path}")
         self._yml["editables"].pop(found_ref)
+        if path in self._yml.get("products", []):
+            self._yml["products"].remove(path)
         save(self._yml_file, yaml.dump(self._yml))
         return found_ref
 
     def editables(self):
+        """
+        @return: Returns {RecipeReference: {"path": full abs-path, "output_folder": abs-path}}
+        """
         if not self._folder:
             return
         editables = self._attr("editables")
         if editables:
             editables = {RecipeReference.loads(r): v.copy() for r, v in editables.items()}
             for v in editables.values():
-                v["path"] = os.path.normpath(os.path.join(self._folder, v["path"]))
+                v["path"] = os.path.normpath(os.path.join(self._folder, v["path"], "conanfile.py"))
                 if v.get("output_folder"):
                     v["output_folder"] = os.path.normpath(os.path.join(self._folder,
                                                                        v["output_folder"]))
@@ -158,4 +177,5 @@ def serialize(self):
         self._check_ws()
         return {"name": self.name,
                 "folder": self._folder,
+                "products": self._attr("products"),
                 "editables": self._attr("editables")}


Test patch:
diff --git a/test/integration/workspace/test_workspace.py b/test/integration/workspace/test_workspace.py
index 2dbb50917ba..18b25b2567a 100644
--- a/test/integration/workspace/test_workspace.py
+++ b/test/integration/workspace/test_workspace.py
@@ -153,8 +153,7 @@ def editables():
                             name = open(os.path.join(workspace_folder, f, "name.txt")).read().strip()
                             version = open(os.path.join(workspace_folder, f,
                                                         "version.txt")).read().strip()
-                            p = os.path.join(f, "conanfile.py").replace("\\\\", "/")
-                            result[f"{name}/{version}"] = {"path": p}
+                            result[f"{name}/{version}"] = {"path": f}
                     return result
                 """)
         else:
@@ -166,8 +165,7 @@ def editables(*args, **kwargs):
                    result = {}
                    for f in os.listdir(workspace_api.folder):
                        if os.path.isdir(os.path.join(workspace_api.folder, f)):
-                           f = os.path.join(f, "conanfile.py").replace("\\\\", "/")
-                           conanfile = workspace_api.load(f)
+                           conanfile = workspace_api.load(os.path.join(f, "conanfile.py"))
                            result[f"{conanfile.name}/{conanfile.version}"] = {"path": f}
                    return result
                """)
@@ -178,12 +176,12 @@ def editables(*args, **kwargs):
                 "dep1/version.txt": "2.1"})
         c.run("workspace info --format=json")
         info = json.loads(c.stdout)
-        assert info["editables"] == {"pkg/2.1": {"path": "dep1/conanfile.py"}}
+        assert info["editables"] == {"pkg/2.1": {"path": "dep1"}}
         c.save({"dep1/name.txt": "other",
                 "dep1/version.txt": "14.5"})
         c.run("workspace info --format=json")
         info = json.loads(c.stdout)
-        assert info["editables"] == {"other/14.5": {"path": "dep1/conanfile.py"}}
+        assert info["editables"] == {"other/14.5": {"path": "dep1"}}
         c.run("install --requires=other/14.5")
         # Doesn't fail
         assert "other/14.5 - Editable" in c.out
@@ -239,6 +237,15 @@ def test_add_open_error(self):
         c.run("workspace open dep/0.1", assert_error=True)
         assert "ERROR: Can't open a dependency that is already an editable: dep/0.1" in c.out
 
+    def test_remove_product(self):
+        c = TestClient(light=True)
+        c.save({"conanws.py": "name='myws'",
+                "mydeppkg/conanfile.py": GenConanfile("mydeppkg", "0.1")})
+        c.run("workspace add mydeppkg --product")
+        c.run("workspace remove mydeppkg")
+        c.run("workspace info")
+        assert "mydeppkg" not in c.out
+
 
 class TestOpenAdd:
     def test_without_git(self):
@@ -319,3 +326,80 @@ def test_workspace_build_editables(self):
                                              "EditableBuild")})
         assert "pkga/0.1: WARN: BUILD PKGA!" in c.out
         assert "pkgb/0.1: WARN: BUILD PKGB!" in c.out
+
+
+class TestWorkspaceBuild:
+    def test_dynamic_products(self):
+        c = TestClient(light=True)
+
+        workspace = textwrap.dedent("""\
+            import os
+            name = "myws"
+
+            workspace_folder = os.path.dirname(os.path.abspath(__file__))
+
+            def products():
+                result = []
+                for f in os.listdir(workspace_folder):
+                    if os.path.isdir(os.path.join(workspace_folder, f)):
+                        if f.startswith("product"):
+                            result.append(f)
+                return result
+                    """)
+
+        c.save({"conanws.py": workspace,
+                "lib1/conanfile.py": GenConanfile("lib1", "0.1"),
+                "product_app1/conanfile.py": GenConanfile("app1", "0.1")})
+        c.run("workspace add lib1")
+        c.run("workspace add product_app1")
+        c.run("workspace info --format=json")
+        info = json.loads(c.stdout)
+        assert info["products"] == ["product_app1"]
+
+    def test_build(self):
+        c = TestClient(light=True)
+        c.save({"conanws.yml": ""})
+
+        c.save({"pkga/conanfile.py": GenConanfile("pkga", "0.1").with_build_msg("BUILD PKGA!"),
+                "pkgb/conanfile.py": GenConanfile("pkgb", "0.1").with_build_msg("BUILD PKGB!")
+               .with_requires("pkga/0.1")})
+        c.run("workspace add pkga")
+        c.run("workspace add pkgb --product")
+        c.run("workspace info --format=json")
+        assert json.loads(c.stdout)["products"] == ["pkgb"]
+        c.run("workspace build")
+        c.assert_listed_binary({"pkga/0.1": ("da39a3ee5e6b4b0d3255bfef95601890afd80709",
+                                             "EditableBuild")})
+        assert "pkga/0.1: WARN: BUILD PKGA!" in c.out
+        assert "conanfile.py (pkgb/0.1): WARN: BUILD PKGB!" in c.out
+
+        # It is also possible to build a specific package by path
+        # equivalent to ``conan build <path> --build=editable``
+        # This can be done even if it is not a product
+        c.run("workspace build pkgc", assert_error=True)
+        assert "ERROR: Product 'pkgc' not defined in the workspace as editable" in c.out
+        c.run("workspace build pkgb")
+        c.assert_listed_binary({"pkga/0.1": ("da39a3ee5e6b4b0d3255bfef95601890afd80709",
+                                             "EditableBuild")})
+        assert "pkga/0.1: WARN: BUILD PKGA!" in c.out
+        assert "conanfile.py (pkgb/0.1): WARN: BUILD PKGB!" in c.out
+
+        c.run("workspace build pkga")
+        assert "conanfile.py (pkga/0.1): Calling build()" in c.out
+        assert "conanfile.py (pkga/0.1): WARN: BUILD PKGA!" in c.out
+
+    def test_error_if_no_products(self):
+        c = TestClient(light=True)
+        c.save({"conanws.yml": ""})
+        c.run("workspace build", assert_error=True)
+        assert "There are no products defined in the workspace, can't build" in c.out
+
+
+def test_new():
+    # Very basic workspace for testing
+    c = TestClient()
+    c.run("new workspace")
+    c.run("workspace info")
+    assert "liba/0.1" in c.out
+    assert "libb/0.1" in c.out
+    assert "app1/0.1" in c.out