Repository: conan-io/conan

Issue:
[feature] Add support for creating MultiPackagesList from graph
### What is your suggestion?

Currently, it's possible to create this object from [graph file](https://github.com/conan-io/conan/blob/release/2.0/conan/api/model.py#L75), however conan graph API returns the [graph object](https://github.com/conan-io/conan/blob/release/2.0/conan/api/subapi/graph.py#L141).

Now, in order to use `MultiPackagesList` to filter the graph and obtain the list of packages that have been built by dependency I need to serialize the graph object into the file and then load it. This is inefficient.

I'm currently working on a custom command that will:
- invoke `conan install --build missing` for each and every build type that we use in the company
- invoke `cmake --preset <generated-preset>` to automatically create the cmake project with installed dependencies

The idea is for this command to also return the list of packages that have been built by dependency, so I need to filter those packages from each and every dependency graph (for each build type), merge everything into a single PackageList, and serialize that into JSON using formatters so that the produced JSON is then compatible with `conan upload` command.

The custom command is intended to be used both by developers (in which case, the JSON pkglist of built dependencies is not important) and CI, where I would like to use the pkglist to upload the built binaries to the Artifactory server, similar to [this example in the Conan documentation](https://docs.conan.io/2/examples/commands/pkglists.html#building-and-uploading-packages).

Currently, I can copy the implementation of the linked code into my custom command, but it uses Conan's private API (in the `conans` package), which I'm not very comfortable using, as even a minor change in the conan version can break it.

Alternative workarounds are also welcome.

### Have you read the CONTRIBUTING guide?

- [ ] I've read the CONTRIBUTING guide
[feature] `conan list merge` command
### What is your suggestion?

This is akin to `conan lock merge`.

This is useful when you create a package and immediately build multiple binaries for it, with different configurations (e.g. debug, release, various options) and you want to upload all binaries in a single `conan upload` call so that you benefit from Conan's parallel upload feature.

The flow would be something like this (avoiding multiple `conan create` calls because each call creates a different revision for some reason).

```
conan export . --name pkg --version 1.0

conan install --requires pkg/1.0 -s build_type=Debug --format=json > graph-debug.json
conan install --requires pkg/1.0 -s build_type=Release --format=json > graph-release.json

conan list --graph=graph-debug.json --graph-binaries=build --format=json > built-debug.json
conan list --graph=graph-release.json --graph-binaries=build --format=json > built-release.json

conan list merge --list=built-debug.json --list=built-release.json --list-out=built.json

conan upload -r myremote -c --list=built.json
```

Of course, this can be implemented as a  custom command (but not as `conan list merge` because `list` command is already reserved). However, I think that having this command by default would be a good value.


Discussion:
Hi @DoDoENT 

Thanks for your suggestion.

It would be good to better understand this request, to see it as a more detailed interface.

At the moment, if you want to create a ``MultiPackagesLists`` from graph, it would be done by:

```python
conan_api.install.install_binaries(deps_graph=deps_graph, remotes=remotes)

# This is the current interface
save("myfile.json",  json.dumps({"graph": deps_graph.serialize()})
multi_pkg_list = MultiPackagesList.load_graph("myfile.json", graph_binaries=["build"])
```

Yes, maybe it is not the most efficient thing, but even for huge graphs this should take less than one second, so I am not sure if I understand the pain. What would be your suggestion here, to have a ``MultiPackagesList.from_graph(deps_graph, ...)`` directly?
It's not the pain - it's just the OCD 😃 . It annoys me to be inefficient when I don't need to be 😂 

I've seen in other parts of conan API that you sometimes use `load` to load something from a file, and `loads` to load the same thing from a string.

`from_graph` sounds like a good idea.

Btw, the support for custom commands and Conan Python API is really a game changer compared to v1. They are very flexible and provide the ability to create workflows that were really difficult with v1 and required a lot of custom scripts around conan. Great work! 👏 

Hi @DoDoENT thanks a lot for your request. While this is something that we think has some value being added, we need to think the UX/implementation of this one, as internally it's not straightforward to create subcommands for `conan list`.

In the meantime and for anyone else finding this problem, if you are using `jq` ([A really useful json manipulation tool](https://jqlang.github.io/jq/)), this is a built-in functionality of it: (A bit advanced with a peculiar syntax and not as easy as what conan will provide, but can help for now)

```
$ conan list "zlib/*:*#*" -p "build_type=Debug" --format=json > graph-debug.json

$ conan list "zlib/*:*#*" -p "build_type=Release" --format=json > graph-release.json

$ jq -s '.[0] * .[1]' graph-release.json graph-debug.json 
{
  "Local Cache": {
    "zlib/1.2.13": {
      "revisions": {
        "e377bee636333ae348d51ca90874e353": {
          "timestamp": 1682597484.674,
          "packages": {
            "76f7d863f21b130b4e6527af3b1d430f7f8edbea": {
              "revisions": {
                "75c9bf351695f8ce2f6b17458bd507ed": {
                  "timestamp": 1686666077.41796
                }
              },
              "info": {
                "settings": {
                  "arch": "armv8",
                  "build_type": "Release",
                  "compiler": "apple-clang",
                  "compiler.version": "14",
                  "os": "Macos"
                },
                "options": {
                  "fPIC": "True",
                  "shared": "False"
                }
              }
            },
            "8359538fee5b146800be789e3d3f504010b3bc01": {
              "revisions": {
                "dba9c2b0bd86a13f5770b91195e73888": {
                  "timestamp": 1686672985.748242
                }
              },
              "info": {
                "settings": {
                  "arch": "armv8",
                  "build_type": "Debug",
                  "compiler": "apple-clang",
                  "compiler.version": "14",
                  "os": "Macos"
                },
                "options": {
                  "fPIC": "True",
                  "shared": "False"
                }
              }
            }
          }
        }
      }
    }
  }
}
```
Great idea for jq! Never thought of it, even though I used it before for other stuff.

In the meantime, I've already created a custom command `merge-lists` that does the merging of `MultiPackageList` objects.

UX-wise, since subcommands for `list` are complicated, maybe

```
conan list merge --list=built-debug.json --list=built-release.json --list-out=built.json
```

could be something like

```
conan list --merge=built-debug.json --merge=built-release.json --format=json > built.json
```

I don't know if it's too confusing or not. It does look a bit unintuitive.



Patch:
diff --git a/conan/api/model.py b/conan/api/model.py
index 4a29bbea05f..9e23cdfe10e 100644
--- a/conan/api/model.py
+++ b/conan/api/model.py
@@ -47,6 +47,9 @@ class MultiPackagesList:
     def __init__(self):
         self.lists = {}
 
+    def setdefault(self, key, default):
+        return self.lists.setdefault(key, default)
+
     def __getitem__(self, name):
         try:
             return self.lists[name]
@@ -63,6 +66,10 @@ def serialize(self):
         return {k: v.serialize() if isinstance(v, PackagesList) else v
                 for k, v in self.lists.items()}
 
+    def merge(self, other):
+        for k, v in other.lists.items():
+            self.lists.setdefault(k, PackagesList()).merge(v)
+
     @staticmethod
     def load(file):
         content = json.loads(load(file))
@@ -76,9 +83,18 @@ def load(file):
         pkglist.lists = result
         return pkglist
 
+    @staticmethod
+    def from_graph(graph, graph_recipes=None, graph_binaries=None):
+        graph = {"graph": graph.serialize()}
+        return MultiPackagesList._define_graph(graph, graph_recipes, graph_binaries)
+
     @staticmethod
     def load_graph(graphfile, graph_recipes=None, graph_binaries=None):
         graph = json.loads(load(graphfile))
+        return MultiPackagesList._define_graph(graph, graph_recipes, graph_binaries)
+
+    @staticmethod
+    def _define_graph(graph, graph_recipes=None, graph_binaries=None):
         pkglist = MultiPackagesList()
         cache_list = PackagesList()
         if graph_recipes is None and graph_binaries is None:
@@ -143,6 +159,16 @@ class PackagesList:
     def __init__(self):
         self.recipes = {}
 
+    def merge(self, other):
+        def recursive_dict_update(d, u):  # TODO: repeated from conandata.py
+            for k, v in u.items():
+                if isinstance(v, dict):
+                    d[k] = recursive_dict_update(d.get(k, {}), v)
+                else:
+                    d[k] = v
+            return d
+        recursive_dict_update(self.recipes, other.recipes)
+
     def split(self):
         """
         Returns a list of PackageList, splitted one per reference.
diff --git a/conan/cli/commands/pkglist.py b/conan/cli/commands/pkglist.py
new file mode 100644
index 00000000000..ee8d4aaaa6e
--- /dev/null
+++ b/conan/cli/commands/pkglist.py
@@ -0,0 +1,93 @@
+import copy
+
+from conan.api.conan_api import ConanAPI
+from conan.api.model import MultiPackagesList, PackagesList
+from conan.cli import make_abs_path
+from conan.cli.command import conan_command, conan_subcommand
+from conan.cli.commands.list import print_list_text, print_list_json
+from conan.cli.formatters.list import list_packages_html
+from conans.errors import NotFoundException
+
+
+@conan_command(group="Consumer")
+def pkglist(conan_api: ConanAPI, parser, *args):  # noqa
+    """
+    Several operations over package lists
+    """
+
+
+@conan_subcommand(formatters={"text": print_list_text,
+                              "json": print_list_json,
+                              "html": list_packages_html})
+def pkglist_find_remote(conan_api, parser, subparser, *args):
+    """
+    (Experimental) Find the remotes of a list of packages in the cache
+    """
+    subparser.add_argument('list', help="Input package list")
+    subparser.add_argument("-r", "--remote", default=None, action="append",
+                           help="Remote names. Accepts wildcards "
+                                "('*' means all the remotes available)")
+    args = parser.parse_args(*args)
+
+    listfile = make_abs_path(args.list)
+    multi_pkglist = MultiPackagesList.load(listfile)
+    package_list = multi_pkglist["Local Cache"]
+    selected_remotes = conan_api.remotes.list(args.remote)
+
+    result = MultiPackagesList()
+    for r in selected_remotes:
+        result_pkg_list = PackagesList()
+        for ref, recipe_bundle in package_list.refs().items():
+            ref_no_rev = copy.copy(ref)  # TODO: Improve ugly API
+            ref_no_rev.revision = None
+            try:
+                revs = conan_api.list.recipe_revisions(ref_no_rev, remote=r)
+            except NotFoundException:
+                continue
+            if ref not in revs:  # not found
+                continue
+            result_pkg_list.add_refs([ref])
+            for pref, pref_bundle in package_list.prefs(ref, recipe_bundle).items():
+                pref_no_rev = copy.copy(pref)  # TODO: Improve ugly API
+                pref_no_rev.revision = None
+                try:
+                    prevs = conan_api.list.package_revisions(pref_no_rev, remote=r)
+                except NotFoundException:
+                    continue
+                if pref in prevs:
+                    result_pkg_list.add_prefs(ref, [pref])
+                    info = recipe_bundle["packages"][pref.package_id]["info"]
+                    result_pkg_list.add_configurations({pref: info})
+        if result_pkg_list.recipes:
+            result.add(r.name, result_pkg_list)
+
+    return {
+        "results": result.serialize(),
+        "conan_api": conan_api,
+        "cli_args": " ".join([f"{arg}={getattr(args, arg)}"
+                              for arg in vars(args) if getattr(args, arg)])
+    }
+
+
+@conan_subcommand(formatters={"text": print_list_text,
+                              "json": print_list_json,
+                              "html": list_packages_html})
+def pkglist_merge(conan_api, parser, subparser, *args):
+    """
+    (Experimental) Merge several package lists into a single one
+    """
+    subparser.add_argument("-l", "--list", help="Package list file", action="append")
+    args = parser.parse_args(*args)
+
+    result = MultiPackagesList()
+    for pkglist in args.list:
+        listfile = make_abs_path(pkglist)
+        multi_pkglist = MultiPackagesList.load(listfile)
+        result.merge(multi_pkglist)
+
+    return {
+        "results": result.serialize(),
+        "conan_api": conan_api,
+        "cli_args": " ".join([f"{arg}={getattr(args, arg)}"
+                              for arg in vars(args) if getattr(args, arg)])
+    }


Test patch:
diff --git a/conans/test/integration/command_v2/test_combined_pkglist_flows.py b/conans/test/integration/command_v2/test_combined_pkglist_flows.py
index 115f9d1acdc..fbd88317d0a 100644
--- a/conans/test/integration/command_v2/test_combined_pkglist_flows.py
+++ b/conans/test/integration/command_v2/test_combined_pkglist_flows.py
@@ -1,9 +1,10 @@
 import json
+from collections import OrderedDict
 
 import pytest
 
 from conans.test.assets.genconanfile import GenConanfile
-from conans.test.utils.tools import TestClient
+from conans.test.utils.tools import TestClient, TestServer
 
 
 class TestListUpload:
@@ -168,6 +169,92 @@ def test_graph_pkg_list_only_built(self):
         assert len(pkglist["default"]) == 2
 
 
+class TestPkgListFindRemote:
+    """ we can recover a list of remotes for an already installed graph, for metadata download
+    """
+    def test_graph_2_pkg_list_remotes(self):
+        servers = OrderedDict([("default", TestServer()), ("remote2", TestServer())])
+        c = TestClient(servers=servers, inputs=2 * ["admin", "password"], light=True)
+        c.save({"zlib/conanfile.py": GenConanfile("zlib", "1.0"),
+                "app/conanfile.py": GenConanfile("app", "1.0").with_requires("zlib/1.0")})
+        c.run("create zlib")
+        c.run("create app ")
+        c.run("upload zlib* -c -r=default")
+        c.run("upload zlib* -c -r=remote2")
+        c.run("upload app* -c -r=remote2")
+
+        # This install, packages will be in the cache
+        c.run("install --requires=app/1.0 --format=json", redirect_stdout="graph.json")
+        # So list, will not have remote at all
+        c.run("list --graph=graph.json --format=json", redirect_stdout="pkglist.json")
+
+        pkglist = json.loads(c.load("pkglist.json"))
+        assert len(pkglist["Local Cache"]) == 2
+        assert "default" not in pkglist  # The remote doesn't even exist
+
+        # Lets now compute a list finding in the remotes
+        c.run("pkglist find-remote pkglist.json --format=json", redirect_stdout="remotepkg.json")
+        pkglist = json.loads(c.stdout)
+        assert "Local Cache" not in pkglist
+        assert len(pkglist["default"]) == 1
+        assert "zlib/1.0" in pkglist["default"]
+        assert len(pkglist["remote2"]) == 2
+        assert "app/1.0" in pkglist["remote2"]
+        assert "zlib/1.0" in pkglist["remote2"]
+
+        c.run("download --list=remotepkg.json -r=default --metadata=*")
+        assert "zlib/1.0: Retrieving recipe metadata from remote 'default'" in c.out
+        assert "zlib/1.0: Retrieving package metadata" in c.out
+        c.run("download --list=remotepkg.json -r=remote2 --metadata=*")
+        assert "app/1.0: Retrieving recipe metadata from remote 'remote2'" in c.out
+        assert "app/1.0: Retrieving package metadata" in c.out
+
+
+class TestPkgListMerge:
+    """ deep merge lists
+    """
+    def test_graph_2_pkg_list_remotes(self):
+        servers = OrderedDict([("default", TestServer()), ("remote2", TestServer())])
+        c = TestClient(servers=servers, inputs=2 * ["admin", "password"])
+        c.save({"zlib/conanfile.py": GenConanfile("zlib", "1.0").with_settings("build_type"),
+                "bzip2/conanfile.py": GenConanfile("bzip2", "1.0").with_settings("build_type"),
+                "app/conanfile.py": GenConanfile("app", "1.0").with_requires("zlib/1.0", "bzip2/1.0")
+                                                              .with_settings("build_type")})
+        c.run("create zlib")
+        c.run("create bzip2")
+        c.run("create app ")
+
+        c.run("list zlib:* --format=json", redirect_stdout="list1.json")
+        c.run("list bzip2:* --format=json", redirect_stdout="list2.json")
+        c.run("list app:* --format=json", redirect_stdout="list3.json")
+        c.run("pkglist merge --list=list1.json --list=list2.json --list=list3.json --format=json",
+              redirect_stdout="release.json")
+        final = json.loads(c.stdout)
+        assert "app/1.0" in final["Local Cache"]
+        assert "zlib/1.0" in final["Local Cache"]
+        assert "bzip2/1.0" in final["Local Cache"]
+
+        c.run("create zlib -s build_type=Debug")
+        c.run("create bzip2 -s build_type=Debug")
+        c.run("create app -s build_type=Debug")
+        c.run("list *:* -fs build_type=Debug --format=json", redirect_stdout="debug.json")
+        c.run("pkglist merge --list=release.json --list=debug.json --format=json",
+              redirect_stdout="release.json")
+        final = json.loads(c.stdout)
+        rev = final["Local Cache"]["zlib/1.0"]["revisions"]["11f74ff5f006943c6945117511ac8b64"]
+        assert len(rev["packages"]) == 2  # Debug and Release
+        settings = rev["packages"]["efa83b160a55b033c4ea706ddb980cd708e3ba1b"]["info"]["settings"]
+        assert settings == {"build_type": "Release"}
+        settings = rev["packages"]["9e186f6d94c008b544af1569d1a6368d8339efc5"]["info"]["settings"]
+        assert settings == {"build_type": "Debug"}
+        rev = final["Local Cache"]["bzip2/1.0"]["revisions"]["9e0352b3eb99ba4ac79bc7eeae2102c5"]
+        assert len(rev["packages"]) == 2  # Debug and Release
+        settings = rev["packages"]["efa83b160a55b033c4ea706ddb980cd708e3ba1b"]["info"]["settings"]
+        assert settings == {"build_type": "Release"}
+        settings = rev["packages"]["9e186f6d94c008b544af1569d1a6368d8339efc5"]["info"]["settings"]
+        assert settings == {"build_type": "Debug"}
+
+
 class TestDownloadUpload:
     @pytest.fixture()
     def client(self):