Repository: conan-io/conan

Issue:
No way to query existence of package on remote using profiles
I would like to be able to check whether a binary package for a particular set of settings and options exists on a remote.
As of Conan 1.16.1 there is no way to do this. There are two approaches that almost work:

```sh
conan install -r remote Package/1.0.0@user/channel -pr profileA -pr profile B
```
* Settings and options can be specified through profiles (good)
* Exit code is nonzero if binary package doesn't exist on remote (good)
* Binary package has to actually be downloaded; there is no flag such as `--dry-run` which would simply report if it exists


```sh
conan search -r remote Package/1.0.0@user/channel --query "os=Linux AND compiler=gcc AND compiler.version=5 AND option1=True AND option2=CustomValue
```
* Does not support settings and options from profiles; all this information has to be replicated in the query syntax, which is cumbersome
* Exit code is always 0, even if no packages matching the query are found

- [x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).
- [x] I've specified the Conan version, operating system version and any tool that can be relevant.
- [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.


[bug] Inconsistent behaviour between package queries and package-id
### Environment details

* Operating System+version: Ubuntu 22.04
* Compiler+version: N/A
* Conan version: 2.1.0
* Python version: 3.10.8


### Steps to reproduce

1. Have a package that does not define any "settings".
2. Try to download this package, but filter by some setting, e.g: ```conan download package/1.0.0 -r myremote -p os=Linux```
3. Conan will return ```ERROR: 404: Not Found. [Remote: myremote]```

It is possible to remove settings which are not relevant for the package, as documented here:
https://docs.conan.io/2/reference/conanfile/methods/package_id.html

```
A given package recipe can decide in its package_id() that the final binary is independent of some settings, for example if it is a header-only library, that uses input settings to build some tests, it might completely clear all configuration, so the resulting package_id is always the same irrespective of the inputs.
```
This is a very useful feature, since Conan is famously very strict about package-id. However, having queries filter by these deleted settings brings back the problem for me.

FYI:
What I'm _really_ trying to achieve is to download a pre-compiled package so I can access the binaries it contains, but there is no simple way to do this. You can use ```conan install```, but then all of the package's dependencies are also downloaded, which I'm not interested in (they could also be very large). Or, you can use ```conan download```, but then you have to use a completely different syntax to narrow down which package you want, which is confusing for my team and, I assume, other users as well.

What I would really love is to be able to use profiles with ```conan download```, and have the semantics be the same as when you use ```conan install```. Such as:
```conan download package/1.0.0 -r myremote -pr myprofile --settings build_type=Release```

And if ```build_type``` is not a setting for ```package/1.0.0``` it should be ignored.

In absence of that, I could live with settings being treated the same for package queries and package IDs.

### Logs

_No response_


Discussion:
Also, a close alternative to ``install`` would be ``info`` which in theory it is a ``install`` that does not actually install binaries, so it would be equivalent to ``dry-run`` functionality. 

So:
```bash
conan info boost/1.69.0@conan/stable -r=conan-center -s compiler=qcc -s compiler.libcxx=gpp -s compiler.version=4.4
boost/1.69.0@conan/stable
    ID: 5c1390cbdb5a2bbbd9562d0dbe8333a3234205d6
    ...
    Binary: Missing
```
Will produce an output telling that the binary is missing, but yes, it will return 0, not an error.

Same with ``--json`` would produce a machine readable json with ``{"build_id": null, "binary": "Missing"`` field.

I think this is a reasonable use case, and we need to think of some UI if possible to simplify access to this information.
First step should be investigating and discussing what would be the best UI for this use case.
Closing as solved in Conan v2. The `conan list` command is able to list packages for the passed profiles, `conan graph info`, which is able to report the whole status of the dependency graph for one or multiple references for the passed profiles, and `conan graph explain`, which is able to print information regarding the reasons why a certain binary is missing :)
Reopening to track progress for adding profile passing for `conan list`, which turns out I was misremembering with internal development progress, and does not have profile inputs (yet!). Thanks @memsharded for heads-up :)

Hi @antoinebardoz 

You are right, the ``-p ...`` argument is only looking for "positive" matches.
We had developed the possibility to pass profiles to these commands, with the goal of helping with the "Missing binaries" case, but it was lately replaced with the ``conan graph explain`` command.

The problem with the ``conan download package/1.0.0 --profile=myprofile`` approach is that it doesn't take into account dependencies at all, because they are not modeled in profiles. So if the ``package/1.0.0`` has different binaries, lets say, because it linked with different versions of dependencies ``zlib/1.2.11``, ``zlib/1.2.12``, ..., ``zlib/1.3`` and others, it could have many different binaries and it is not that evident which one would be the one you want to use. Because when you are downloading that package in particular is because you want to use it somehow?

In that regard, the recommendation in general is to use package-lists. When a dependency graph is computed, it can return a json with the graph, that can generate a package-list. That package-list is the mapping between a given profile and the different binaries you need.


In any case, I understand the use case of passing a profile to these commands (though maybe we only consider for ``conan list``, then you pass the result to ``conan download``, to avoid the proliferation of arguments in all commands), so I will present this to the team to reconsider. Thanks for the feedback!
>The problem with the ```conan download package/1.0.0 --profile=myprofile``` approach is that it doesn't take into account dependencies at all

Well, in this case I would say "One man's trash is another man's treasure". For this application, I have no interest in the dependencies, I simply want to download a package so I can get its contents. If I wanted the dependencies, I could use ```conan install``` instead, for example with a custom deployer.

> So if the package/1.0.0 has different binaries, lets say, because it linked with different versions of dependencies zlib/1.2.11, zlib/1.2.12, ..., zlib/1.3 and others, it could have many different binaries and it is not that evident which one would be the one you want to use.

I understand the problem, but this is a good example of how Conan can sometimes be needlessly strict. What if there is only a single package in my remote that matches my query? Why should Conan prevent me from downloading this package without me providing an entire dependency graph?

While I await your possible fix, I have implemented my own custom command. Its behaviour:
1. If there is a single package which matches a reference, like ```package/1.0.0```, you don't even need to give a profile: ```conan mydownload package/1.0.0```
2. If there is more than one package that matches, the command asks if I want to download them all, or cancel the download entirely. Then I can choose if I want to be more specific, and add a profile, option or setting requirement. As in: ```conan mydownload package/1.0.0 -pr myprofile --options myoption=True```.

I wish Conan was a bit more lenient like this in general. 

>  Because when you are downloading that package in particular is because you want to use it somehow?

This may be a strange and ill-considered use of Conan, but we use Conan to manage compiled binaries of our code. It seemed to us a logical use, since Conan has an immutable database of compiled packages, why not use Conan to download certain versions of our binaries? However, considering how hard it is to get this to work smoothly, maybe we're the only ones who thought this was a good idea?
> Well, in this case I would say "One man's trash is another man's treasure". For this application, I have no interest in the dependencies, I simply want to download a package so I can get its contents. If I wanted the dependencies, I could use conan install instead, for example with a custom deployer.


> I understand the problem, but this is a good example of how Conan can sometimes be needlessly strict. What if there is only a single package in my remote that matches my query? Why should Conan prevent me from downloading this package without me providing an entire dependency graph?

Of course, this is why the ``conan download`` command exists, to not download the full dependencies, just the package that you want. I am not saying that it should download the dependencies. Lets say that you are downloading your application executable ``myapp/1.0``, the binary for Linux. This application has been built and linked statically against several versions of ``zlib``, so you have different binaries with different executables for ``myapp/1.0``. When you ``conan download myapp/1.0 -p "os=Linux"`` it will download you 3 different binaries, executables, and it will not download any dependencies. You can execute any of them, and the 3 different executables will run with different speeds because they statically linked different versions of ``zlib``.

> If there is a single package which matches a reference, like package/1.0.0, you don't even need to give a profile: conan mydownload package/1.0.0

But this is already the default behavior of ``conan download`` isn't it? If there is only one binary, it will download only that one binary?


> This may be a strange and ill-considered use of Conan, but we use Conan to manage compiled binaries of our code. It seemed to us a logical use, since Conan has an immutable database of compiled packages, why not use Conan to download certain versions of our binaries? However, considering how hard it is to get this to work smoothly, maybe we're the only ones who thought this was a good idea?

Of course not. This is the reason for example that the ``direct_deploy`` exist, you can do:
```
conan install  --requires=myapp/1.0 --deployer=direct_deploy --output-folder=my_output
```
And it will get you a local copy of the package, with several advantages:
- the Conan binary model will actually "skip" the downloading of unnecessary dependencies. If you are installing an executable that links statically against libraries, those libraries binaries will not be downloaded at all
- You can provide a ``--lockfile=mylockfile`` to guarantee that you are obtaining your ``myapp/1.0`` that was linked against the specific set of dependencies, like the exact version of zlib/1.3 you want. This is a very strict requirement for many of our production users, they don' want to download a binary for Linux of their application, but they want to enforce that binary was linked against a very specific and locked set of dependencies.


In any case, I am not saying that there isn't value in providing this functionality, and we will try to add it soon. I was just suggesting that even if it is provided, there are some use cases that we have learnt from other users that are very important and wouldn't be satisfied by this feature.
Thank you for taking the time to help. It's quite difficult to be precise in what I'm trying to describe, especially because I'm trying not to waste your time with information you don't need, so there are a few misunderstandings.

> I am not saying that it should download the dependencies.

Sorry, I was a bit unclear in my message. I understand why there might be multiple package revisions depending on which dependencies the package was built with. I just meant that in the case where only one package matches there would be no issue, and for us this is a common case.

In fact, now what I think about it, I'm not sure why it would conceptually be a problem that multiple packages could match, versus the current behaviour. The way it works now, ```conan download``` will already download multiple packages if it matches with more than one. Why is that different with profiles?

> Of course not. This is the reason for example that the direct_deploy exist

```conan install``` works well for the case where I'm trying to compile something new, and I want to get the dependencies in a folder. However, the use case I'm trying to describe is different.

Our final output is a pre-compiled image, which we will run on an embedded target. We create a conan package that has the image in it, this is our release. At this stage, the dependencies are already baked into the final package, since it is an image. Therefore it would, in many scenarios, be useful to download just this package, and _ideally_ output it to a folder, instead of the conan cache.

It's this particular case, where we create a conan package of our final image, which will never be consumed by another Conan project, that I wonder if might be a use case you don't consider. And maybe we should instead use another mechanism?

My dream implementation of ```conan download``` would be:
1. Allows us to use profiles, settings and options to filter, rather than package queries (since they do not take into account that some settings may be disabled in some packages). Also, the syntax would be closer to other Conan commands, which would make things less confusing for inexperienced users.
2. Allow us to download the package into an output folder. Maybe this is a case where ```conan download``` could fail if more than one package was found? Ideally, it could print a list of the packages which matched, to help users figure out what they need to specify.
Sure, sounds good.

> Allows us to use profiles, settings and options to filter, rather than package queries (since they do not take into account that some settings may be disabled in some packages). Also, the syntax would be closer to other Conan commands, which would make things less confusing for inexperienced users.

yes, this is what we have in mind, and we agree it makes sense

> Allow us to download the package into an output folder. Maybe this is a case where conan download could fail if more than one package was found? Ideally, it could print a list of the packages which matched, to help users figure out what they need to specify.

That might be a bit too much for the ``download`` command (too much responsibility), but maybe a custom command will be almost trivial for this custom logic (we are also adding the capability of custom commands calling other commands, which will simplify a lot this flow). In any case, lets go first for the ``--profile`` argument which is the core functionality, and we can discuss this later, this might be a feature request to something kind of deployers for ``download`` too.



Patch:
diff --git a/conan/api/subapi/list.py b/conan/api/subapi/list.py
index 343992ac0a2..f748de9ac48 100644
--- a/conan/api/subapi/list.py
+++ b/conan/api/subapi/list.py
@@ -6,7 +6,7 @@
 from conans.errors import ConanException, NotFoundException
 from conans.model.info import load_binary_info
 from conans.model.package_ref import PkgReference
-from conans.model.recipe_ref import RecipeReference
+from conans.model.recipe_ref import RecipeReference, ref_matches
 from conans.search.search import get_cache_packages_binary_info, filter_packages
 from conans.util.dates import timelimit
 
@@ -87,7 +87,35 @@ def filter_packages_configurations(pkg_configurations, query):
         """
         return filter_packages(query, pkg_configurations)
 
-    def select(self, pattern, package_query=None, remote=None, lru=None):
+    @staticmethod
+    def filter_packages_profile(packages, profile, ref):
+        result = {}
+        profile_settings = profile.processed_settings.serialize()
+        # Options are those for dependencies, like *:shared=True
+        profile_options = profile.options._deps_package_options
+        for pref, data in packages.items():
+            settings = data.get("settings", {})
+            settings_match = options_match = True
+            for k, v in settings.items():  # Only the defined settings that don't match
+                value = profile_settings.get(k)
+                if value is not None and value != v:
+                    settings_match = False
+                    break
+            options = data.get("options", {})
+            for k, v in options.items():
+                for pattern, pattern_options in profile_options.items():
+                    if ref_matches(ref, pattern, None):
+                        value = pattern_options.get_safe(k)
+                        if value is not None and value != v:
+                            options_match = False
+                            break
+
+            if settings_match and options_match:
+                result[pref] = data
+
+        return result
+
+    def select(self, pattern, package_query=None, remote=None, lru=None, profile=None):
         if package_query and pattern.package_id and "*" not in pattern.package_id:
             raise ConanException("Cannot specify '-p' package queries, "
                                  "if 'package_id' is not a pattern")
@@ -150,6 +178,8 @@ def msg_format(msg, item, total):
                     packages = self.packages_configurations(rrev, remote)
                     if package_query is not None:
                         packages = self.filter_packages_configurations(packages, package_query)
+                    if profile is not None:
+                        packages = self.filter_packages_profile(packages, profile, rrev)
                     prefs = packages.keys()
                     prefs = pattern.filter_prefs(prefs)
                     packages = {pref: conf for pref, conf in packages.items() if pref in prefs}
diff --git a/conan/cli/commands/list.py b/conan/cli/commands/list.py
index 61b067a766d..743c30ac0dc 100644
--- a/conan/cli/commands/list.py
+++ b/conan/cli/commands/list.py
@@ -212,6 +212,12 @@ def list(conan_api: ConanAPI, parser, *args):
     parser.add_argument('-p', '--package-query', default=None, action=OnceArgument,
                         help="List only the packages matching a specific query, e.g, os=Windows AND "
                              "(arch=x86 OR compiler=gcc)")
+    parser.add_argument('-fp', '--filter-profile', action="append",
+                        help="Profiles to filter the binaries")
+    parser.add_argument('-fs', '--filter-settings', action="append",
+                        help="Settings to filter the binaries")
+    parser.add_argument('-fo', '--filter-options', action="append",
+                        help="Options to filter the binaries")
     parser.add_argument("-r", "--remote", default=None, action="append",
                         help="Remote names. Accepts wildcards ('*' means all the remotes available)")
     parser.add_argument("-c", "--cache", action='store_true', help="Search in the local cache")
@@ -227,10 +233,13 @@ def list(conan_api: ConanAPI, parser, *args):
 
     if args.pattern is None and args.graph is None:
         raise ConanException("Missing pattern or graph json file")
-    if args.pattern and args.graph:
-        raise ConanException("Cannot define both the pattern and the graph json file")
-    if args.graph and args.lru:
-        raise ConanException("Cannot define lru when loading a graph json file")
+    if args.graph:  # a few arguments are not compatible with this
+        if args.pattern:
+            raise ConanException("Cannot define both the pattern and the graph json file")
+        if args.lru:
+            raise ConanException("Cannot define lru when loading a graph json file")
+        if args.filter_profile or args.filter_settings or args.filter_options:
+            raise ConanException("Filtering binaries cannot be done when loading a graph json file")
     if (args.graph_recipes or args.graph_binaries) and not args.graph:
         raise ConanException("--graph-recipes and --graph-binaries require a --graph input")
     if args.remote and args.lru:
@@ -245,8 +254,12 @@ def list(conan_api: ConanAPI, parser, *args):
         pkglist = MultiPackagesList()
         if args.cache or not args.remote:
             try:
+                profile = conan_api.profiles.get_profile(args.filter_profile or [],
+                                                         args.filter_settings,
+                                                         args.filter_options) \
+                    if args.filter_profile or args.filter_settings or args.filter_options else None
                 cache_list = conan_api.list.select(ref_pattern, args.package_query, remote=None,
-                                                   lru=args.lru)
+                                                   lru=args.lru, profile=profile)
             except Exception as e:
                 pkglist.add_error("Local Cache", str(e))
             else:


Test patch:
diff --git a/conans/test/integration/command_v2/list_test.py b/conans/test/integration/command_v2/list_test.py
index 47abd3c4f11..a7f9c109baa 100644
--- a/conans/test/integration/command_v2/list_test.py
+++ b/conans/test/integration/command_v2/list_test.py
@@ -793,3 +793,64 @@ def test_list_compact_no_settings_no_options(self):
             """)
 
         assert expected == expected_output
+
+
+class TestListBinaryFilter:
+    def test_list_filter(self):
+        c = TestClient()
+        c.save({"pkg/conanfile.py": GenConanfile("pkg", "1.0").with_settings("os", "arch")
+                                                              .with_shared_option(False),
+                "header/conanfile.py": GenConanfile("header", "1.0"),
+                "profile_linux": "[settings]\nos=Linux",
+                "profile_armv8": "[settings]\narch=armv8",
+                "profile_shared": "[options]\n*:shared=True"})
+        c.run("create pkg -s os=Windows -s arch=x86")
+        c.run("create pkg -s os=Linux -s arch=armv8")
+        c.run("create pkg -s os=Macos -s arch=armv8 -o shared=True")
+        c.run("create header")
+
+        c.run("list *:* -fp=profile_linux --format=json")
+        result = json.loads(c.stdout)
+        header = result["Local Cache"]["header/1.0"]["revisions"]["747cc49983b14bdd00df50a0671bd8b3"]
+        assert header["packages"] == {"da39a3ee5e6b4b0d3255bfef95601890afd80709": {"info": {}}}
+        pkg = result["Local Cache"]["pkg/1.0"]["revisions"]["03591c8b22497dd74214e08b3bf2a56f"]
+        assert len(pkg["packages"]) == 1
+        settings = pkg["packages"]["2d46abc802bbffdf2af11591e3e452bc6149ea2b"]["info"]["settings"]
+        assert settings == {"arch": "armv8", "os": "Linux"}
+
+        # for linux + x86 only the header-only is a match
+        c.run("list *:* -fp=profile_linux -fs=arch=x86 --format=json")
+        result = json.loads(c.stdout)
+        header = result["Local Cache"]["header/1.0"]["revisions"]["747cc49983b14bdd00df50a0671bd8b3"]
+        assert header["packages"] == {"da39a3ee5e6b4b0d3255bfef95601890afd80709": {"info": {}}}
+        pkg = result["Local Cache"]["pkg/1.0"]["revisions"]["03591c8b22497dd74214e08b3bf2a56f"]
+        assert pkg["packages"] == {}
+
+        c.run("list *:* -fp=profile_armv8 --format=json")
+        result = json.loads(c.stdout)
+        header = result["Local Cache"]["header/1.0"]["revisions"]["747cc49983b14bdd00df50a0671bd8b3"]
+        assert header["packages"] == {"da39a3ee5e6b4b0d3255bfef95601890afd80709": {"info": {}}}
+        pkg = result["Local Cache"]["pkg/1.0"]["revisions"]["03591c8b22497dd74214e08b3bf2a56f"]
+        assert len(pkg["packages"]) == 2
+        settings = pkg["packages"]["2d46abc802bbffdf2af11591e3e452bc6149ea2b"]["info"]["settings"]
+        assert settings == {"arch": "armv8", "os": "Linux"}
+        settings = pkg["packages"]["2a67a51fbf36a4ee345b2125dd2642be60ffd3ec"]["info"]["settings"]
+        assert settings == {"arch": "armv8", "os": "Macos"}
+
+        c.run("list *:* -fp=profile_shared --format=json")
+        result = json.loads(c.stdout)
+        header = result["Local Cache"]["header/1.0"]["revisions"]["747cc49983b14bdd00df50a0671bd8b3"]
+        assert header["packages"] == {"da39a3ee5e6b4b0d3255bfef95601890afd80709": {"info": {}}}
+        pkg = result["Local Cache"]["pkg/1.0"]["revisions"]["03591c8b22497dd74214e08b3bf2a56f"]
+        assert len(pkg["packages"]) == 1
+        settings = pkg["packages"]["2a67a51fbf36a4ee345b2125dd2642be60ffd3ec"]["info"]["settings"]
+        assert settings == {"arch": "armv8", "os": "Macos"}
+
+        c.run("list *:* -fs os=Windows -fo *:shared=False --format=json")
+        result = json.loads(c.stdout)
+        header = result["Local Cache"]["header/1.0"]["revisions"]["747cc49983b14bdd00df50a0671bd8b3"]
+        assert header["packages"] == {"da39a3ee5e6b4b0d3255bfef95601890afd80709": {"info": {}}}
+        pkg = result["Local Cache"]["pkg/1.0"]["revisions"]["03591c8b22497dd74214e08b3bf2a56f"]
+        assert len(pkg["packages"]) == 1
+        settings = pkg["packages"]["d2e97769569ac0a583d72c10a37d5ca26de7c9fa"]["info"]["settings"]
+        assert settings == {"arch": "x86", "os": "Windows"}