Repository: conan-io/conan

Issue:
Weird behavior of --update
I have a bunch of remotes defined in my Conan configuration. We'll call them A, B and C. They are defined in the same order so remote A should be used first, then B, then C.

Now I continuously upload new versions for the package `Pkg/x.x.x.x@sick/dev` to remote C. These are required by a consumer application using `requires = "Pkg/[*]@sick/dev"`. I would have expected that a call to `conan install .. --update` in the consumer's build directory queries all of the remotes for new versions. However, that doesn't seem to be the case. It always resolves to the last installed version in the local cache. I have to explicitly add a `-r C` to the install command to make it fetch the most recent version from the remote. With `--update` Conan emits the message `%s versions not found in remotes` (sic). It seems that the corresponding line in the Conan sources is also missing the format expression.


Discussion:
Without the [revisions mechanism](https://docs.conan.io/en/latest/versioning/revisions.html) enabled, that is the expected (we decided like that) behavior because usually, you don't want to mix binaries from different remotes unless you are sure that the binary corresponds to the recipe. So if you download the recipe from a remote, it will search only by default in that remote unless, as you said, specify manually a different one.  That is different when using revisions because you have the trust that the recipe is the same (because it has the same revision) so it will search in all the remotes for a binary package corresponding to the correct recipe revision.
Nevertheless, I agree with you, the message could be better, but I think it is not obvious to improve.
Actually I have revisions enabled on the remotes and also in the local client. Maybe there is a problem with the installed recipe because I moved the recipes between remotes lately.

Nevertheless I'm not sure if I fully understand the logic that you are describing. My expectation when using version ranges like `Pkg/[*]@sick/dev` is that it resolves to the most recent package version, independent of revisions. If revisions are enabled then it should resolve to the most recent recipe revision of the most recent package version, unless I manually specify a recipe revision using `Pkg/x@sick/dev#12345`. As long as the recipe revision matches, the associated binary packages should be compatible. Can you confirm that it behaves like this?

The next question would be how `--update` interacts with that. What I understand from the documentation is that update enables re-resolving of the dependency graph, making Conan download new **recipes** (if the version range matches a more recent recipe version/revision) and then look for matching binary packages.
Without the `--update` it will use the local cache resolved package.
With the `--update` it should resolve the latest version in the first remote where the recipe is found (or the already associated). That version will have, of course, a recipe revision. Then it should try to find binary packages for that recipe revision, but If I recall correctly it should iterate the remotes sequentially if there is no binary available. Is it not the experienced behavior?
Ah I think I know what the problem is. Where does Conan store the information about the *already associated* remote for a given recipe?
In an internal file at `~/.conan/data/zlib/1.2.11/conan/stable/metadata.json` for each package.
Hm, it was set to `None`, so that can't be the cause of the issue. What I tried now is to clear my local cache completely and then install the requirements for the consumer from scratch using `conan install`. I still need to specify `-r C` explicitly, otherwise Conan does not find the packages. It seems that it's not iterating the remotes sequentially or it's stopping somewhere in between. What I get instead is the error message `ERROR: Version range '*' from requirement 'Pkg/[*]@sick/dev' required by 'conanfile.py (Device/2.10.0)' could not be resolved in local cache`. When I remove the remotes A and B (which come before C) so that C is the first remote in the configuration it seems to work without manually specifying `-r C`. I checked the other remote repositories. There are no matching recipes for that package in `sick/dev` but there are some with another user/channel.
After deleting the old packages from remote B (which had a different user/channel `asaljo/testing`) Conan now finds the packages on C as expected. Do you have an explanation why the packages on B prevented Conan from considering the ones on C? Different user/channel combinations should not influence one another, do they? There was a recipe with revision 0 (not SCM) and exactly one binary package on B. I switched to revision_mode SCM later before uploading the packages to C. Maybe this could be a problem? Generally I think the error messages in Conan should be improved because sometimes it's really hard to find out why something is not working as expected.
Mhhh, no, the user/channel cannot be the cause. Are you sure you have the (revisions)[https://docs.conan.io/en/latest/versioning/revisions.html#package-revisions] enabled? Maybe if you paste here some trace of the install I can guess what is happening.
The revision feature is enabled in the client and supported by our Artifactory server. Here is my list of remotes:

```
gbc05-conan: https://deagxartifactory.sickcn.net/artifactory/api/conan/gbc05-conan [Verify SSL: True]
euler-conan-local: https://deagxartifactory.sickcn.net/artifactory/api/conan/euler-conan-local [Verify SSL: True]
edp-conan-local: https://deagxartifactory.sickcn.net/artifactory/api/conan/edp-conan-local [Verify SSL: True]
cache: https://deagxartifactory.sickcn.net/artifactory/api/conan/conan [Verify SSL: True]
```

My consumer currently has the following requirements:

```
requires = "BASE/[5.x]@sick/dev", "CmnAlg/[*]@sick/dev"
```

Recipes for these are usually uploaded to the `edp-conan-local` remote. As soon as I upload a recipe `BASE/5.0.0.0@asaljo/testing` (note the different user/channel) to one of the remotes further up in the list (`gbc05-conan`, `euler-conan-local`) it seems to be broken.

Here is the trace log of a `conan install .. -pr profile_gcc8_preinstalled --build missing` from the consumer's working copy:

```
{"_action": "COMMAND", "name": "install", "parameters": {"build": ["missing"], "channel": null, "env": null, "generators": null, "install_folder": null, "lockfile": null, "manifests": null, "manifests_interactive": null, "name": null, "no_imports": false, "options": null, "path": "..", "profile_names": ["/mnt/D/git/edp_toolchains/edp_profiles/profiles/profile_gcc8_preinstalled"], "remote_name": null, "settings": null, "update": false, "user": null, "verify": null, "version": null}, "time": 1566388341.6162546}
{"_action": "REST_API_CALL", "duration": 0.03098297119140625, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.18.1 (Python 3.5.2) python-requests/2.9.1", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "asaljo"}, "method": "GET", "time": 1566388341.7016444, "url": "https://deagxartifactory.sickcn.net/artifactory/api/conan/gbc05-conan/v1/ping"}
{"_action": "REST_API_CALL", "duration": 0.10257411003112793, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.18.1 (Python 3.5.2) python-requests/2.9.1", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "asaljo"}, "method": "GET", "time": 1566388341.8054533, "url": "https://deagxartifactory.sickcn.net/artifactory/api/conan/gbc05-conan/v2/conans/search?ignorecase=False&q=BASE"}
{"_action": "REST_API_CALL", "duration": 0.009664773941040039, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.18.1 (Python 3.5.2) python-requests/2.9.1", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "asaljo"}, "method": "GET", "time": 1566388341.8168635, "url": "https://deagxartifactory.sickcn.net/artifactory/api/conan/euler-conan-local/v1/ping"}
{"_action": "REST_API_CALL", "duration": 0.00891566276550293, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.18.1 (Python 3.5.2) python-requests/2.9.1", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "asaljo"}, "method": "GET", "time": 1566388341.8268132, "url": "https://deagxartifactory.sickcn.net/artifactory/api/conan/euler-conan-local/v2/conans/search?ignorecase=False&q=BASE"}
{"_action": "EXCEPTION", "class": "ConanException", "message": "Version range '5.x' from requirement 'BASE/[5.x]@sick/dev' required by 'conanfile.py (TemplateDevice/2.10.0)' could not be resolved in local cache", "time": 1566388341.8285372}
{"_action": "COMMAND", "name": "remote_list", "parameters": {}, "time": 1566388476.684236}
```

As you can see it stops at the remote that has the recipe with different user/channel and doesn't even contact the next one in the list. Binary packages are not even involved here.

If both of the recipes are in the same user/channel, it works fine!
Ok, I think the implementation of the function `RangeResolver._search_remotes()` is wrong. It loops over the remotes in sequence and exits after finding the first match based on the **name** only. That list of search results is then filtered for the matching user/channel **afterwards** which doesn't make sense because it will never look in the remaining remotes after the first user/channel mismatch.
Thanks for the detailed information. I agree something doesn't look good. Labeling as Bug, we will try to take a look asap.
I reproduced the error with a test and I have a fix. Let's try to move it for the next minor 1.18.2.
Thanks for your big help! :)

Resolving patch:
diff --git a/conans/client/graph/range_resolver.py b/conans/client/graph/range_resolver.py
--- a/conans/client/graph/range_resolver.py
+++ b/conans/client/graph/range_resolver.py
@@ -143,16 +143,16 @@ def _resolve_local(self, search_ref, version_range):
         if local_found:
             return self._resolve_version(version_range, local_found)
 
-    def _search_remotes(self, pattern, remotes):
-        remote = remotes.selected
-        if remote:
-            search_result = self._remote_manager.search_recipes(remote, pattern, ignorecase=False)
-            return search_result, remote.name
-
+    def _search_remotes(self, search_ref, remotes):
         for remote in remotes.values():
-            search_result = self._remote_manager.search_recipes(remote, pattern, ignorecase=False)
-            if search_result:
-                return search_result, remote.name
+            if not remotes.selected or remote == remotes.selected:
+                search_result = self._remote_manager.search_recipes(remote, search_ref.name,
+                                                                    ignorecase=False)
+                search_result = [ref for ref in search_result
+                                 if ref.user == search_ref.user and
+                                 ref.channel == search_ref.channel]
+                if search_result:
+                    return search_result, remote.name
         return None, None
 
     def _resolve_remote(self, search_ref, version_range, remotes):
@@ -160,10 +160,7 @@ def _resolve_remote(self, search_ref, version_range, remotes):
         found_refs, remote_name = self._cached_remote_found.get(search_ref, (None, None))
         if found_refs is None:
             # Searching for just the name is much faster in remotes like Artifactory
-            found_refs, remote_name = self._search_remotes(search_ref.name, remotes)
-            if found_refs:
-                found_refs = [r for r in found_refs
-                              if r.user == search_ref.user and r.channel == search_ref.channel]
+            found_refs, remote_name = self._search_remotes(search_ref, remotes)
             if found_refs:
                 self._result.append("%s versions found in '%s' remote" % (search_ref, remote_name))
             else:


Test patch:
diff --git a/conans/test/unittests/client/graph/version_ranges_graph_test.py b/conans/test/unittests/client/graph/version_ranges_graph_test.py
--- a/conans/test/unittests/client/graph/version_ranges_graph_test.py
+++ b/conans/test/unittests/client/graph/version_ranges_graph_test.py
@@ -1,4 +1,5 @@
 
+from collections import OrderedDict
 from collections import namedtuple
 
 import six
@@ -9,6 +10,8 @@
 from conans.model.requires import Requirements
 from conans.test.unittests.model.transitive_reqs_test import GraphTest
 from conans.test.utils.conanfile import TestConanFile
+from conans.test.utils.tools import GenConanfile, TurboTestClient, TestServer, \
+    NO_SETTINGS_PACKAGE_ID
 from conans.test.utils.tools import test_processed_profile
 
 
@@ -266,3 +269,23 @@ class Project(ConanFile):
         conanfile = other.conanfile
         self.assertEqual(conanfile.version, "2.0.11549")
         self.assertEqual(conanfile.name, "other")
+
+    def different_user_channel_resolved_correctly_test(self):
+        server1 = TestServer()
+        server2 = TestServer()
+        servers = OrderedDict([("server1", server1), ("server2", server2)])
+
+        client = TurboTestClient(servers=servers)
+        ref1 = ConanFileReference.loads("lib/1.0@conan/stable")
+        ref2 = ConanFileReference.loads("lib/1.0@conan/testing")
+
+        client.create(ref1, conanfile=GenConanfile())
+        client.upload_all(ref1, remote="server1")
+
+        client.create(ref2, conanfile=GenConanfile())
+        client.upload_all(ref2, remote="server2")
+
+        client2 = TurboTestClient(servers=servers)
+        client2.run("install lib/[>=1.0]@conan/testing")
+        self.assertIn("lib/1.0@conan/testing: Retrieving package {} "
+                      "from remote 'server2' ".format(NO_SETTINGS_PACKAGE_ID), client2.out)