Repository: conan-io/conan

Issue:
[question] Resolving missing binaries problem if python-require package changed in one of dependencies
### What is your question?

Let's assume, we have `MyApplication` with the following requirements:

```
    def requirements(self):
        self.requires("some_package/1.2.3")
        self.requires("pkg_b/1.2.3")
        self.requires("pkg_c/1.2.3")
```
`pkg_b` itself:
```
    def requirements(self):
        self.requires("pkg_c/[>=1.0.0 <2.0.0]")
```
`pkg_c` itself:
```
    def requirements(self):
        self.requires("some_package/[>=1.0.0 <2.0.0]")
```
Everything works here, and now `some_package/1.2.4` is introdued with no changes, but with new minor version of `python-require`.
Now, `MyApplication` suddenly is broken, because `pkg_b` binaries are now `Missing`, according to conan (of course, in the release day :) ). It's now required to rebuild the whole depndency tree.

As far as I can see, this is expected behavior, because of change of in `python-require` package. Is that right?


I actually have 2 questions here:
1. If `MyApplication` explicitly requests the older version of `some_package`, why rebuild of `pkg_b` and `pkg_c` is required in this case? We are not using newer `some_package` here, so I would expect no rebuild request here. Instead, with no changes in conanfile, and with explicitly set valid and compatible dependencies, we still have our whole dependency tree broken.

2. How to change default packge ID mode for `python-require`?
We have dozens of packages now in our private repository and changing global config would break all our recipes and applications, which relies on conan (all of them use `python-require`). 
So, I'm looking for easier way to pull this off, maybe to change packages one by one with using conanfile attiribute, but there's no `package_id_python_mode`, as, for example `package_id_embed_mode`

UPD:
3. I'm thinking of changing default package ID mode, as documented. Is there any way to do it without breaking all existing recipes?

Thank you in advance.

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Hi @apyvovar 

Thanks for your question

> If MyPackage explicitly requests the older version of some_package, why rebuild of pkg_b and pkg_c is required in this case? We are not using newer some_package here, so I would expect no rebuild request here. Instead, with no changes in conanfile, and with explicitly set valid and compatible dependencies, we still have our whole dependency tree broken.

It is not very clear what is ``MyPackage``. You have defined ``MyApplication`` and ``some_package``, but not ``MyPackage``.

> Everything works here, and now some_package/1.2.4 is introdued with no changes, but with new minor version of python-require.

It is not clear how is it introduced. Changing the ``requires`` in ``MyApplication``? Otherwise, just creating it doens't seem that it will result in resolving to 1.2.4, because it is defining a pinned dependency to 1.2.3. It is also not clear if this is the only package that contains a python-require, it would be helpful to have the definition of ``some_package`` too.



> 3. I'm thinking of changing default package ID mode, as documented. Is there any way to do it without breaking all existing recipes?

Changing the package-id-modes do not break any recipe. It will only result in new computed ``package_ids`` and it will require building from source those new binaries, but that shouldn't require any change in recipes, so recipes shouldn't be broken for this change.

> How to change default packge ID mode for python-require?

It is not clear why you are trying to change it or what problem you are trying to solve. The default mode is pretty useful, allowing full control over when things have to be built:
- increasing the patch means binaries don't need to be rebuilt
- increasing the minor is done when we want to force the binaries to be rebuilt
- increasing the major is done when we are doing breaking API changes, so the major will typically be discarded, because the range will be like ``python_requires = "mypyrequire/[>=1.0 <2]"``

So increasing the minor effectively means "I want the packages using this python-require to rebuild their binaries". You can change the python-require-id-mode, but that basically means shifting the above logic left or right in the version digits, but still the same logic.

> If MyPackage explicitly requests the older version of some_package, why rebuild of pkg_b and pkg_c is required in this case? 

We need to check this. If the packages are static-libraries requiring other static libraries, they won't require re-building unless the minor version of the dependencies change. Having them changing only their python-requires should not be requiring the consumers to re-build (only applications, but not static libraries). So a full reproducible case would be necessary to validate this.
> It is not very clear what is MyPackage. You have defined MyApplication and some_package, but not MyPackage.

Oh, sorry. It's the same. Basically, it means "root package consumer". I've updated the original message.

> It is not clear how is it introduced.

What I meant here is new version of `some_package` was created. Something like this:
| Before  | After |
| ------------- | ------------- |
| name = "some_package"  | name = "some_package" |
| version = "1.2.3"  | version = "1.2.4" |
| python_requires = "conan_utils/2.1.0"  | python_requires = "conan_utils/2.2.0" |

So, `some_package` was rebuilt with updated `conan_utils` python_requires package, minor version of which was bumped.

> It is not clear why you are trying to change it or what problem you are trying to solve

After `some_package/1.2.4` is released, dependencies of `MyApplication` binaries of `pkg_b` and `pkg_c` became `Missing` and conan fails with error. That's a big problem, because we don't uploading sources to remote with packages, so each and every dependency has to be rebuilt and uploaded to Artifactory one by one.

I want to emphasize here, that `MyApplication` explicitly set `some_package/1.2.3` - old version, which worked before. I just simply don't understand in what way `some_package/1.2.4` is somehow related here.

> increasing the minor is done when we want to force the binaries to be rebuilt

No, we don't want to force the binaries to be rebuilt. Increasing the minor version is "add functionality in a backward compatible manner", according to semver. When we add functionality to python script, we don't need anything to rebuilt.

> So a full reproducible case would be necessary to validate this.

Ok, I'll try to create reproducible example
> No, we don't want to force the binaries to be rebuilt. Increasing the minor version is "add functionality in a backward compatible manner", according to semver. When we add functionality to python script, we don't need anything to rebuilt.

Semver is more or less ok for API-compatibility. But Conan (or C++ in general) also needs to model ABI, and the need of re-building packages even when the API is fully compatible. There are multiple cases of this, like when modifying headers that contains templates or structs definitions that can change memory layout, or when a static library is linked by a shared library.

So it is necessary to define the different cases:
- Breaking API => Major
- Not breaking API but need to rebuild binaries => Minor
- Not breaking API, no need to rebuild binaries => Patch

There are cases that the API doesn't change at all, and still binaries must be rebuild, like the shared->static case. Conan will automatically manage this, even if the version is not bumped, as long as ``package_type`` is correctly defined.

In short, semver is not optimal for C/C++ build and compilation model, better take the above considerations in mind when defining the versioning scheme
> After some_package/1.2.4 is released, dependencies of MyApplication binaries of pkg_b and pkg_c became Missing and conan fails with error. 

This should only happen if ``pkg_b`` and ``pkg_c`` are shared libraries linking a static or header-only library or if ``some-package`` is a header-only library, and that is totally expected and desired behavior, the risk is too high, things will be mosly broken if things are not rebuild. If all are static libraries, then it shouldn't happen, as re-building is not necessary.
> This should only happen if pkg_b and pkg_c are shared libraries linking a static or header-only library or if some-package is a header-only library, and that is totally expected and desired behavior, the risk is too high, things will be mosly broken if things are not rebuild. If all are static libraries, then it shouldn't happen, as re-building is not necessary.

That is understandable in case, I'm actually using version 1.2.4. But I am not. In my case, all dependencies - `some_package`, `pkg_b` and `pkg_c` - are requested explicitly in `MyApplication` with exact versions. This means the binaries and dependency graph must remain absolutely the same regaredless of versions of other released depedencies on Artifactory. 

Maybe I don't fully understand how it meant to work, but in our development flow the release of new package doesn't mean, that all packages and applications in organisation should immediately use it. Changes have to be tested first and applied only if everything works as expected

I'm sorry, but I still don't understand why rebuild is required here. 
Let me also rephrase my other question. If I want to set package ID mode for some package (not default for all in config), I have only the following options:
```
package_id_embed_mode = ...
package_id_non_embed_mode = ...
package_id_unknown_mode = ...
```
What should I do if I want to change package ID mode for `python_require` package in the same manner? 
Looks like, the only possibility is via config file, but I don't want change default for all.
What I'd like to see is possibility to do something like this in conanfile attribute:
```
package_id_python_mode = ...
```
Exactly like the entry in config file, but for single package.
Is this possible?
> That is understandable in case, I'm actually using version 1.2.4. But I am not. In my case, all dependencies - some_package, pkg_b and pkg_c - are requested explicitly in MyApplication with exact versions. This means the binaries and dependency graph must remain absolutely the same regaredless of versions of other released depedencies on Artifactory.

There are different aspects here. Depending on a fixed version doesn't guarantee anything if that version for example gets a new revision or a new binary. For example if ``pkg_b`` and ``pkg_c`` are static libraries, even if they are the same version, if the are re-built from source, because they depend on a transitive dependency, lets say a new header-only new version, both ``pkg_b`` and ``pkg_c`` will be different binaries with different functionality that has been compiled from the headers of the transitive dependency. And if that is the case, of course the final consumer application will have to be re-built, to link both ``pkg_b`` and ``pkg_c`` new binaries, of course, otherwise the application will not integrate the changes done in the upstream code.

In short, when dependencies change, in many situations it is necessary to re-build the consumers binaries.
In your described situation, there is not enough information to know why it is happening, because there are many other factors that are important, settings, options, package-types, configuration, etc. If we want to understand what is happening we would need a complete reproducible case, that we can run and check, otherwise it is nearly impossible to diagnose.

> Maybe I don't fully understand how it meant to work, but in our development flow the release of new package doesn't mean, that all packages and applications in organisation should immediately use it. Changes have to be tested first and applied only if everything works as expected

The way new changes are integrated in the downstream is defined by the versioning scheme. You can have version ranges that exclude certain versions, if you are releasing a new version, in the same server repository together with the other packages that contain ``requires`` with version-ranges that include the new version, then the new version will automatically be used, this is how dependency resolution works. You can follow different strategies:

- Publish a new version to a secondary server repository. From there run the tests, build necessary packages of consumers, etc, and only copy the new package version and new binaries to the main server repository when everything is working and approved. This is a very recommended CI flow
- Make sure that the new version is outside of the defined version ranges. You can use only exact dependencies and not version-ranges, that way, when a new version is published it will not be automatically used. Then, when packages want to use the newer version, change their requirements and publish new versions of those too. This can be a bit tedious and require a lot of manual work, but is "exact" with versions
- Use lockfiles to guarantee you are using a fixed set of versions irrespective of the versions that have been published to the server.

Ok, thank you for all the info, I'll try to adjust my development flow to better fit this logic.

And, please, take a look at this https://github.com/conan-io/conan/issues/15327#issuecomment-1867146009 , it would be helpful for some of my python-require packages



Patch:
diff --git a/conans/client/graph/compute_pid.py b/conans/client/graph/compute_pid.py
index 0948a03fe3a..a81497bb002 100644
--- a/conans/client/graph/compute_pid.py
+++ b/conans/client/graph/compute_pid.py
@@ -21,7 +21,7 @@ def compute_package_id(node, new_config):
 
     python_requires = getattr(conanfile, "python_requires", None)
     if python_requires:
-        python_requires = python_requires.all_refs()
+        python_requires = python_requires.info_requires()
 
     data = OrderedDict()
     build_data = OrderedDict()
diff --git a/conans/client/graph/python_requires.py b/conans/client/graph/python_requires.py
index dc24ff7fb51..a861e53df44 100644
--- a/conans/client/graph/python_requires.py
+++ b/conans/client/graph/python_requires.py
@@ -31,6 +31,10 @@ def serialize(self):
     def all_refs(self):
         return [r.ref for r in self._pyrequires.values()]
 
+    def info_requires(self):
+        return {pyreq.ref: getattr(pyreq.conanfile, "package_id_python_mode", None)
+                for pyreq in self._pyrequires.values()}
+
     def items(self):
         return self._pyrequires.items()
 
diff --git a/conans/model/info.py b/conans/model/info.py
index 00743eb3884..a2b9ee85387 100644
--- a/conans/model/info.py
+++ b/conans/model/info.py
@@ -66,6 +66,7 @@ def __init__(self, ref, package_id, default_package_id_mode):
         self._package_id = package_id
         self.name = self.version = self.user = self.channel = self.package_id = None
         self.recipe_revision = None
+        self.package_id_mode = default_package_id_mode
 
         try:
             func_package_id_mode = getattr(self, default_package_id_mode)
@@ -234,14 +235,15 @@ class PythonRequiresInfo:
     def __init__(self, refs, default_package_id_mode):
         self._default_package_id_mode = default_package_id_mode
         if refs:
-            self._refs = [RequirementInfo(r, None, default_package_id_mode=default_package_id_mode)
-                          for r in sorted(refs)]
+            self._refs = [RequirementInfo(r, None,
+                                          default_package_id_mode=mode or default_package_id_mode)
+                          for r, mode in sorted(refs.items())]
         else:
             self._refs = None
 
     def copy(self):
         # For build_id() implementation
-        refs = [r._ref for r in self._refs] if self._refs else None
+        refs = {r._ref: r.package_id_mode for r in self._refs} if self._refs else None
         return PythonRequiresInfo(refs, self._default_package_id_mode)
 
     def __bool__(self):


Test patch:
diff --git a/conans/test/integration/package_id/package_id_modes_test.py b/conans/test/integration/package_id/package_id_modes_test.py
index 16b01a1284b..f8555058658 100644
--- a/conans/test/integration/package_id/package_id_modes_test.py
+++ b/conans/test/integration/package_id/package_id_modes_test.py
@@ -104,3 +104,37 @@ class Dep(ConanFile):
         c.run("create dep --version=0.2")
         c.run("create pkg")
         c.assert_listed_binary({"pkg/0.1": ("56934f87c11792e356423e081c7cd490f3c1fbe0", "Build")})
+
+    def test_dep_python_require_defined(self):
+        c = TestClient()
+        dep = textwrap.dedent("""
+            from conan import ConanFile
+            class Dep(ConanFile):
+                name = "dep"
+                package_type = "python-require"
+                package_id_python_mode = "major_mode"
+            """)
+        c.save({"dep/conanfile.py": dep,
+                "pkg/conanfile.py": GenConanfile("pkg", "0.1").with_python_requires("dep/[*]")})
+        c.run("create dep --version=0.1")
+        c.run("create pkg")
+        c.assert_listed_binary({"pkg/0.1": ("331c17383dcdf37f79bc2b86fa55ac56afdc6fec", "Build")})
+
+        # using dep 0.2, still same, because dependency chose "major"
+        c.run("create dep --version=0.1.1")
+        c.run("create pkg")
+        c.assert_listed_binary({"pkg/0.1": ("331c17383dcdf37f79bc2b86fa55ac56afdc6fec", "Build")})
+
+        # using dep 0.2, still same, because dependency chose "major"
+        c.run("create dep --version=0.2")
+        c.run("create pkg")
+        c.assert_listed_binary({"pkg/0.1": ("331c17383dcdf37f79bc2b86fa55ac56afdc6fec", "Build")})
+        c.run("list *:*")
+        assert "dep/0.Y.Z" in c.out
+
+        # using dep 0.2, new package_id, because dependency chose "major"
+        c.run("create dep --version=1.0")
+        c.run("create pkg")
+        c.assert_listed_binary({"pkg/0.1": ("9b015e30b768df0217ffa2c270f60227c998e609", "Build")})
+        c.run("list *:*")
+        assert "dep/1.Y.Z" in c.out