Repository: conan-io/conan
Pull request: Feature/lockfiles partial
Changelog: Fix: Removed check in lockfiles computed from other lockfile that it should be part of it. Users can check the resulting lockfile themselves if they want to.
Docs: https://github.com/conan-io/docs/pull/1868

Fix https://github.com/conan-io/conan/issues/7739

Problem statement:
[bug][question] Conan lockfile behavior change between conan 1.26 and conan 1.29. 
We used conan.lock files from a CI build for a big set of comonents where all component versions were synchronized. Later developers used these lockfiles to build new versions of one of those components in the context of that lockfile locally, so they were not affected by new versions of upstreams that some other developer might have created in the meantime.

Simple example:
CA/1.0.0 <- CB/1.0.0  (read CB uses CA)

This combination was built in a CI job and the lockfile was generated. Then someone created a CA/2.0.0 package.
Now another developer works on a CB/2.0.0 package, but still wants to work in the context of the last successfull CI build, so he uses the lockfile and creates a CB/2.0.0 locally on his machine, but still wants to use CA/1.0.0 (which was locked in the CI lockfile).

This scenario used to work perfectly with conan 1.26, but fails with conan 1.29 with the error message: 
`ERROR: Couldn't find 'CB/2.0.0@local/testing' in lockfile`.

We need an alternative for the old behavior with the new conan version, but are at a loss how to do that, as lockfiles have become quite restrictive about building new versions of packages that were not previously locked.

I have attached a simple python script, that reproduces the case explained above. The script works with conan 1.26, but fails with the above error message with conan 1.29.

(Of course, our real word scenario involves a complete tree of sometimes dozens of upstream dependencies with potentially several version conflicts, not just one upstream package. That is why we need a way to fix the local environment to a known-good state until we push the new version of CB to the CI which then builds a new consistent state of all components.)

Hints from issue discussion:
Here ist the attachment:
[use_segment_build_lockfile_for_local_component_builds.py.gz](https://github.com/conan-io/conan/files/5262083/use_segment_build_lockfile_for_local_component_builds.py.gz)

Hi @fourbft 

The lockfiles have become not more restrictive, but their checks more thorough and complete.
It is true, that now, if you capture a lockfile, and it locks ``CB/1.0.0``, you will not be able to change that, because it is locked. This was a bug fix, because if the lockfile contained other packages that depended on ``CB/1.0.0``, changing it will be automatically violating the locked version.

The new lockfiles should allow this flow, but in a more controlled way. I have hit a limitation of the commands to allow it, but the logic is almost there. I will be providing a pull request with this flow implemented and update here.
> I will be providing a pull request with this flow implemented and update here.

Thanks @memsharded 
Apart from the workflow described above we also have implemented the workflow described in the former conan docs here:
https://docs.conan.io/en/1.26/versioning/lockfiles.html
in the chapter "How to use lockfiles in CI". Especially this step:

> Now we can safely create the new version of pkga/0.2, that will resolve to use pkgz/0.1 instead of the latest 0.2, if we use the lockfile:
> cd pkga && conan create . pkga/0.2@user/testing --lockfile=../release
> \# lockfile in release/conan.lock is modified to contain pkga/0.2

This workflow is different from the one above in that it modifies the lockfile during the "conan create". It would be great, if such a workflow would be possible again, too.

Commit message: removing overlap/contained check

Patch to review:
diff --git a/conans/client/conan_api.py b/conans/client/conan_api.py
index ca387242949..e4ac5a514d1 100644
--- a/conans/client/conan_api.py
+++ b/conans/client/conan_api.py
@@ -1317,6 +1317,7 @@ def lock_create(self, path, lockfile_out,
         if path and reference:
             raise ConanException("Both path and reference arguments were provided. Please provide "
                                  "only one of them")
+
         if path:
             ref_or_path = _make_abs_path(path, cwd)
             if not os.path.isfile(ref_or_path):
@@ -1362,8 +1363,6 @@ def lock_create(self, path, lockfile_out,
         graph_lock_file = GraphLockFile(phost, pbuild, graph_lock)
         if lockfile:
             new_graph_lock = GraphLock(deps_graph, self.app.config.revisions_enabled)
-            # check if the lockfile provided was used or not
-            new_graph_lock.check_contained(graph_lock)
             graph_lock_file = GraphLockFile(phost, pbuild, new_graph_lock)
         if base:
             graph_lock_file.only_recipes()
diff --git a/conans/model/graph_lock.py b/conans/model/graph_lock.py
index e303b26f384..b30a126cad8 100644
--- a/conans/model/graph_lock.py
+++ b/conans/model/graph_lock.py
@@ -474,15 +474,6 @@ def update_lock(self, new_lock):
             if current.prev is None:
                 current.prev = node.prev
 
-    def check_contained(self, other):
-        """ if lock create is provided a lockfile, it should be used, and it should contain it
-        otherwise, it was useless to pass it, and it is dangerous to continue, recommended to
-        create a fresh lockfile"""
-        other_root_id = other.root_node_id()
-        if other_root_id not in self._nodes:
-            raise ConanException("The provided lockfile was not used, there is no overlap. You "
-                                 "might want to create a fresh lockfile")
-
     def pre_lock_node(self, node):
         if node.recipe == RECIPE_VIRTUAL:
             return
diff --git a/conans/test/functional/graph_lock/dynamic_test.py b/conans/test/functional/graph_lock/dynamic_test.py
index 286ba4d9e32..e41d815aaae 100644
--- a/conans/test/functional/graph_lock/dynamic_test.py
+++ b/conans/test/functional/graph_lock/dynamic_test.py
@@ -128,8 +128,10 @@ def partial_lock_root_unused_test(self):
         self.assertIn("Couldn't find 'LibC/1.0' in lockfile", client.out)
 
         client.run("lock create conanfile.py --name=LibC --version=1.0 --lockfile=libb.lock "
-                   "--lockfile-out=libc.lock", assert_error=True)
-        self.assertIn("ERROR: The provided lockfile was not used, there is no overlap.", client.out)
+                   "--lockfile-out=libc.lock")
+        # Users can validate themselves if relevant package is in the lockfile or not
+        libc_lock = client.load("libc.lock")
+        self.assertNotIn("LibB/1.0", libc_lock)
 
     def remove_dep_test(self):
         client = TestClient()
@@ -254,3 +256,35 @@ def augment_test_package_requires(self):
         else:
             self.assertEqual(dep["ref"], "dep/0.1")
             self.assertEqual(dep["prev"], "0")
+
+    def partial_intermediate_package_lock_test(self):
+        client = TestClient()
+        client.save({"conanfile.py": GenConanfile()})
+        client.run("create . LibA/1.0@")
+        client.save({"conanfile.py": GenConanfile().with_require("LibA/[>=1.0]")})
+        client.run("create . LibB/1.0@")
+        client.save({"conanfile.py": GenConanfile().with_require("LibB/[>=1.0]")})
+        client.run("create . LibC/1.0@")
+        client.run("lock create --reference=LibC/1.0 --lockfile-out=libc.lock")
+
+        # New version of LibA/1.0.1, that should never be used
+        client.save({"conanfile.py": GenConanfile()})
+        client.run("create . LibA/1.0.1@")
+
+        # Go back to B, we want to develop but keep depending on LibA/1.0.0
+        client.save({"conanfile.py": GenConanfile().with_require("LibA/[>=1.0]")})
+        client.run("create . LibB/1.1@ --lockfile=libc.lock", assert_error=True)
+        self.assertIn("Couldn't find 'LibB/1.1' in lockfile", client.out)
+
+        client.run("lock create conanfile.py --name=LibB --version=1.1 --lockfile=libc.lock "
+                   "--lockfile-out=libb.lock")
+        self.assertIn("LibA/1.0 from local cache", client.out)
+        self.assertNotIn("LibA/1.0.1", client.out)
+        libb_lock = client.load("libb.lock")
+        self.assertIn("LibA/1.0", libb_lock)
+        self.assertNotIn("LibA/1.0.1", libb_lock)
+
+        client.run("create . LibB/1.1@")
+        self.assertIn("LibA/1.0.1 from local cache - Cache", client.out)
+        client.run("create . LibB/1.1@ --lockfile=libb.lock")
+        self.assertIn("LibA/1.0 from local cache - Cache", client.out)
diff --git a/conans/test/functional/graph_lock/graph_lock_ci_test.py b/conans/test/functional/graph_lock/graph_lock_ci_test.py
index 84ea55dbe8c..4f0617cbd06 100644
--- a/conans/test/functional/graph_lock/graph_lock_ci_test.py
+++ b/conans/test/functional/graph_lock/graph_lock_ci_test.py
@@ -526,11 +526,13 @@ def test_version_ranges_partial_unused(self):
         self.assertNotIn("pyreq/0.2", client.out)
 
         # Go back to main orchestrator
-        # This should fail, as PkgB/0.2 is not involved in the new resolution
+        # This should fail, as PkgB/1.0 is not involved in the new resolution
         client.run("lock create --reference=PkgD/0.1@user/channel "
-                   "--lockfile=buildb.lock --lockfile-out=conan.lock", assert_error=True)
-        self.assertIn("ERROR: The provided lockfile was not used, there is no overlap",
-                      client.out)
+                   "--lockfile=buildb.lock --lockfile-out=error.lock")
+        # User can perfectly go and check the resulting lockfile and check if PkgB/0.1 is there
+        # We can probably help automate this with a "conan lock find" subcommand
+        error_lock = client.load("error.lock")
+        self.assertNotIn("PkgB/1.0@user/channel", error_lock)
 
         client.run("lock build-order conan.lock --json=build_order.json")
         json_file = client.load("build_order.json")


Code review comments:
diff_hunk: @@ -254,3 +256,35 @@ def augment_test_package_requires(self):
         else:
             self.assertEqual(dep["ref"], "dep/0.1")
             self.assertEqual(dep["prev"], "0")
+
+    def partial_intermediate_package_lock_test(self):
+        client = TestClient()
+        client.save({"conanfile.py": GenConanfile()})
+        client.run("create . LibA/1.0@")
+        client.save({"conanfile.py": GenConanfile().with_require("LibA/[>=1.0]")})
+        client.run("create . LibB/1.0@")
+        client.save({"conanfile.py": GenConanfile().with_require("LibB/[>=1.0]")})
+        client.run("create . LibC/1.0@")
+        client.run("lock create --reference=LibC/1.0 --lockfile-out=libc.lock")
+
+        # New version of LibA/1.0.1, that should never be used
+        client.save({"conanfile.py": GenConanfile()})
+        client.run("create . LibA/1.0.1@")
+
+        # Go back to B, we want to develop but keep depending on LibA/1.0.0
path: conans/test/functional/graph_lock/dynamic_test.py
text: @user1:
```suggestion
        # Go back to B, we want to develop but keep depending on LibA/1.0
```