Repository: conan-io/conan

Issue:
[question] How to update a reference in a lockfile?
### What is your question?

Hi. I started to use lockfiles and I have my libraries locked inside conan.lock (I used `conan install ... -lockfile-out=conan.lock` to create it and didn't edit it or used `conan lock ...` commands on it). Among them - `qt/5.15.10#3c0bbf763b778e7b7fc6663eea043447` which I want to update. I've added some patches, built the package and now I have a new revision of it (`qt/5.15.10#64e483ec8cfebb74e03da891c848e5a4`) in our Artifactory. I also have a new requirement in my conanfile which I want to lock (let's say `cpython`). What I've tried so far:

1. `conan install . --profile:host ... --profile:build ... -o=&:my_option=my_value --lockfile-partial --lockfile-out=conan.lock --update` returns `ERROR: Requirement 'cpython/...' not in lockfile`. But the documentation clearly states `--lockfile-partial    Do not raise an error if some dependency is not found in lockfile`. Why do I have an error then?

2. On the other hand this doesn't return an error (I moved `--lockfile-partial` to the beginning) and it adds `cpython` to the lockfile, though it doesn't update a reference of a `qt` in a lockfile:
`conan install . --lockfile-partial --profile:host ... --profile:build ... -o=&:my_option=my_value --lockfile-out=conan.lock --update`. Again, according documentation `--update Will install newer versions and/or revisions in the local cache for the given reference, or all in case no argument is supplied.` and I believe that's what I want.

    In the logs I have:

        ```
        ======== Computing dependency graph ========
        Graph root
            conanfile.py: some\path\to\conanfile.py
        Requirements
            qt/5.15.10#3c0bbf763b778e7b7fc6663eea043447 - Cache
        ...
        ======== Computing necessary packages ========
        Requirements
           qt/5.15.10#3c0bbf763b778e7b7fc6663eea043447:fa0e758b93c42da18a2a0ee6f099fb6c0047624d#4e0a88d78229da2fd9a3835c25388f79 - Cache
        ```

3. If I specify that I want to update `qt` it also doesn't update a reference of a `qt` in a lockfile: `conan install . --lockfile-partial --profile:host ... --profile:build ... -o=&:my_option=my_value --lockfile-out=conan.lock --update=qt`.

     Same logs.

So, how do I update the reference of `qt` in the lockfile to the latest available in the remote revision of `qt`?

conan 2.5.0

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Have you seen [`conan lock update`](https://docs.conan.io/2/reference/commands/lock/update.html)? It's available from Conan 2.5.0, and is meant to solve cases like this where it's a bit confusing as to how to update a specific reference :)
> Have you seen [`conan lock update`](https://docs.conan.io/2/reference/commands/lock/update.html)? It's available from Conan 2.5.0, and is meant to solve cases like this where it's a bit confusing as to how to update a specific reference :)

I've seen it. From the docs `The command will replace existing locked references that matches the same package name with the provided argument values`. I do not want to provide anything by hand. I want conan to get latest revision of `qt` from a remote (in my case it is `64e483ec8cfebb74e03da891c848e5a4`), to put it cache and then update the lockfile with the new reference
@memsharded could you please comment on this issue?
> I've seen it. From the docs The command will replace existing locked references that matches the same package name with the provided argument values. I do not want to provide anything by hand. I want conan to get latest revision of qt from a remote (in my case it is 64e483ec8cfebb74e03da891c848e5a4), to put it cache and then update the lockfile with the new reference

The ``lockfiles`` are actually locking things, this is its main purpose. Trying to have both packages locked and updated are contradictory goals.

If you want to get some specific package version "unlocked", then you can remove it from the lockfile with ``conan lock remove``. Then, the next ``conan lock create`` will resolve to the latest one, or if using other command like ``conan install``, then you can provide ``--lockfile-partial`` argument to let it resolve even if it is not in the lockfile.
Any further question here @maksim-petukhov?
I am a bit confused after reading this issue. I was expecting that the purpose of `conan lock update` was so that I can say `take dependency X and update the lock file it to a new version while keeping transitive dependencies consistent" and conan goes and computes a new partical dependency graph taking only that version into account (finding the correct revision if the version is ambiguous) and then produces a new lock file with minimal changes to other packages.

Is that not how it works?

So if I already have a lock file created and I want to make a minimal update to the lock file to bump a single package to a new version (without specifying the exact revision) how would I do that?
Hi @s-geiger-si 

> Is that not how it works?

No, ``conan lock update`` is basically a ``conan lock remove`` + ``conan lock add``. No computation of the graph is done at all, just simple manipulation of the lockfile files.

> So if I already have a lock file created and I want to make a minimal update to the lock file to bump a single package to a new version (without specifying the exact revision) how would I do that?

There are 2 alternatives:

- You apply that lockfile at the moment you are bumping the single package into the new version, and capture the output lockfile. The process of ``conan create`` will automatically put the new version (even with its revision) into the lockfile, no need to call ``conan lock update``. As in general it is desired to apply the lockfile anyway to the "single package" build, this might be a possibility
- You can ``conan lock update`` to update the lockfile without revision, then any other lockfile operation like ``conan lock create`` or ``conan install`` that actually resolve the dependency graph can resolve the revision to the latest and add it to the lockfile.


Thanks for the clarification, it might help to document explicitly which `conan lock` commands only perform local lock file changes and which talk to the remote registry.

Regarding your suggestion, I am not sure how that would help me. I am usually not calling `conan create` in my local workspace. Instead, this happens in the CI/CD pipeline. When I run the build, install, create workflow in the pipeline I want to use the lockfile and ensure that the build is immutable. This currently works fine.

But on my local workspace I want to manage the lifecycle of the lock file without creating the package. I usually only run `conan install` and `conan build` and if that works then I push and let the pipeline do the testing, package creation and upload to the registry if everything checks out.

If I have an updated package version, what I can do is:

```bash
rm conan.lock
conan install conanfile.py --lockfile-out=conan.lock
```

This generates a new lock file using the latest versions in my local conan registry pulling in newer versions of the components that were just built by my CI/CD pipeline. But it has the risk that some unrelated libraries get also updated that I might not want to update that this point in time (usually 3rd party libraries like spdlog or boost).

So what I am really looking for is some command that also me to say: "Update a single package (or a list) to their respective latest versions (within the version range specified in the conanfile.py), make transitive changes if its needed to satisfy any dependencies and leave all other package versions constant."

Is this an uncommon workflow?
Maybe what you are looking for is the ``conan lock remove`` to unlock whatever dependencies you want, followed by a ``conan install`` or ``conan lock create`` command with the ``--lockfile-partial`` argument, that allows to use a lockfile in a way that if some dependency is not lock, it can freely resolve it without raising an error (strict mode which is the default).
You are right, the following is basically what I was looking for:

```bash
conan lock remove --requires libfoo/*
conan lock remove --requires libbar/*
conan lock remove --requires libbaz/*
conan install ./conanfile.py --update --lockfile-partial --lockfile-out=conan.lock --output ./cmake-build-debug
```

Afterwards my three libraries `libfoo`, `libbar` and `libbaz` are all updated to their respective latest versions but everything else remains at the previous version. Important aspects here:

- The `--update` argument here was needed otherwise conan did not resolve newer versions from the registry
- We cannot just write `--require libbaz` but need to use `--require libbaz/*` (or specify the exact version)
- The `--lockfile-partial` argument is needed otherwise there is some difficult to understand error about a required library that does not appear in the lock file.

I wonder if there could be a short-hand for this to make this scenario easier for users that are not conan experts. Something like: `conan update libfoo libbar libbaz` and this would unlock the specified packages and perform an installation of newer versions with the remaining partial lock file using the `--update` argument and which generates a new lock file in place.
I think you can do them in one line:
```
conan lock remove --requires libfoo/* --requires=libbar/* ...
```

> The --update argument here was needed otherwise conan did not resolve newer versions from the registry

Yes, indeed. By default Conan resolves from the Conan cache for performance reasons. It will only check for latest in the servers with ``--update``, good point.

> We cannot just write --require libbaz but need to use --require libbaz/* (or specify the exact version)

Also true, Conan 2 made patterns more explicit everywhere.

> The --lockfile-partial argument is needed otherwise there is some difficult to understand error about a required library that does not appear in the lock file.

For ``conan lock create``, this argument is implicit, it is not needed


> I wonder if there could be a short-hand for this to make this scenario easier for users that are not conan experts. Something like: conan update libfoo libbar libbaz and this would unlock the specified packages and perform an installation of newer versions with the remaining partial lock file using the --update argument and which generates a new lock file in place.

The problem with that is that there are a lot of semantics and inputs missing:
- Which lockfile (need --lockfile and --lockfile-out arguments)
- Which configuration (profiles, settings, options, both for build and host) inputs are needed to build the graph
- The same name of the command is not evident that it is updating a lockfile
- It is making implicit the "root" conanfile of the project, which is always necessary. If it is not in the current folder, it can be quite confusing

This is still a relatively advanced operation, and while the proposal is easier to type, we are talking mostly about:
```
$ conan update libfoo libbar
# vs 
$ conan lock remove --requires libfoo/* --requires=libbar/*
$ conan lock create --update
```

I'll check with the team for considering something like this for inclusion to Conan, but at the moment it seems a bit too minor to be built-in.

Conan has the "custom commands" framework, so organizations can define their own command, if you want to try to implement your own ``conan update``, that would be a good use case.

I'll check with the team for considering something like this for inclusion to Conan, but at the moment it seems a bit too minor to be built-in.





I have discussed with the team, we would like to explore this idea a bit further.

As I see it, there could be 2 different approaches:

- ``conan lock upgrade``. It receives same arguments as ``conan install``, ``<path/to/conanfile>`` or ``--requires=xxx/version`` plus all profile, settings, options, arguments, and ``--lockfile`` arguments defaulted to ``conan.lock``. Then it needs some argument such as ``--update-requires=zlib/*`` to indicate which dependencies have to be updated. I have named this command ``conan lock upgrade`` to differentiate it with the existing ``conan lock update`` that we cannot break
- using ``conan install ....`` with a new argument such as ``--lockfile-update=zlib/*``, or maybe extending and reusing ``--lockfile-partial=zlib/*``. It might also work together with the ``--update=zlib/*`` argument with a boolean ``--lockfile-update`` that leverages the ``--update`` values.

There are pros and cons to both approaches:
- The first one needs users to be aware, learn and use a new command, and understand what it does. But it is a bit more explicit about the intention, and can define ``--lockfile`` default values.
- The second one is more natural to most users, a regular ``conan install``, specifying by arguments the things to be "unlocked" and updated. But the intention is not that evident. It also does a full installation of the binary, so most likely need to be implemented too in the ``conan graph info`` command



Patch:
diff --git a/conan/cli/commands/lock.py b/conan/cli/commands/lock.py
index 35fe47e6f05..55c48e9e101 100644
--- a/conan/cli/commands/lock.py
+++ b/conan/cli/commands/lock.py
@@ -1,3 +1,4 @@
+from collections import defaultdict
 import os
 
 from conan.api.output import ConanOutput
@@ -6,6 +7,7 @@
 from conan.cli import make_abs_path
 from conan.cli.args import common_graph_args, validate_common_graph_args
 from conan.cli.printers.graph import print_graph_packages, print_graph_basic
+from conan.errors import ConanException
 from conan.api.model import RecipeReference
 
 
@@ -172,3 +174,63 @@ def lock_update(conan_api, parser, subparser, *args):
     lockfile.update(requires=args.requires, build_requires=args.build_requires,
                     python_requires=args.python_requires, config_requires=args.config_requires)
     conan_api.lockfile.save_lockfile(lockfile, args.lockfile_out)
+
+
+
+
+@conan_subcommand()
+def lock_upgrade(conan_api, parser, subparser, *args):
+    """
+    (Experimental) Upgrade requires, build-requires or python-requires from an existing lockfile given a conanfile
+    or a reference.
+    """
+
+    common_graph_args(subparser)
+    subparser.add_argument('--update-requires', action="append", help='Update requires from lockfile')
+    subparser.add_argument('--update-build-requires', action="append", help='Update build-requires from lockfile')
+    subparser.add_argument('--update-python-requires', action="append", help='Update python-requires from lockfile')
+    subparser.add_argument('--update-config-requires', action="append", help='Update config-requires from lockfile')
+    subparser.add_argument('--build-require', action='store_true', default=False, help='Whether the provided reference is a build-require')
+    args = parser.parse_args(*args)
+
+    # parameter validation
+    validate_common_graph_args(args)
+
+    if not any([args.update_requires, args.update_build_requires, args.update_python_requires, args.update_config_requires]):
+        raise ConanException("At least one of --update-requires, --update-build-requires, "
+                             "--update-python-requires or --update-config-requires should be specified")
+
+    cwd = os.getcwd()
+    path = conan_api.local.get_conanfile_path(args.path, cwd, py=None) if args.path else None
+    remotes = conan_api.remotes.list(args.remote) if not args.no_remote else []
+    overrides = eval(args.lockfile_overrides) if args.lockfile_overrides else None
+    lockfile = conan_api.lockfile.get_lockfile(lockfile=args.lockfile, conanfile_path=path,
+                                               cwd=cwd, partial=True, overrides=overrides)
+    profile_host, profile_build = conan_api.profiles.get_profiles_from_args(args)
+
+    # Remove the lockfile entries that will be updated
+    lockfile = conan_api.lockfile.remove_lockfile(lockfile,
+                                                  requires=args.update_requires,
+                                                  python_requires=args.update_python_requires,
+                                                  build_requires=args.update_build_requires,
+                                                  config_requires=args.update_config_requires)
+    # Resolve new graph
+    if path:
+        graph = conan_api.graph.load_graph_consumer(path, args.name, args.version,
+                                                    args.user, args.channel,
+                                                    profile_host, profile_build, lockfile,
+                                                    remotes, args.update,
+                                                    is_build_require=args.build_require)
+    else:
+        graph = conan_api.graph.load_graph_requires(args.requires, args.tool_requires,
+                                                    profile_host, profile_build, lockfile,
+                                                    remotes, args.update)
+    print_graph_basic(graph)
+    graph.report_graph_error()
+    conan_api.graph.analyze_binaries(graph, args.build, remotes=remotes, update=args.update,
+                                     lockfile=lockfile)
+    print_graph_packages(graph)
+
+    lockfile = conan_api.lockfile.update_lockfile(lockfile, graph, args.lockfile_packages,
+                                                  clean=args.lockfile_clean)
+    conan_api.lockfile.save_lockfile(lockfile, args.lockfile_out or "conan.lock")


Test patch:
diff --git a/test/integration/lockfile/test_user_overrides.py b/test/integration/lockfile/test_user_overrides.py
index e3e033969a3..f41a3a979a1 100644
--- a/test/integration/lockfile/test_user_overrides.py
+++ b/test/integration/lockfile/test_user_overrides.py
@@ -4,7 +4,7 @@
 import pytest
 
 from conan.test.assets.genconanfile import GenConanfile
-from conan.test.utils.tools import TestClient
+from conan.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient
 
 
 def test_user_overrides():
@@ -360,3 +360,78 @@ def test_lock_update(self, kind, old, new):
         lock = c.load("conan.lock")
         assert old not in lock
         assert new in lock
+
+
+
+class TestLockUpgrade:
+    @pytest.mark.parametrize("kind, pkg, old, new", [
+        ("requires", "math", "math/1.0", "math/1.1"),
+        ("build-requires", "cmake", "cmake/1.0", "cmake/1.1"),     # TODO there is not a --build-requires
+        # ("python-requires", "mytool", "mytool/1.0", "mytool/1.1"), # TODO nor a --python-requires
+    ])
+    def test_lock_upgrade(self, kind, pkg, old, new):
+        c = TestClient(light=True)
+        c.save({f"{pkg}/conanfile.py": GenConanfile(pkg)})
+
+        c.run(f"export {pkg} --version=1.0")
+        rev0 = c.exported_recipe_revision()
+        kind_create = "tool-requires" if "build-requires" == kind else kind
+        c.run(f"lock create --{kind_create}={pkg}/[*]")
+        lock = c.load("conan.lock")
+        assert f"{old}#{rev0}" in lock
+
+        c.run(f"export {pkg} --version=1.1")
+        rev1 = c.exported_recipe_revision()
+        c.run(f"lock upgrade --{kind_create}={pkg}/[*] --update-{kind}={pkg}/[*]")
+        lock = c.load("conan.lock")
+        print(lock)
+        assert f"{old}#{rev0}" not in lock
+        assert f"{new}#{rev1}" in lock
+
+
+    def test_lock_upgrade_path(self):
+        c = TestClient(light=True)
+        c.save({"liba/conanfile.py": GenConanfile("liba"),
+                "libb/conanfile.py": GenConanfile("libb"),
+                "libc/conanfile.py": GenConanfile("libc"),
+                "libd/conanfile.py": GenConanfile("libd")})
+        c.run(f"export liba --version=1.0")
+        c.run(f"export libb --version=1.0")
+        c.run(f"export libc --version=1.0")
+        c.run(f"export libd --version=1.0")
+        c.save(
+            {
+                f"conanfile.py": GenConanfile()
+                .with_requires(f"liba/[>=1.0 <2]")
+                .with_requires("libb/[<1.2]")
+                .with_tool_requires("libc/[>=1.0]")
+                .with_python_requires("libd/[>=1.0 <1.2]")
+            }
+        )
+
+        c.run("lock create .")
+        lock = c.load("conan.lock")
+        assert "liba/1.0" in lock
+        assert "libb/1.0" in lock
+        assert "libc/1.0" in lock
+        assert "libd/1.0" in lock
+
+        # Check versions are updated accordingly
+        c.run(f"export liba --version=1.9")
+        c.run(f"export libb --version=1.1")
+        c.run(f"export libb --version=1.2")
+        c.run(f"export libc --version=1.1")
+        c.run(f"export libd --version=1.1")
+        c.run("lock upgrade . --update-requires=liba/1.0 --update-requires=libb/[*] --update-build-requires=libc/[*] --update-python-requires=libd/1.0")
+        lock = c.load("conan.lock")
+        assert "liba/1.9" in lock
+        assert "libb/1.1" in lock
+        assert "libc/1.1" in lock
+        assert "libd/1.1" in lock
+
+        # Check version conanfile version range is respected
+        c.run(f"export libd --version=1.2")
+        c.run("lock upgrade . --update-python-requires=libd/*")
+        lock = c.load("conan.lock")
+        assert "libd/1.1" in lock
+        assert "libd/1.2" not in lock