Repository: conda/conda

Issue:
Make pinned packages optional/constrained dependencies
In the conda 4.3 feature branch, make the `get_pinned_specs()` function in `conda/plan.py` return `MatchSpec` objects with the `optional=True` flag.

Make sure (by writing extra unit tests) that the `conda.cli.common.spec_from_line()` function is both working correctly and being used correctly.

Package pinning configuration was recently implemented within condarc files via https://github.com/conda/conda/pull/4921/files.  Reviewing that PR would be a good place to start.
Make pinned packages optional/constrained dependencies
In the conda 4.3 feature branch, make the `get_pinned_specs()` function in `conda/plan.py` return `MatchSpec` objects with the `optional=True` flag.

Make sure (by writing extra unit tests) that the `conda.cli.common.spec_from_line()` function is both working correctly and being used correctly.

Package pinning configuration was recently implemented within condarc files via https://github.com/conda/conda/pull/4921/files.  Reviewing that PR would be a good place to start.


Discussion:
While working on the test I notice something isn't working quite right, I suspect the solver:

```
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
run_command(Commands.INSTALL, prefix, "flask")
assert package_is_installed(prefix, "python-3.6.1-2")
```
works fine, while
```
run_command(Commands.CONFIG, prefix,
                        "--add pinned_packages", "python=3.6.1-2")
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
run_command(Commands.INSTALL, prefix, "flask")
assert package_is_installed(prefix, "python-3.6.1-2")
```
yields the error
```
UnsatisfiableError: The following specifications were found to be in conflict:
  - flask
  - python 3.6.1-2 (optional)
```
@mcg1969 , do you have any idea why this may be happening, and where I should follow up (file a separate bug report, etc.) ?
Interesting. Normally I tell people _never doubt the solver_ but this is certainly sufficiently new functionality that we should check. I'll see what I can find.
ha ha! _never doubt the solver_.

`python=3.6.1-2` is not the correct spec. It's either `python=3.6.1=2` or `python 3.6.1 2`, depending on how you've implemented it. `python=3.6.1-2` means `Python version 3.6.1-2`, not `Python version 3.6.1 build 2`.
Thanks ! (Just a shot in the dark: the solver itself only treats a bunch of numbers (roughly speaking :-) ), which we generate from all the various inputs (match-spec, index content, etc.). I could imagine that our encoding of the request from those inputs into numbers might be wrong somehow, i.e. the query we submit to the solver proper may not be what we intended.)
Oh ! Now that isn't very intuitive, is it ? Is that documented somewhere ?
Anyhow, thanks for the quick find, I'll adjust my test logic and see whether it works.
Well, `python=3.6.1=2` is the proper command-line syntax, and that's documented. `python 3.6.1 2` is the proper MatchSpec syntax, which is also documented, but perhaps internally.
OK, fair enough. Thanks for your assistance !
```
run_command(Commands.CONFIG, prefix,
            "--add pinned_packages", "python=3.6.1=2")
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
```
is failing, i.e. python is being installed even though openssl doesn't depend on it. The MatchSpec objects look good, though.
@stefanseefeld 

> is failing, i.e. python is being installed even though openssl doesn't depend on it

Are you working in 4.3.x or 4.4.x?
Right, I understand.
This is on 4.4.x.
I guess that means then that the `optional=True` flag sent to the MatchSpec object isn't working correctly within resolve?
Is this on windows? If so, features are to blame, and we're discussing that in another issue.
This is GNU/Linux.
Ah, I'll bet I know why. You're passing an optional MatchSpec right to `Resolve.solve`, I'll bet.

That's not going to work, I don't think. The reason why is that I've been using the optional specification for a long time to effect package _removals_. When you do a `conda remove xxx`, the solver gets a set of specs that exclude `xxx`, and which make _all other packages optional_. That way, if it has to remove other packages because they have a dependency relationship with `xxx`, it is given permission to do so.

But of course, we don't _want_ to remove all the packages in the environment; indeed, we want to keep as many of them as possible. So the solver has a specific pass that _maximizes the number of optional specs it keeps around_.
I was frankly unaware that y'all were trying to do this. It makes sense, don't get me wrong, but optional specs fed directly to `Resolve.solve` have had a specific meaning for quite some time. It's not entirely clear to me how this is going to be fixed.
So it sounds like we need two kinds of optional dependencies: 1) optional, but try hard to include it; and 2) optional, but try hard _not_ to include it.

It's important to note that pinned dependencies are _not_ always optional. In Stefan's example above, for instance, `python=3.6.1=2` starts out as "optional but try hard not to include it". But as soon as it _does_ get included in the environment, it is _no longer optional at all_.

Yeah, from a user's perspective I find the terminology confusing. It's the "pinned" that is the operational term here. Optional merely means that the pinning should have no bearing on whether or not the package gets installed.
So perhaps we can just find a better term to express the concept, to be less confusing to ourselves.
Well, we're trying to _implement_ pinning using conda's internal optional dependency mechanism. We don't necessarily need to use the term "optional" in public facing documentation, that's for sure.

But I realized over lunch how we can get this to work. We can create a "virtual package" that is installed in each environment. That package would include as optional dependencies every pinned package. This would ensure that conda behaves properly no matter what the operations are.
That sounds good. (And yes, please let's make sure the user-facing information deploys a less ambiguous term to flag this). I'm getting confused each time I try to parse the title of this very issue. ;-)
@stefanseefeld I just found an error that will/should be affecting your tests here...

https://github.com/conda/conda/blob/4.4.x/conda/models/index_record.py#L126

Should be `optional=True`, not `option=True`.
Good catch ! Unfortunately, the test still fails with that fixed. I'll keep looking...
The tests _should_ fail unless the metapackage idea is implemented. If you supply an "optional=True" directly to the `conda.resolve.Resolve.solve` spec list, it's going to try to install it---it just won't complain if it can't.
OK, so it sounds like we need a plan for how to attack this. I'd like to help, but don't think I have enough knowledge about the solver internals to even understand what needs to be done.
While working on the test I notice something isn't working quite right, I suspect the solver:

```
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
run_command(Commands.INSTALL, prefix, "flask")
assert package_is_installed(prefix, "python-3.6.1-2")
```
works fine, while
```
run_command(Commands.CONFIG, prefix,
                        "--add pinned_packages", "python=3.6.1-2")
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
run_command(Commands.INSTALL, prefix, "flask")
assert package_is_installed(prefix, "python-3.6.1-2")
```
yields the error
```
UnsatisfiableError: The following specifications were found to be in conflict:
  - flask
  - python 3.6.1-2 (optional)
```
@mcg1969 , do you have any idea why this may be happening, and where I should follow up (file a separate bug report, etc.) ?
Interesting. Normally I tell people _never doubt the solver_ but this is certainly sufficiently new functionality that we should check. I'll see what I can find.
ha ha! _never doubt the solver_.

`python=3.6.1-2` is not the correct spec. It's either `python=3.6.1=2` or `python 3.6.1 2`, depending on how you've implemented it. `python=3.6.1-2` means `Python version 3.6.1-2`, not `Python version 3.6.1 build 2`.
Thanks ! (Just a shot in the dark: the solver itself only treats a bunch of numbers (roughly speaking :-) ), which we generate from all the various inputs (match-spec, index content, etc.). I could imagine that our encoding of the request from those inputs into numbers might be wrong somehow, i.e. the query we submit to the solver proper may not be what we intended.)
Oh ! Now that isn't very intuitive, is it ? Is that documented somewhere ?
Anyhow, thanks for the quick find, I'll adjust my test logic and see whether it works.
Well, `python=3.6.1=2` is the proper command-line syntax, and that's documented. `python 3.6.1 2` is the proper MatchSpec syntax, which is also documented, but perhaps internally.
OK, fair enough. Thanks for your assistance !
```
run_command(Commands.CONFIG, prefix,
            "--add pinned_packages", "python=3.6.1=2")
run_command(Commands.INSTALL, prefix, "openssl")
assert not package_is_installed(prefix, "python")
```
is failing, i.e. python is being installed even though openssl doesn't depend on it. The MatchSpec objects look good, though.
@stefanseefeld 

> is failing, i.e. python is being installed even though openssl doesn't depend on it

Are you working in 4.3.x or 4.4.x?
Right, I understand.
This is on 4.4.x.
I guess that means then that the `optional=True` flag sent to the MatchSpec object isn't working correctly within resolve?
Is this on windows? If so, features are to blame, and we're discussing that in another issue.
This is GNU/Linux.
Ah, I'll bet I know why. You're passing an optional MatchSpec right to `Resolve.solve`, I'll bet.

That's not going to work, I don't think. The reason why is that I've been using the optional specification for a long time to effect package _removals_. When you do a `conda remove xxx`, the solver gets a set of specs that exclude `xxx`, and which make _all other packages optional_. That way, if it has to remove other packages because they have a dependency relationship with `xxx`, it is given permission to do so.

But of course, we don't _want_ to remove all the packages in the environment; indeed, we want to keep as many of them as possible. So the solver has a specific pass that _maximizes the number of optional specs it keeps around_.
I was frankly unaware that y'all were trying to do this. It makes sense, don't get me wrong, but optional specs fed directly to `Resolve.solve` have had a specific meaning for quite some time. It's not entirely clear to me how this is going to be fixed.
So it sounds like we need two kinds of optional dependencies: 1) optional, but try hard to include it; and 2) optional, but try hard _not_ to include it.

It's important to note that pinned dependencies are _not_ always optional. In Stefan's example above, for instance, `python=3.6.1=2` starts out as "optional but try hard not to include it". But as soon as it _does_ get included in the environment, it is _no longer optional at all_.

Yeah, from a user's perspective I find the terminology confusing. It's the "pinned" that is the operational term here. Optional merely means that the pinning should have no bearing on whether or not the package gets installed.
So perhaps we can just find a better term to express the concept, to be less confusing to ourselves.
Well, we're trying to _implement_ pinning using conda's internal optional dependency mechanism. We don't necessarily need to use the term "optional" in public facing documentation, that's for sure.

But I realized over lunch how we can get this to work. We can create a "virtual package" that is installed in each environment. That package would include as optional dependencies every pinned package. This would ensure that conda behaves properly no matter what the operations are.
That sounds good. (And yes, please let's make sure the user-facing information deploys a less ambiguous term to flag this). I'm getting confused each time I try to parse the title of this very issue. ;-)
@stefanseefeld I just found an error that will/should be affecting your tests here...

https://github.com/conda/conda/blob/4.4.x/conda/models/index_record.py#L126

Should be `optional=True`, not `option=True`.
Good catch ! Unfortunately, the test still fails with that fixed. I'll keep looking...
The tests _should_ fail unless the metapackage idea is implemented. If you supply an "optional=True" directly to the `conda.resolve.Resolve.solve` spec list, it's going to try to install it---it just won't complain if it can't.
OK, so it sounds like we need a plan for how to attack this. I'd like to help, but don't think I have enough knowledge about the solver internals to even understand what needs to be done.

Resolving patch:
diff --git a/conda/resolve.py b/conda/resolve.py
--- a/conda/resolve.py
+++ b/conda/resolve.py
@@ -510,6 +510,9 @@ def generate_feature_metric(self, C):
     def generate_removal_count(self, C, specs):
         return {'!'+self.push_MatchSpec(C, ms.name): 1 for ms in specs}
 
+    def generate_install_count(self, C, specs):
+        return {self.push_MatchSpec(C, ms.name): 1 for ms in specs if ms.optional}
+
     def generate_package_count(self, C, missing):
         return {self.push_MatchSpec(C, nm): 1 for nm in missing}
 
@@ -687,7 +690,7 @@ def install_specs(self, specs, installed, update_deps=True):
     def install(self, specs, installed=None, update_deps=True, returnall=False):
         # type: (List[str], Option[?], bool, bool) -> List[Dist]
         specs, preserve = self.install_specs(specs, installed or [], update_deps)
-        pkgs = self.solve(specs, returnall=returnall)
+        pkgs = self.solve(specs, returnall=returnall, _remove=False)
         self.restore_bad(pkgs, preserve)
         return pkgs
 
@@ -725,11 +728,11 @@ def remove_specs(self, specs, installed):
 
     def remove(self, specs, installed):
         specs, preserve = self.remove_specs(specs, installed)
-        pkgs = self.solve(specs)
+        pkgs = self.solve(specs, _remove=True)
         self.restore_bad(pkgs, preserve)
         return pkgs
 
-    def solve(self, specs, returnall=False):
+    def solve(self, specs, returnall=False, _remove=False):
         # type: (List[str], bool) -> List[Dist]
         try:
             stdoutlog.info("Solving package specifications: ")
@@ -771,9 +774,10 @@ def mysat(specs, add_if=False):
             speca.extend(MatchSpec(s) for s in specm)
 
             # Removed packages: minimize count
-            eq_optional_c = r2.generate_removal_count(C, speco)
-            solution, obj7 = C.minimize(eq_optional_c, solution)
-            log.debug('Package removal metric: %d', obj7)
+            if _remove:
+                eq_optional_c = r2.generate_removal_count(C, speco)
+                solution, obj7 = C.minimize(eq_optional_c, solution)
+                log.debug('Package removal metric: %d', obj7)
 
             # Requested packages: maximize versions
             eq_req_v, eq_req_b = r2.generate_version_metrics(C, specr)
@@ -795,6 +799,12 @@ def mysat(specs, add_if=False):
             solution, obj4 = C.minimize(eq_req_b, solution)
             log.debug('Initial package build metric: %d', obj4)
 
+            # Optional installations: minimize count
+            if not _remove:
+                eq_optional_install = r2.generate_install_count(C, speco)
+                solution, obj49 = C.minimize(eq_optional_install, solution)
+                log.debug('Optional package install metric: %d', obj49)
+
             # Dependencies: minimize the number of packages that need upgrading
             eq_u = r2.generate_update_count(C, speca)
             solution, obj50 = C.minimize(eq_u, solution)


Test patch:
diff --git a/tests/test_create.py b/tests/test_create.py
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -714,6 +714,16 @@ def test_package_pinning(self):
             assert not package_is_installed(prefix, "python-2.7")
             assert not package_is_installed(prefix, "itsdangerous-0.23")
 
+    @pytest.mark.xfail(on_win, strict=True, reason="On Windows the Python package is pulled in as 'vc' feature-provider.")
+    def test_package_optional_pinning(self):
+        with make_temp_env("") as prefix:
+            run_command(Commands.CONFIG, prefix,
+                        "--add pinned_packages", "python=3.6.1=2")
+            run_command(Commands.INSTALL, prefix, "openssl")
+            assert not package_is_installed(prefix, "python")
+            run_command(Commands.INSTALL, prefix, "flask")
+            assert package_is_installed(prefix, "python-3.6.1")
+
     def test_update_deps_flag_absent(self):
         with make_temp_env("python=2 itsdangerous=0.23") as prefix:
             assert package_is_installed(prefix, 'python-2')