Repository: conda/conda

Issue:
Can't install Boost 1.70.0 from conda-forge in 4.7.2 (possible regression with `run_constrained`/`constrains`)
<!--
Hi!  Read this; it's important.

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

If your issue is a bug report for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.anaconda.com access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

If you continue on,

   **please include all requested information below.**

If a maintainer determines the information is required to understand 
your issue, and if it is not provided, your issue may be
closed automatically.

-->

## Current Behavior
<!-- What actually happens?
     If you want to include console output, please use "Steps to Reproduce" below. -->
Conda 4.7.2 fails to create an environment with `python` and `boost 1.70.0`, when trying to create it as is described below. Note that the same command works in conda 4.6.14, but it has a slowdown problem, as described in https://github.com/conda-forge/boost-cpp-feedstock/issues/49.

### Steps to Reproduce
<!-- If the current behavior is a bug, please provide specific, minimal steps to independently reproduce.
     Include the exact conda commands that reproduce the issue and their output between the ticks below. -->
```
λ conda create -n boost_env -c conda-forge python boost==1.70.0 boost-cpp==1.70.0
Collecting package metadata (current_repodata.json): done
Solving environment: failed
Collecting package metadata (repodata.json): done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

  - boost-cpp==1.70.0 -> libboost[version='<0']
  - boost==1.70.0 -> boost-cpp=1.70.0 -> libboost[version='<0']
```

<details open><summary>If `boost` and `boost-cpp` are not pinned, it tries to install an older version (1.68.0):</summary><p>

```
λ conda create -n boost_env -c conda-forge python boost boost-cpp
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: W:\conda-canary\envs\boost_env

  added / updated specs:
    - boost
    - boost-cpp
    - python


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    boost-1.68.0               |py37hf75dd32_1001         763 KB  conda-forge
    boost-cpp-1.68.0           |    h6a4c333_1000        31.1 MB  conda-forge
    ca-certificates-2019.3.9   |       hecc5488_0         184 KB  conda-forge
    certifi-2019.3.9           |           py37_0         149 KB  conda-forge
    intel-openmp-2019.4        |              245         1.7 MB
    libblas-3.8.0              |            8_mkl         3.5 MB  conda-forge
    libcblas-3.8.0             |            8_mkl         3.5 MB  conda-forge
    liblapack-3.8.0            |            8_mkl         3.5 MB  conda-forge
    mkl-2019.4                 |              245       157.5 MB
    numpy-1.15.4               |py37h8078771_1002         3.8 MB  conda-forge
    openssl-1.1.1b             |       hfa6e2cd_2         4.8 MB  conda-forge
    pip-19.1.1                 |           py37_0         1.8 MB  conda-forge
    python-3.7.3               |       hb12ca83_0        17.8 MB  conda-forge
    setuptools-41.0.1          |           py37_0         658 KB  conda-forge
    sqlite-3.28.0              |       hfa6e2cd_0         985 KB  conda-forge
    vc-14.1                    |       h0510ff6_4           6 KB
    vs2015_runtime-14.15.26706 |       h3a45250_4         1.1 MB
    wheel-0.33.4               |           py37_0          52 KB  conda-forge
    wincertstore-0.2           |        py37_1002          13 KB  conda-forge
    zlib-1.2.11                |    h2fa13f4_1004         236 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       233.3 MB

The following NEW packages will be INSTALLED:

  boost              conda-forge/win-64::boost-1.68.0-py37hf75dd32_1001
  boost-cpp          conda-forge/win-64::boost-cpp-1.68.0-h6a4c333_1000
  ca-certificates    conda-forge/win-64::ca-certificates-2019.3.9-hecc5488_0
  certifi            conda-forge/win-64::certifi-2019.3.9-py37_0
  intel-openmp       pkgs/main/win-64::intel-openmp-2019.4-245
  libblas            conda-forge/win-64::libblas-3.8.0-8_mkl
  libcblas           conda-forge/win-64::libcblas-3.8.0-8_mkl
  liblapack          conda-forge/win-64::liblapack-3.8.0-8_mkl
  mkl                pkgs/main/win-64::mkl-2019.4-245
  numpy              conda-forge/win-64::numpy-1.15.4-py37h8078771_1002
  openssl            conda-forge/win-64::openssl-1.1.1b-hfa6e2cd_2
  pip                conda-forge/win-64::pip-19.1.1-py37_0
  python             conda-forge/win-64::python-3.7.3-hb12ca83_0
  setuptools         conda-forge/win-64::setuptools-41.0.1-py37_0
  sqlite             conda-forge/win-64::sqlite-3.28.0-hfa6e2cd_0
  vc                 pkgs/main/win-64::vc-14.1-h0510ff6_4
  vs2015_runtime     pkgs/main/win-64::vs2015_runtime-14.15.26706-h3a45250_4
  wheel              conda-forge/win-64::wheel-0.33.4-py37_0
  wincertstore       conda-forge/win-64::wincertstore-0.2-py37_1002
  zlib               conda-forge/win-64::zlib-1.2.11-h2fa13f4_1004
```

</p></details>

## Expected Behavior
It should create an env with `python`, `boost 1.70.0` and some dependencies.


## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
     active environment : None
       user config file : C:\Users\tadeu\.condarc
 populated config files :
          conda version : 4.7.2
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : __cuda=9.2
       base environment : W:\conda-canary  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : W:\conda-canary\pkgs
                          C:\Users\tadeu\.conda\pkgs
                          C:\Users\tadeu\AppData\Local\conda\conda\pkgs
       envs directories : W:\conda-canary\envs
                          C:\Users\tadeu\.conda\envs
                          C:\Users\tadeu\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.2 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
          administrator : True
             netrc file : None
           offline mode : False
```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

(empty)
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
λ conda list --show-channel-urls
# packages in environment at W:\conda-canary:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0    defaults
bzip2                     1.0.6                hfa6e2cd_5    defaults
ca-certificates           2019.5.15                     0    defaults
certifi                   2019.3.9                 py37_0    defaults
cffi                      1.12.2           py37h7a1dbc1_1    defaults
chardet                   3.0.4                    py37_1    defaults
conda                     4.7.2                    py37_0    conda-canary
conda-package-handling    1.3.0                    py37_0    defaults
console_shortcut          0.1.1                         3    defaults
cryptography              2.6.1            py37h7a1dbc1_0    defaults
idna                      2.8                      py37_0    defaults
libarchive                3.3.3                h0643e63_5    defaults
libiconv                  1.15                 h1df5818_7    defaults
libxml2                   2.9.9                h464c3ec_0    defaults
lz4-c                     1.8.1.2              h2fa13f4_0    defaults
lzo                       2.10                 h6df0209_2    defaults
menuinst                  1.4.16           py37he774522_0    defaults
openssl                   1.1.1c               he774522_1    defaults
pip                       19.0.3                   py37_0    defaults
powershell_shortcut       0.0.1                         2    defaults
pycosat                   0.6.3            py37hfa6e2cd_0    defaults
pycparser                 2.19                     py37_0    defaults
pyopenssl                 19.0.0                   py37_0    defaults
pysocks                   1.6.8                    py37_0    defaults
python                    3.7.3                h8c8aaf0_0    defaults
python-libarchive-c       2.8                      py37_6    defaults
pywin32                   223              py37hfa6e2cd_1    defaults
requests                  2.21.0                   py37_0    defaults
ruamel_yaml               0.15.46          py37hfa6e2cd_0    defaults
setuptools                41.0.0                   py37_0    defaults
six                       1.12.0                   py37_0    defaults
sqlite                    3.27.2               he774522_0    defaults
tqdm                      4.32.1                     py_0    defaults
urllib3                   1.24.1                   py37_0    defaults
vc                        14.1                 h0510ff6_4    defaults
vs2015_runtime            14.15.26706          h3a45250_0    defaults
wheel                     0.33.1                   py37_0    defaults
win_inet_pton             1.1.0                    py37_0    defaults
wincertstore              0.2                      py37_0    defaults
xz                        5.2.4                h2fa13f4_4    defaults
yaml                      0.1.7                hc54c509_2    defaults
zlib                      1.2.11               h62dcd97_3    defaults
zstd                      1.3.7                h508b16e_0    defaults
```
</p></details>



Discussion:
Note that this was also reported here: https://github.com/conda-forge/boost-cpp-feedstock/issues/51 (in case there's a particular fix for the package, although this looks more like a conda problem)

Resolving patch:
diff --git a/conda/core/solve.py b/conda/core/solve.py
--- a/conda/core/solve.py
+++ b/conda/core/solve.py
@@ -78,6 +78,7 @@ def __init__(self, prefix, channels, subdirs=(), specs_to_add=(), specs_to_remov
         self._index = None
         self._r = None
         self._prepared = False
+        self._pool_cache = {}
 
     def solve_for_transaction(self, update_modifier=NULL, deps_modifier=NULL, prune=NULL,
                               ignore_pinned=NULL, force_remove=NULL, force_reinstall=NULL):
@@ -388,9 +389,15 @@ def _find_inconsistent_packages(self, ssc):
         return ssc
 
     def _get_package_pool(self, ssc, specs):
-        pool = ssc.r.get_reduced_index(specs)
-        grouped_pool = groupby(lambda x: x.name, pool)
-        return {k: set(v) for k, v in iteritems(grouped_pool)}
+        specs = frozenset(specs)
+        if specs in self._pool_cache:
+            pool = self._pool_cache[specs]
+        else:
+            pool = ssc.r.get_reduced_index(specs)
+            grouped_pool = groupby(lambda x: x.name, pool)
+            pool = {k: set(v) for k, v in iteritems(grouped_pool)}
+            self._pool_cache[specs] = pool
+        return pool
 
     def _package_has_updates(self, ssc, spec, installed_pool):
         installed_prec = installed_pool.get(spec.name)
@@ -403,6 +410,7 @@ def _should_freeze(self, ssc, target_prec, conflict_specs, explicit_pool, instal
         # never, ever freeze anything if we have no history.
         if not ssc.specs_from_history_map:
             return False
+
         pkg_name = target_prec.name
         # if we are FREEZE_INSTALLED (first pass for install)
         freeze = ssc.update_modifier == UpdateModifier.FREEZE_INSTALLED
@@ -418,7 +426,6 @@ def _should_freeze(self, ssc, target_prec, conflict_specs, explicit_pool, instal
             if update_added:
                 spec_package_pool = self._get_package_pool(ssc, (target_prec.to_match_spec(), ))
                 for spec in self.specs_to_add_names:
-
                     new_explicit_pool = explicit_pool
                     ms = MatchSpec(spec)
                     updated_spec = self._package_has_updates(ssc, ms, installed_pool)
@@ -436,9 +443,22 @@ def _should_freeze(self, ssc, target_prec, conflict_specs, explicit_pool, instal
                         break
 
         # if all package specs have overlapping package choices (satisfiable in at least one way)
-        no_conflict = (freeze or update_added) and pkg_name not in conflict_specs
+        no_conflict = ((freeze or update_added) and
+                       pkg_name not in conflict_specs and
+                       (pkg_name not in explicit_pool or
+                        target_prec in explicit_pool[pkg_name]) and
+                       self._compare_pools(ssc, explicit_pool, target_prec.to_match_spec()))
         return no_conflict
 
+    def _compare_pools(self, ssc, explicit_pool, ms):
+        other_pool = self._get_package_pool(ssc, (ms, ))
+        match = True
+        for k in set(other_pool.keys()) & set(explicit_pool.keys()):
+            if not bool(other_pool[k] & explicit_pool[k]):
+                match = False
+                break
+        return match
+
     def _add_specs(self, ssc):
         # For the remaining specs in specs_map, add target to each spec. `target` is a reference
         # to the package currently existing in the environment. Setting target instructs the
@@ -462,6 +482,19 @@ def _add_specs(self, ssc):
              for _ in ssc.prefix_data.iter_records())))) or []
         conflict_specs = set(_.name for _ in conflict_specs)
 
+        for spec in ssc.specs_map.values():
+            ms = MatchSpec(spec)
+            if (ms.name in explicit_pool and
+                    not bool(set(ssc.r.find_matches(ms)) & explicit_pool[ms.name])):
+                conflict_specs.add(ms.name)
+        # PrefixGraph here does a toposort, so that we're iterating from parents downward.  This
+        #    ensures that any conflict from an indirect child should also get picked up.
+        for prec in PrefixGraph(ssc.prefix_data.iter_records()).records:
+            if prec.name in conflict_specs:
+                continue
+            if any(MatchSpec(dep).name in conflict_specs for dep in prec.get("depends", [])):
+                conflict_specs.add(prec.name)
+
         for pkg_name, spec in iteritems(ssc.specs_map):
             matches_for_spec = tuple(prec for prec in ssc.solution_precs if spec.match(prec))
             if matches_for_spec:
@@ -492,8 +525,8 @@ def _add_specs(self, ssc):
         pin_overrides = set()
         for s in ssc.pinned_specs:
             if s.name in explicit_pool:
-                if s.name not in self.specs_to_add_names:
-                    ssc.specs_map[s.name] = s
+                if s.name not in self.specs_to_add_names and not ssc.ignore_pinned:
+                    ssc.specs_map[s.name] = MatchSpec(s, optional=False)
                 elif explicit_pool[s.name] & self._get_package_pool(ssc, [s])[s.name]:
                     ssc.specs_map[s.name] = MatchSpec(s, optional=False)
                     pin_overrides.add(s.name)
@@ -505,7 +538,13 @@ def _add_specs(self, ssc):
             for prec in ssc.prefix_data.iter_records():
                 if prec.name not in ssc.specs_map:
                     if (prec.name not in conflict_specs and
-                            (prec.name not in explicit_pool or prec in explicit_pool[prec.name])):
+                            (prec.name not in explicit_pool or
+                             prec in explicit_pool[prec.name]) and
+                            # because it's not just immediate deps, but also
+                            # upstream things that matter, we must ensure
+                            # overlap for dependencies of things that have
+                            # otherwise passed our tests
+                            self._compare_pools(ssc, explicit_pool, prec.to_match_spec())):
                         ssc.specs_map[prec.name] = prec.to_match_spec()
                     else:
                         ssc.specs_map[prec.name] = MatchSpec(
@@ -616,13 +655,18 @@ def _run_sat(self, ssc):
             # track_features_specs or pinned_specs, which we should raise an error on.
             specs_map_set = set(itervalues(ssc.specs_map))
             grouped_specs = groupby(lambda s: s in specs_map_set, conflicting_specs)
-            conflicting_pinned_specs = groupby(lambda s: s in ssc.pinned_specs, conflicting_specs)
+            # force optional to true. This is what it is originally in
+            # pinned_specs, but we override that in _add_specs to make it
+            # non-optional when there's a name match in the explicit package
+            # pool
+            conflicting_pinned_specs = groupby(lambda s: MatchSpec(s, optional=True)
+                                               in ssc.pinned_specs, conflicting_specs)
 
             if conflicting_pinned_specs.get(True):
                 in_specs_map = grouped_specs.get(True, ())
                 pinned_conflicts = conflicting_pinned_specs.get(True, ())
                 in_specs_map_or_specs_to_add = ((set(in_specs_map) | set(self.specs_to_add))
-                                                - set(ssc.pinned_specs))
+                                                - set(pinned_conflicts))
 
                 raise SpecsConfigurationConflictError(
                     sorted(s.__str__() for s in in_specs_map_or_specs_to_add),
diff --git a/conda/resolve.py b/conda/resolve.py
--- a/conda/resolve.py
+++ b/conda/resolve.py
@@ -9,12 +9,13 @@
 
 from ._vendor.auxlib.collection import frozendict
 from ._vendor.auxlib.decorators import memoize, memoizemethod
-from ._vendor.toolz import concat, concatv, groupby
+from ._vendor.toolz import concat, groupby
 from .base.constants import ChannelPriority, MAX_CHANNEL_PRIORITY, SatSolverChoice
 from .base.context import context
 from .common.compat import iteritems, iterkeys, itervalues, odict, on_win, text_type
 from .common.io import time_recorder
-from .common.logic import Clauses, CryptoMiniSatSolver, PycoSatSolver, PySatSolver
+from .common.logic import (Clauses, CryptoMiniSatSolver, PycoSatSolver, PySatSolver,
+                           minimal_unsatisfiable_subset)
 from .common.toposort import toposort
 from .exceptions import (CondaDependencyError, InvalidSpec, ResolvePackageNotFound,
                          UnsatisfiableError)
@@ -421,8 +422,8 @@ def find_conflicts(self, specs, specs_to_add=None, history_specs=None):
             sdeps_with_dep = {k: v.get(dep) for k, v in sdeps.items() if dep in v.keys()}
             if len(sdeps_with_dep) <= 1:
                 continue
-            intersection = set.intersection(*sdeps_with_dep.values())
-            if len(intersection) != 0:
+            # if the two pools overlap, we're good.  Next dep.
+            if bool(set.intersection(*sdeps_with_dep.values())):
                 continue
             filter = {}
             for fkeys in sdeps_with_dep.values():
@@ -586,6 +587,18 @@ def filter_group(_specs):
                 reduced = filter_group([s])
                 if reduced:
                     slist.append(s)
+                elif reduced is None:
+                    break
+            if reduced is None:
+                # This filter reset means that unsatisfiable indexes leak through.
+                filter_out = {prec: False if val else "feature not enabled"
+                              for prec, val in iteritems(self.default_filter(features))}
+                # TODO: raise unsatisfiable exception here
+                # Messaging to users should be more descriptive.
+                # 1. Are there no direct matches?
+                # 2. Are there no matches for first-level dependencies?
+                # 3. Have the first level dependencies been invalidated?
+                break
 
         # Determine all valid packages in the dependency graph
         reduced_index2 = {prec: prec for prec in (make_feature_record(fstr) for fstr in features)}
@@ -983,31 +996,40 @@ def environment_is_consistent(self, installed):
         solution = C.sat(constraints)
         return bool(solution)
 
-    def get_conflicting_specs(self, specs, original_specs=None):
+    def get_conflicting_specs(self, specs):
         if not specs:
             return ()
+        reduced_index = self.get_reduced_index(specs)
+
+        # Check if satisfiable
+        def mysat(specs, add_if=False):
+            constraints = r2.generate_spec_constraints(C, specs)
+            return C.sat(constraints, add_if)
 
-        reduced_index = self.get_reduced_index(tuple(specs), sort_by_exactness=False)
         r2 = Resolve(reduced_index, True, channels=self.channels)
-        scp = context.channel_priority == ChannelPriority.STRICT
-        unsat_specs = {s for s in specs if not r2.find_matches_with_strict(s, scp)}
-        if not unsat_specs:
+        C = r2.gen_clauses()
+        solution = mysat(specs, True)
+        if solution:
             return ()
-
-        # This first result is just a single unsatisfiable core. There may be several.
-        satisfiable_specs = set(specs) - set(unsat_specs)
-
-        # In this loop, we test each unsatisfiable spec individually against the satisfiable
-        # specs to ensure there are no other unsatisfiable specs in the set.
-        final_unsat_specs = set()
-        while unsat_specs:
-            this_spec = unsat_specs.pop()
-            final_unsat_specs.add(this_spec)
-            test_specs = tuple(concatv((this_spec, ), satisfiable_specs))
-            r2 = Resolve(self.get_reduced_index(test_specs), True, channels=self.channels)
-            _unsat_specs = {s for s in specs if not r2.find_matches_with_strict(s, scp)}
-            unsat_specs.update(_unsat_specs - final_unsat_specs)
-            satisfiable_specs -= set(unsat_specs)
+        else:
+            # This first result is just a single unsatisfiable core. There may be several.
+            unsat_specs = list(minimal_unsatisfiable_subset(specs, sat=mysat))
+            satisfiable_specs = set(specs) - set(unsat_specs)
+
+            # In this loop, we test each unsatisfiable spec individually against the satisfiable
+            # specs to ensure there are no other unsatisfiable specs in the set.
+            final_unsat_specs = set()
+            while unsat_specs:
+                this_spec = unsat_specs.pop(0)
+                final_unsat_specs.add(this_spec)
+                test_specs = satisfiable_specs | {this_spec}
+                C = r2.gen_clauses()  # TODO: wasteful call, but Clauses() needs refactored
+                solution = mysat(test_specs, True)
+                if not solution:
+                    these_unsat = minimal_unsatisfiable_subset(test_specs, sat=mysat)
+                    if len(these_unsat) > 1:
+                        unsat_specs.extend(these_unsat)
+                        satisfiable_specs -= set(unsat_specs)
         return tuple(final_unsat_specs)
 
     def bad_installed(self, installed, new_specs):


Test patch:
diff --git a/tests/test_create.py b/tests/test_create.py
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -2778,6 +2778,18 @@ def test_legacy_repodata(self):
             assert exists(join(prefix, PYTHON_BINARY))
             assert package_is_installed(prefix, 'moto=1.3.7')
 
+    def test_cross_channel_incompatibility(self):
+        # regression test for https://github.com/conda/conda/issues/8772
+        # conda-forge puts a run_constrains on libboost, which they don't have on conda-forge.
+        #   This is a way of forcing libboost to be removed.  It's a way that they achieve
+        #   mutual exclusivity with the boost from defaults that works differently.
+
+        # if this test passes, we'll hit the DryRunExit exception, instead of an UnsatisfiableError
+        with pytest.raises(DryRunExit):
+            stdout, stderr, _ = run_command(Commands.CREATE, "dummy_channel_incompat_test",
+                                            '--dry-run', '-c', 'conda-forge', 'python',
+                                            'boost==1.70.0', 'boost-cpp==1.70.0')
+
 @pytest.mark.skipif(True, reason="get the rest of Solve API worked out first")
 @pytest.mark.integration
 class PrivateEnvIntegrationTests(TestCase):