Repository: conda/conda

Issue:
conda 4.7 does not find a solution when installing two packages separately.
When creating an environment with `libpysal` and then installing `geopandas`,

```shell
conda create --name TEST libpysal --yes
conda activate TEST
conda install geopandas --yes
```

`conda 4.7` returns `UnsatisfiableError` b/c it refuses to downgrade a `python` build number to get a version with `readline 7`.

This is probably an edge case that will go away once the `readline 8` migration on `conda-forge` is complete. But CIs will be broken every migration that touches a `python` dependency.

Note that creating the environment in a single go like: `conda create --name TEST libpysal geopandas --yes`, or downgrading to `conda 4.6` works as expected.


Discussion:
While I understand that `conda 4.7` make it harder to downgrade the `python` version I guess that, b/c we are using build number to provide variants, a smaller build number is not a downgrade.
I do not believe the issue here is with the Python dowgrade since the following works:
```
conda create -c conda-forge --name TEST libpysal --yes
source activate TEST
conda install -c conda-forge readline=7
```
Odd, the following works as well:
```
conda create -c conda-forge --name TEST libpysal --yes
source activate TEST
conda install -c conda-forge readline=7 --yes
conda install -c conda-forge geopandas --yes
```
Can you confirm that

```
conda create -c conda-forge --name TEST libpysal --yes
source activate TEST
conda install -c conda-forge geopandas --yes
```

works? I forgot to mention but I'm using `strict`.


```
conda create -c conda-forge --name TEST libpysal --yes
source activate TEST
conda install -c conda-forge geopandas --yes
```
Does not work, I need to add `conda install -c conda-forge readline=7 --yes`  before the install of `geopandas`.   This command should not be required, conda should find the solution without it.
I'll look into this.  The freezing is probably being a little aggressive.  A better way to say it is actually "the conflict finding is not working well enough" because conda tries to not freeze anything that has a conflict.
conda on master doesn't show this problem anymore.  I don't understand why - I'm not sure that we've changed anything.  Perhaps the CF readline migration has just gotten far enough that it isn't an issue anymore?
> Perhaps the CF readline migration has just gotten far enough that it isn't an issue anymore?

Yep. I was unable to reproduce it a few days ago with the current release of conda. The bug is still there, we just don't have an easy way to reproduce it :smile: 
I think this issue has to do with the change to using `FREEZE_INSTALLED` as the default for the solver's update handling in 4.7.  After the environment with `libpysal` is created those packages are frozen unless a conflict is found.  If certain packages are frozen the solver is unable to find a solution.
Here is an example using some simple packages:

```
(base) ~$ conda --version
conda 4.7.5
(base) ~$ conda create -n test -c jjhelmus foobar qux -y --dry-run
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/jhelmus/conda/envs/test

  added / updated specs:
    - foobar
    - qux


The following NEW packages will be INSTALLED:

  foobar             jjhelmus/linux-64::foobar-1.0-0
  libbar             jjhelmus/linux-64::libbar-1.0-0
  libfoo             jjhelmus/linux-64::libfoo-2.0-0
  qux                jjhelmus/linux-64::qux-2.0-0



DryRunExit: Dry run. Exiting.
(base) ~$ conda create -n test -c jjhelmus foobar -y
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/jhelmus/conda/envs/test

  added / updated specs:
    - foobar


The following NEW packages will be INSTALLED:

  foobar             jjhelmus/linux-64::foobar-2.0-0
  libbar             jjhelmus/linux-64::libbar-2.0-0
  libfoo             jjhelmus/linux-64::libfoo-2.0-0


Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) ~$ conda install qux -c jjhelmus -n test
Collecting package metadata (current_repodata.json): done
Solving environment: failed
Collecting package metadata (repodata.json): done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

  - jjhelmus/linux-64::foobar==2.0=0
  - jjhelmus/linux-64::libbar==2.0=0
  - jjhelmus/linux-64::libfoo==2.0=0
  - qux
```

This issue here is that the `foobar` package already installed in the environment is frozen when installing  `qux` and therefore cannot be downgraded to version 1.0 which is needed for a solution.  Adding a fallback to the solver when satisfiability is not possible with FREEZE_INSTALL will address this problem.

Resolving patch:
diff --git a/conda/cli/conda_argparse.py b/conda/cli/conda_argparse.py
--- a/conda/cli/conda_argparse.py
+++ b/conda/cli/conda_argparse.py
@@ -1542,6 +1542,14 @@ def add_parser_update_modifiers(solver_mode_options):
         help="Update all installed packages in the environment.",
         default=NULL,
     )
+    update_modifiers.add_argument(
+        "--update-specs",
+        action="store_const",
+        const=UpdateModifier.UPDATE_SPECS,
+        dest="update_modifier",
+        help="Update based on provided specifications.",
+        default=NULL,
+    )
 
 
 def add_parser_prune(p):
diff --git a/conda/cli/install.py b/conda/cli/install.py
--- a/conda/cli/install.py
+++ b/conda/cli/install.py
@@ -295,12 +295,27 @@ def install(args, parser, command='install'):
                 raise PackagesNotFoundError(e._formatted_chains, channels_urls)
 
         except (UnsatisfiableError, SystemExit, SpecsConfigurationConflictError) as e:
-            # end of the line.  Raise the exception
-            if repodata_fn == repodata_fns[-1]:
-                # Unsatisfiable package specifications/no such revision/import error
-                if e.args and 'could not import' in e.args[0]:
-                    raise CondaImportError(text_type(e))
-                raise
+            # Quick solve with frozen env failed.  Try again without that.
+            if isinstall and args.update_modifier == NULL:
+                try:
+                    log.info("Initial quick solve with frozen env failed.  "
+                             "Unfreezing env and trying again.")
+                    unlink_link_transaction = solver.solve_for_transaction(
+                        deps_modifier=deps_modifier,
+                        update_modifier=UpdateModifier.UPDATE_SPECS,
+                        force_reinstall=context.force_reinstall or context.force,
+                    )
+                except (UnsatisfiableError, SystemExit, SpecsConfigurationConflictError) as e:
+                    # Unsatisfiable package specifications/no such revision/import error
+                    if e.args and 'could not import' in e.args[0]:
+                        raise CondaImportError(text_type(e))
+            else:
+                # end of the line.  Raise the exception
+                if repodata_fn == repodata_fns[-1]:
+                    # Unsatisfiable package specifications/no such revision/import error
+                    if e.args and 'could not import' in e.args[0]:
+                        raise CondaImportError(text_type(e))
+                    raise
     handle_txn(unlink_link_transaction, prefix, args, newenv)
 
 
diff --git a/conda/core/solve.py b/conda/core/solve.py
--- a/conda/core/solve.py
+++ b/conda/core/solve.py
@@ -25,8 +25,7 @@
 from ..common.constants import NULL
 from ..common.io import Spinner, dashlist, time_recorder
 from ..common.path import get_major_minor_version, paths_equal
-from ..exceptions import (PackagesNotFoundError, SpecsConfigurationConflictError,
-                          ResolvePackageNotFound)
+from ..exceptions import PackagesNotFoundError, SpecsConfigurationConflictError
 from ..history import History
 from ..models.channel import Channel
 from ..models.enums import NoarchType
@@ -400,41 +399,13 @@ def _should_freeze(self, ssc, target_prec, conflict_specs, explicit_pool, instal
         # never, ever freeze anything if we have no history.
         if not ssc.specs_from_history_map:
             return False
-
-        pkg_name = target_prec.name
-        # if we are FREEZE_INSTALLED (first pass for install)
-        freeze = ssc.update_modifier == UpdateModifier.FREEZE_INSTALLED
-
-        if not freeze:
-            # we are UPDATE_SPECS (solve_for_diff or update specs),
-            #       AND the spec is not an explicit spec
-            update_added = (ssc.update_modifier == UpdateModifier.UPDATE_SPECS and
-                            pkg_name not in self.specs_to_add_names)
-            # more expensive, but better check.  If anything in the package pool for this spec
-            #    overlaps with the explicit package pool (by name), but they don't share any
-            #    actual records, then this target_prec conflicts and should not be pinned
-            if update_added:
-                spec_package_pool = ssc.r._get_package_pool((target_prec.to_match_spec(), ))
-                for spec in self.specs_to_add_names:
-                    new_explicit_pool = explicit_pool
-                    ms = MatchSpec(spec)
-                    updated_spec = self._package_has_updates(ssc, ms, installed_pool)
-                    if updated_spec:
-                        try:
-                            new_explicit_pool = ssc.r._get_package_pool((updated_spec, ))
-                        except ResolvePackageNotFound:
-                            update_added = False
-                            break
-
-                    if ms.name in spec_package_pool:
-                        update_added &= bool(spec_package_pool[ms.name] &
-                                             new_explicit_pool[ms.name])
-                    if not update_added:
-                        break
+        # never freeze if not in FREEZE_INSTALLED mode
+        if ssc.update_modifier != UpdateModifier.FREEZE_INSTALLED:
+            return False
 
         # if all package specs have overlapping package choices (satisfiable in at least one way)
-        no_conflict = ((freeze or update_added) and
-                       pkg_name not in conflict_specs and
+        pkg_name = target_prec.name
+        no_conflict = (pkg_name not in conflict_specs and
                        (pkg_name not in explicit_pool or
                         target_prec in explicit_pool[pkg_name]) and
                        self._compare_pools(ssc, explicit_pool, target_prec.to_match_spec()))


Test patch:
diff --git a/tests/core/test_solve.py b/tests/core/test_solve.py
--- a/tests/core/test_solve.py
+++ b/tests/core/test_solve.py
@@ -23,7 +23,7 @@
 from conda.models.records import PrefixRecord
 from conda.resolve import MatchSpec
 from ..helpers import get_index_r_1, get_index_r_2, get_index_r_4, \
-    get_index_r_5, get_index_cuda
+    get_index_r_5, get_index_cuda, get_index_must_unfreeze
 
 from conda.common.compat import iteritems
 
@@ -116,6 +116,19 @@ def get_solver_aggregate_2(specs_to_add=(), specs_to_remove=(), prefix_records=(
         yield solver
 
 
+@contextmanager
+def get_solver_must_unfreeze(specs_to_add=(), specs_to_remove=(), prefix_records=(), history_specs=()):
+    PrefixData._cache_.clear()
+    pd = PrefixData(TEST_PREFIX)
+    pd._PrefixData__prefix_records = {rec.name: PrefixRecord.from_objects(rec) for rec in prefix_records}
+    spec_map = {spec.name: spec for spec in history_specs}
+    get_index_must_unfreeze(context.subdir)
+    with patch.object(History, 'get_requested_specs_map', return_value=spec_map):
+        solver = Solver(TEST_PREFIX, (Channel('channel-freeze'),), (context.subdir,),
+                        specs_to_add=specs_to_add, specs_to_remove=specs_to_remove)
+        yield solver
+
+
 @contextmanager
 def get_solver_cuda(specs_to_add=(), specs_to_remove=(), prefix_records=(), history_specs=()):
     PrefixData._cache_.clear()
@@ -266,15 +279,24 @@ def test_prune_1():
         pprint(convert_to_dist_str(link_precs))
         unlink_order = (
             'channel-1::accelerate-1.1.0-np16py27_p0',
+            'channel-1::mkl-11.0-np16py27_p0',
+            'channel-1::scikit-learn-0.13.1-np16py27_p0',
             'channel-1::numbapro-0.11.0-np16py27_p0',
+            'channel-1::scipy-0.12.0-np16py27_p0',
+            'channel-1::numexpr-2.1-np16py27_p0',
             'channel-1::numba-0.8.1-np16py27_0',
+            'channel-1::numpy-1.6.2-py27_p4',
+            'channel-1::mkl-service-1.0.0-py27_p0',
             'channel-1::meta-0.4.2.dev-py27_0',
             'channel-1::llvmpy-0.11.2-py27_0',
             'channel-1::bitarray-0.8.1-py27_0',
             'channel-1::llvm-3.2-0',
+            'channel-1::mkl-rt-11.0-p0',
             'channel-1::libnvvm-1.0-p0',
         )
-        link_order = tuple()
+        link_order = (
+            'channel-1::numpy-1.6.2-py27_4',
+        )
         assert convert_to_dist_str(unlink_precs) == unlink_order
         assert convert_to_dist_str(link_precs) == link_order
 
@@ -814,6 +836,62 @@ def test_conda_downgrade():
         sys.prefix = saved_sys_prefix
 
 
+def test_unfreeze_when_required():
+    # The available packages are:
+    # libfoo 1.0, 2.0
+    # libbar 1.0, 2.0
+    # foobar 1.0 : depends on libfoo 1.0, libbar 2.0
+    # foobar 2.0 : depends on libfoo 2.0, libbar 2.0
+    # qux 1.0: depends on libfoo 1.0, libbar 2.0
+    # qux 2.0: depends on libfoo 2.0, libbar 1.0
+    #
+    # qux 1.0 and foobar 1.0 can be installed at the same time but
+    # if foobar is installed first it must be downgraded from 2.0.
+    # If foobar is frozen then no solution exists.
+
+    specs = [MatchSpec("foobar"), MatchSpec('qux')]
+    with get_solver_must_unfreeze(specs) as solver:
+        final_state_1 = solver.solve_final_state()
+        print(convert_to_dist_str(final_state_1))
+        order = (
+            'channel-freeze::libbar-2.0-0',
+            'channel-freeze::libfoo-1.0-0',
+            'channel-freeze::foobar-1.0-0',
+            'channel-freeze::qux-1.0-0',
+        )
+        assert convert_to_dist_str(final_state_1) == order
+
+    specs = MatchSpec("foobar"),
+    with get_solver_must_unfreeze(specs) as solver:
+        final_state_1 = solver.solve_final_state()
+        print(convert_to_dist_str(final_state_1))
+        order = (
+            'channel-freeze::libbar-2.0-0',
+            'channel-freeze::libfoo-2.0-0',
+            'channel-freeze::foobar-2.0-0',
+        )
+        assert convert_to_dist_str(final_state_1) == order
+
+    # When frozen there is no solution
+    specs_to_add = MatchSpec("qux"),
+    with get_solver_must_unfreeze(specs_to_add, prefix_records=final_state_1, history_specs=specs) as solver:
+        with pytest.raises(UnsatisfiableError):
+            solver.solve_final_state(update_modifier=UpdateModifier.FREEZE_INSTALLED)
+
+    specs_to_add = MatchSpec("qux"),
+    with get_solver_must_unfreeze(specs_to_add, prefix_records=final_state_1, history_specs=specs) as solver:
+        final_state_2 = solver.solve_final_state(update_modifier=UpdateModifier.UPDATE_SPECS)
+        # PrefixDag(final_state_2, specs).open_url()
+        print(convert_to_dist_str(final_state_2))
+        order = (
+            'channel-freeze::libbar-2.0-0',
+            'channel-freeze::libfoo-1.0-0',
+            'channel-freeze::foobar-1.0-0',
+            'channel-freeze::qux-1.0-0',
+        )
+        assert convert_to_dist_str(final_state_2) == order
+
+
 def test_auto_update_conda():
     specs = MatchSpec("conda=1.3"),
     with get_solver(specs) as solver:
@@ -1346,12 +1424,18 @@ def test_fast_update_with_update_modifier_not_set():
         pprint(convert_to_dist_str(unlink_precs))
         pprint(convert_to_dist_str(link_precs))
         unlink_order = (
+            'channel-4::python-2.7.14-h89e7a4a_22',
             'channel-4::sqlite-3.21.0-h1bed415_2',
+            'channel-4::libedit-3.1-heed3624_0',
             'channel-4::openssl-1.0.2l-h077ae2c_5',
+            'channel-4::ncurses-6.0-h9df7e31_2',
         )
         link_order = (
+            'channel-4::ncurses-6.1-hf484d3e_0',
             'channel-4::openssl-1.0.2p-h14c3975_0',
-            'channel-4::sqlite-3.23.1-he433501_0',
+            'channel-4::libedit-3.1.20170329-h6b74fdf_2',
+            'channel-4::sqlite-3.24.0-h84994c4_0',  # sqlite is upgraded
+            'channel-4::python-2.7.15-h1571d57_0',  # python is not upgraded
         )
         assert convert_to_dist_str(unlink_precs) == unlink_order
         assert convert_to_dist_str(link_precs) == link_order
diff --git a/tests/helpers.py b/tests/helpers.py
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -259,6 +259,116 @@ def get_index_r_5(subdir=context.subdir):
     return index, r
 
 
+@memoize
+def get_index_must_unfreeze(subdir=context.subdir):
+    repodata = {
+        "info": {
+            "subdir": subdir,
+            "arch": context.arch_name,
+            "platform": context.platform,
+        },
+        "packages": {
+            "foobar-1.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [
+                    "libbar 2.0.*",
+                    "libfoo 1.0.*"
+                ],
+                "md5": "11ec1194bcc56b9a53c127142a272772",
+                "name": "foobar",
+                "timestamp": 1562861325613,
+                "version": "1.0"
+            },
+            "foobar-2.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [
+                    "libbar 2.0.*",
+                    "libfoo 2.0.*"
+                ],
+                "md5": "f8eb5a7fa1ff6dead4e360631a6cd048",
+                "name": "foobar",
+                "version": "2.0"
+            },
+
+            "libbar-1.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [],
+                "md5": "f51f4d48a541b7105b5e343704114f0f",
+                "name": "libbar",
+                "timestamp": 1562858881022,
+                "version": "1.0"
+            },
+            "libbar-2.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [],
+                "md5": "27f4e717ed263f909074f64d9cbf935d",
+                "name": "libbar",
+                "timestamp": 1562858881748,
+                "version": "2.0"
+            },
+
+            "libfoo-1.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [],
+                "md5": "ad7c088566ffe2389958daedf8ff312c",
+                "name": "libfoo",
+                "timestamp": 1562858763881,
+                "version": "1.0"
+            },
+            "libfoo-2.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [],
+                "md5": "daf7af7086d8f22be49ae11bdc41f332",
+                "name": "libfoo",
+                "timestamp": 1562858836924,
+                "version": "2.0"
+            },
+
+            "qux-1.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [
+                    "libbar 2.0.*",
+                    "libfoo 1.0.*"
+                ],
+                "md5": "18604cbe4f789fe853232eef4babd4f9",
+                "name": "qux",
+                "timestamp": 1562861393808,
+                "version": "1.0"
+            },
+            "qux-2.0-0.tar.bz2": {
+                "build": "0",
+                "build_number": 0,
+                "depends": [
+                    "libbar 1.0.*",
+                    "libfoo 2.0.*"
+                ],
+                "md5": "892aa4b9ec64b67045a46866ef1ea488",
+                "name": "qux",
+                "timestamp": 1562861394828,
+                "version": "2.0"
+            }
+        }
+    }
+    channel = Channel('https://conda.anaconda.org/channel-freeze/%s' % subdir)
+    sd = SubdirData(channel)
+    with env_var("CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY", "false", stack_callback=conda_tests_ctxt_mgmt_def_pol):
+        sd._process_raw_repodata_str(json.dumps(repodata))
+    sd._loaded = True
+    SubdirData._cache_[channel.url(with_credentials=True)] = sd
+
+    index = {prec: prec for prec in sd._package_records}
+    r = Resolve(index, channels=(channel,))
+
+    return index, r
+
+
 # Do not memoize this get_index to allow different CUDA versions to be detected
 def get_index_cuda(subdir=context.subdir):
     with open(join(dirname(__file__), 'data', 'index.json')) as fi:
@@ -283,4 +393,4 @@ def get_index_cuda(subdir=context.subdir):
 
     add_feature_records_legacy(index)
     r = Resolve(index, channels=(channel,))
-    return index, r
\ No newline at end of file
+    return index, r