Repository: conda/conda

Issue:
Conflict with package, which just has been built
@marscher commented on [Fri Feb 17 2017](https://github.com/conda/conda-build/issues/1748)

The build phase succeeded, but when the test phase tries to create the environment a conflict is raised containing only the actual package, which just has been built. I've checked manually, that all dependencies are available for Python 3.6 and the desired Numpy version (1.11), so it is pretty unclear to me, why this is happening.

* conda-build is 2.1.4. 
* The recipe can be found here:
   https://github.com/omnia-md/conda-recipes/tree/master/pyemma
* conda build log attached (debug level):
  [build_debug_log.txt](https://github.com/conda/conda-build/files/783354/build_debug_log.txt)


cc omnia-md/conda-recipes/issues/702


---

@msarahan commented on [Fri Feb 17 2017](https://github.com/conda/conda-build/issues/1748#issuecomment-280668089)

Are all dependencies actually installable for py 36?

If I try to create an environment with this command (just taken from your meta.yaml), it shows that bhmm is not available:

```
[dev@e9f4c54fd4ff downloads]$ conda create -n pyemma -c omnia bhmm=0.6.1 decorator=4 funcsigs joblib=0.9.4 matplotlib mdtraj=1.8 mock msmtools=1.2 numpy=1.11 progress_reporter=1.3.1 psutil=3.1.1 python=3.6 pyyaml scipy setuptools six thermotools=0.2.3
Fetching package metadata .........
Solving package specifications: ....


UnsatisfiableError: The following specifications were found to be in conflict:
  - bhmm 0.6*
  - python 3.6*
Use "conda info <package>" to see the dependencies for each package.
```

Conda search shows that it is, though:

```
[dev@e9f4c54fd4ff downloads]$ conda search -c omnia bhmm
Fetching package metadata .........
bhmm                         0.2.0                    py27_0  omnia
                             0.3.0                np18py27_0  omnia
                             0.3.0                np19py27_0  omnia
                             0.4.2                np18py27_0  omnia
                             0.4.2                np19py27_0  omnia
                             0.5.0                np18py27_0  omnia
                             0.5.0                np19py27_0  omnia
                             0.5.1               np110py27_0  omnia
                             0.5.1               np110py34_0  omnia
                             0.5.1               np110py35_0  omnia
                             0.5.1                np18py27_0  omnia
                             0.5.1                np18py33_0  omnia
                             0.5.1                np18py34_0  omnia
                             0.5.1                np19py27_0  omnia
                             0.5.1                np19py33_0  omnia
                             0.5.1                np19py34_0  omnia
                             0.5.2               np110py27_0  omnia
                             0.5.2               np110py34_0  omnia
                             0.5.2               np110py35_0  omnia
                             0.5.2                np19py27_0  omnia
                             0.5.2                np19py34_0  omnia
                             0.5.2                np19py35_0  omnia
                             0.6.1               np110py27_0  omnia
                             0.6.1               np110py34_0  omnia
                             0.6.1               np110py35_0  omnia
                             0.6.1               np111py27_0  omnia
                             0.6.1               np111py34_0  omnia
                             0.6.1               np111py35_0  omnia
                             0.6.1               np111py36_0  omnia
                             0.6.1                np19py27_0  omnia
                             0.6.1                np19py34_0  omnia
                             0.6.1                np19py35_0  omnia
```

This means that one of bhmm's dependencies (or even further up the chain) is not installable for python 3.6.  However, creating an env with just bhmm works:

```
[dev@e9f4c54fd4ff downloads]$ conda create -n bhmm bhmm=0.6.1 numpy=1.11 python=3.6 -c omnia
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /opt/miniconda/envs/bhmm:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-3.6.0               |                0        16.3 MB
    cython-0.25.2              |           py36_0         6.5 MB
    decorator-4.0.11           |           py36_0          15 KB
    numpy-1.11.3               |           py36_0         6.7 MB
    setuptools-27.2.0          |           py36_0         523 KB
    six-1.10.0                 |           py36_0          19 KB
    wheel-0.29.0               |           py36_0          88 KB
    pip-9.0.1                  |           py36_1         1.7 MB
    scipy-0.18.1               |      np111py36_1        31.2 MB
    msmtools-1.2               |      np111py36_0         1.2 MB  omnia
    bhmm-0.6.1                 |      np111py36_0         481 KB  omnia
    ------------------------------------------------------------
                                           Total:        64.7 MB

The following NEW packages will be INSTALLED:

    bhmm:        0.6.1-np111py36_0  omnia
    cython:      0.25.2-py36_0
    decorator:   4.0.11-py36_0
    libgfortran: 3.0.0-1
    mkl:         2017.0.1-0
    msmtools:    1.2-np111py36_0    omnia
    numpy:       1.11.3-py36_0
    openssl:     1.0.2k-0
    pip:         9.0.1-py36_1
    python:      3.6.0-0
    readline:    6.2-2
    scipy:       0.18.1-np111py36_1
    setuptools:  27.2.0-py36_0
    six:         1.10.0-py36_0
    sqlite:      3.13.0-0
    tk:          8.5.18-0
    wheel:       0.29.0-py36_0
    xz:          5.2.2-1
    zlib:        1.2.8-3

Proceed ([y]/n)? n
```

This is some deeper conflict between dependencies, and conda's hints are just not helping much here.  I'm moving this issue to the conda repo to see if they can help you figure out a way to debug this.

Note: I have used conda 4.3.11 for this.  It has better hints than earlier versions, but apparently still confusing sometimes.




Discussion:
cc @mcg1969 - any tips on getting better hints here?
I'm afraid we just need to keep working on improving the conflict hints. I'm not sure I have any short-term advice. It's really an extended R&D struggle.
I know that there has been some effort before to create better hints: https://github.com/rmcgibbo/conda-hint
But as far as I remember it requires at least Python 3.5 to run.
I see that funcsigs is the conflicting package. Note that "py36" is not yet a valid selector. Is there any sane way to select this package only for python version smaller than 3.6 with the selector feature?
```[py<36]``` should work?
On 02/17/2017 04:20 PM, Mike Sarahan wrote:
> |[py>35]| should work?
yes, thanks you!


Just a question, why does conda-build not use the logic of conda(-env) to create the test env in the first place in order to get the same behaviour (hint generation)?
Speed and spec format.

Conda build allows specs according to https://conda.io/docs/spec.html#package-match-specifications

The command line interface and the newer conda API, which is just a tiny bit lower level than the CLI, do not support those match specs.  Part of the reason is that characters like > and < get misinterpreted by the terminal. Even if you put them on quotes, though, the CLI still doesn't support them.

Speed is also drastically slower, because conda build is able to control caching more intelligently.

I'm surprised that the hints are different, but it probably comes down to the input specs being different. I had to alter them in order to make them valid CLI input. Ultimately conda's solver is being called similarly in both conda build and conda.

We're working on a conda API that will unify behavior while maintaining the speed advantages.

Resolving patch:
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -6,14 +6,16 @@
 import json
 from logging import getLogger
 from os.path import dirname, join
+from random import random
 import re
+from time import sleep
 
 from .linked_data import delete_linked_data, get_python_version_for_prefix, load_linked_data
 from .portability import _PaddingError, update_prefix
 from .._vendor.auxlib.compat import with_metaclass
 from .._vendor.auxlib.ish import dals
 from ..base.context import context
-from ..common.compat import iteritems, on_win
+from ..common.compat import iteritems, on_win, range
 from ..common.path import (get_bin_directory_short_path, get_leaf_directories,
                            get_python_noarch_target_path, get_python_short_path,
                            parse_entry_point_def, preferred_env_to_prefix, pyc_path, url_to_path,
@@ -133,6 +135,7 @@ def source_full_path(self):
 
 
 class LinkPathAction(CreateInPrefixPathAction):
+    _verify_max_backoff_reached = False
 
     @classmethod
     def create_file_link_actions(cls, transaction_context, package_info, target_prefix,
@@ -213,13 +216,28 @@ def __init__(self, transaction_context, package_info,
     def verify(self):
         # TODO: consider checking hashsums
         if self.link_type != LinkType.directory and not lexists(self.source_full_path):
-            return CondaVerificationError(dals("""
-            The package for %s located at %s
-            appears to be corrupted. The path '%s'
-            specified in the package manifest cannot be found.
-            """ % (self.package_info.index_json_record.name,
-                   self.package_info.extracted_package_dir,
-                   self.source_short_path)))
+            # This backoff loop is added because of some weird race condition conda-build
+            # experiences. Would be nice at some point to get to the bottom of why it happens.
+
+            # with max_retries = 2, max total time ~= 0.4 sec
+            # with max_retries = 6, max total time ~= 6.5 sec
+            max_retries = 2 if LinkPathAction._verify_max_backoff_reached else 6
+            for n in range(max_retries):
+                sleep_time = ((2 ** n) + random()) * 0.1
+                log.trace("retrying lexists(%s) in %g sec", self.source_full_path, sleep_time)
+                sleep(sleep_time)
+                if lexists(self.source_full_path):
+                    break
+            else:
+                # only run the 6.5 second backoff time once
+                LinkPathAction._verify_max_backoff_reached = True
+                return CondaVerificationError(dals("""
+                The package for %s located at %s
+                appears to be corrupted. The path '%s'
+                specified in the package manifest cannot be found.
+                """ % (self.package_info.index_json_record.name,
+                       self.package_info.extracted_package_dir,
+                       self.source_short_path)))
         self._verified = True
 
     def execute(self):


Test patch: