Repository: conda/conda-build

Issue:
3.0.10 raises "Unsatisfiable dependencies for platform osx-64"
I am attempting to build trilinos (the bane of my existence), which requires the numpy C API, but my build fails at test with:

```
TEST START: /Users/guyer/anaconda/conda-bld/osx-64/trilinos-12.10.1-py27hadd81b6_2.tar.bz2
Renaming work directory,  /Users/guyer/anaconda/conda-bld/trilinos_1503630186988/work  to  /Users/guyer/anaconda/conda-bld/trilinos_1503630186988/work_moved_trilinos-12.10.1-py27hadd81b6_2
Traceback (most recent call last):
  File "/Users/guyer/anaconda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 387, in main
    execute(sys.argv[1:])
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 378, in execute
    noverify=args.no_verify)
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/api.py", line 184, in build
    need_source_download=need_source_download, config=config, variants=variants)
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/build.py", line 1739, in build_tree
    test(pkg, config=metadata.config)
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/build.py", line 1518, in test
    channel_urls=tuple(metadata.config.channel_urls))
  File "/Users/guyer/anaconda/lib/python2.7/site-packages/conda_build/environ.py", line 688, in get_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform osx-64: ['trilinos', 'numpy']
```

This seems similar to #2191 and #2197, but both of those issues are nominally fixed and I am using `conda-build` variants for numpy. 

This recipe successfully built with `conda-build 2.1.1` (before I foolishly heeded the warning to update).

I invoke
```
conda build trilinos --channel guyer --channel conda-forge
```
and my recipes are at https://github.com/guyer/conda-recipes/tree/trilinos_upgrade/trilinos

```
Current conda install:

               platform : osx-64
          conda version : 4.3.25
       conda is private : False
      conda-env version : 4.3.25
    conda-build version : 3.0.10
         python version : 2.7.13.final.0
       requests version : 2.14.2
       root environment : /Users/guyer/anaconda  (writable)
    default environment : /Users/guyer/anaconda
       envs directories : /Users/guyer/anaconda/envs
                          /Users/guyer/.conda/envs
          package cache : /Users/guyer/anaconda/pkgs
                          /Users/guyer/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/osx-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /Users/guyer/.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.25 requests/2.14.2 CPython/2.7.13 Darwin/16.6.0 OSX/10.12.5    
                UID:GID : 17723:17723
```


Discussion:
Sorry conda's hints aren't more helpful.  It looks to me like mpich is missing:

```
[root@19638a023b77 conda-recipes]# conda render trilinos
Returning non-final recipe for trilinos-12.10.1-py36_2; one or more dependencies was unsatisfiable:
Build: mpich
Host: None
package:
    name: trilinos
    version: 12.10.1
source:
    fn: trilinos-12.10.1-Source.tar.bz2
    md5: 1e14df93afaddb12cf840da56c5132ce
    url: https://trilinos.org/oldsite/download/files/trilinos-12.10.1-Source.tar.bz2
build:
    number: '2'
requirements:
    build:
        - python 3.6.*
        - numpy 1.10
        - nomkl
        - swig ==3.0.10
        - cmake
        - mpich
        - openblas
        - boost
    run:
        - openblas
        - numpy 1.10
        - python
        - mpich
        - nomkl
test:
    imports:
        - PyTrilinos
        - PyTrilinos._Teuchos
        - PyTrilinos.ML
extra:
    copy_test_source_files: true
    final: true
```

If you include conda-forge as a channel, it should work:

```
[root@19638a023b77 conda-recipes]# conda render trilinos -c conda-forge
package:
    name: trilinos
    version: 12.10.1
source:
    fn: trilinos-12.10.1-Source.tar.bz2
    md5: 1e14df93afaddb12cf840da56c5132ce
    url: https://trilinos.org/oldsite/download/files/trilinos-12.10.1-Source.tar.bz2
build:
    number: '2'
requirements:
    build:
        - bzip2 1.0.6 1
        - openblas 0.2.19 2
        - openssl 1.0.2l 0
        - libssh2 1.8.0 1
        - curl 7.54.1 0
        - sqlite 3.13.0 1
        - libgfortran 3.0.0 1
        - readline 6.2 0
        - krb5 1.14.2 0
        - pcre 8.39 0
        - ncurses 5.9 10
        - cmake 3.9.1 0
        - boost-cpp 1.64.0 1
        - libgcc 7.1.0 h9c9919c_0
        - python 3.6.2 0
        - icu 58.1 1
        - numpy 1.10.4 py36_blas_openblas_205
        - mpich 3.2 4
        - expat 2.2.1 0
        - tk 8.5.19 2
        - libgcc_linux-64 7.1.0 h12e1119_0
        - libgfortran_linux-64 7.1.0 ha5e90dd_0
        - libuv 1.11.0 0
        - libstdcxx_linux-64 7.1.0 h4f7dd76_0
        - nomkl 1.0 0
        - rhash 1.3.4 0
        - ca-certificates 2017.7.27.1 0
        - boost 1.64.0 py36_4
        - swig 3.0.10 2
        - blas 1.1 openblas
        - xz 5.2.3 0
        - zlib 1.2.11 0
    run:
        - mpich
        - python >=3.6,<3.7.0a0
        - openblas
        - numpy 1.10
        - nomkl
test:
    imports:
        - PyTrilinos
        - PyTrilinos._Teuchos
        - PyTrilinos.ML
extra:
    copy_test_source_files: true
    final: true
```
Thanks for the reply and the hint about `conda render`. Unfortunately, I already pass `--channel conda-forge`.

The package builds, but it fails to test. `mpich` is required for both.

FWIW, this is what I get. 
```

package:
    name: trilinos
    version: 12.10.1
source:
    fn: trilinos-12.10.1-Source.tar.bz2
    md5: 1e14df93afaddb12cf840da56c5132ce
    url: https://trilinos.org/oldsite/download/files/trilinos-12.10.1-Source.tar.bz2
build:
    number: '2'
requirements:
    build:
        - zlib 1.2.11 0
        - krb5 1.14.2 0
        - pip 9.0.1 py27_0
        - tk 8.5.19 2
        - mpich 3.2 4
        - swig 3.0.10 2
        - python 2.7.13 1
        - openblas 0.2.20 1
        - nomkl 1.0 0
        - bzip2 1.0.6 1
        - libgfortran 3.0.0 0
        - rhash 1.3.4 0
        - cmake 3.9.1 0
        - wheel 0.29.0 py27_0
        - xz 5.2.3 0
        - openssl 1.0.2l 0
        - pcre 8.39 0
        - ncurses 5.9 10
        - libssh2 1.8.0 1
        - boost 1.64.0 py27_4
        - icu 58.1 1
        - curl 7.54.1 0
        - readline 6.2 0
        - libuv 1.11.0 0
        - setuptools 36.2.2 py27_0
        - sqlite 3.13.0 1
        - certifi 2017.7.27.1 py27_0
        - ca-certificates 2017.7.27.1 0
        - boost-cpp 1.64.0 1
        - expat 2.2.1 0
        - numpy 1.10.4 py27_nomkl_2
    run:
        - mpich
        - numpy 1.10
        - python >=2.7,<2.8.0a0
        - nomkl
        - openblas
test:
    imports:
        - PyTrilinos
        - PyTrilinos._Teuchos
        - PyTrilinos.ML
extra:
    copy_test_source_files: true
    final: true
```

I don't know if it matters that I'm building with python 2.7 (PyTrilinos isn't ready for py3k just yet).
Yes, sorry.  This appears to be a bug.  Here's a couple of workarounds:

The right way, which you should do anyway:

```
requirements:
  build:
    - python
    - numpy {{ numpy }}
    - nomkl
    - swig ==3.0.10
    - cmake
    - mpich
    - openblas
    - boost
  run:
    - python
    - {{ pin_compatible('numpy') }}
    - nomkl
    - mpich
    - openblas
```

This takes advantage of the fact that numpy is forward-compatible (a recent discovery in our community).  So you'd build with 1.10, but you could install with any version of numpy, >1.10,<2.0a0

You could disable the upper bound completely with ```- {{ pin_compatible('numpy', max_pin=None) }}```

The crappy hack:

```
requirements:
  build:
    - python
    - numpy {{ numpy }}
    - nomkl
    - swig ==3.0.10
    - cmake
    - mpich
    - openblas
    - boost
  run:
    - python
    - numpy {{ numpy }}.*
    - nomkl
    - mpich
    - openblas
```

The .* is what is tripping up conda.  We add that in the build deps, but apparently not in the run deps...  I don't think this is a new bug, but I think that people knew to add the .* before, while the use of variants makes things more complicated, and people forgot.  I will add the automatic addition in the run deps.
Thanks for the fix and the hack. Thanks to trilinos' ludicrous build time, I'll let you know tomorrow if it worked.
That worked, thanks! (specifically, the `{{ pin_compatible('numpy', max_pin=None) }}`. Trilinos takes hours to build, so I didn't try the other variants).

Resolving patch:
diff --git a/conda_build/render.py b/conda_build/render.py
--- a/conda_build/render.py
+++ b/conda_build/render.py
@@ -327,6 +327,7 @@ def finalize_metadata(m, permit_unsatisfiable_variants=False):
     full_build_dep_versions = {dep.split()[0]: " ".join(dep.split()[1:]) for dep in full_build_deps}
     versioned_run_deps = [get_pin_from_build(m, dep, full_build_dep_versions) for dep in run_deps]
     versioned_run_deps.extend(extra_run_specs)
+    versioned_run_deps = [utils.ensure_valid_spec(spec, warn=True) for spec in versioned_run_deps]
 
     for _env, values in (('build', build_deps), ('host', host_deps), ('run', versioned_run_deps)):
         if values:
@@ -340,6 +341,8 @@ def finalize_metadata(m, permit_unsatisfiable_variants=False):
     if test_deps:
         versioned_test_deps = list({get_pin_from_build(m, dep, full_build_dep_versions)
                                     for dep in test_deps})
+        versioned_test_deps = [utils.ensure_valid_spec(spec, warn=True)
+                               for spec in versioned_test_deps]
         rendered_metadata.meta['test']['requires'] = versioned_test_deps
     rendered_metadata.meta['extra']['copy_test_source_files'] = m.config.copy_test_source_files
 
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -1068,6 +1068,13 @@ def filter(self, record):
 warning_error_stderr_filter = GreaterThanFilter(logging.INFO)
 
 
+def reset_deduplicator():
+    """Most of the time, we want the deduplication.  There are some cases (tests especially)
+    where we want to be able to control the duplication."""
+    global dedupe_filter
+    dedupe_filter = DuplicateFilter()
+
+
 def get_logger(name, level=logging.INFO, dedupe=True, add_stdout_stderr_handlers=True):
     config_file = cc_conda_build.get('log_config_file')
     # by loading config file here, and then only adding handlers later, people
@@ -1233,11 +1240,11 @@ def sort_list_in_nested_structure(dictionary, omissions=''):
 # if you are seeing mysterious unsatisfiable errors, with the package you're building being the
 #    unsatisfiable part, then you probably need to update this regex.
 
-spec_needing_star_re = re.compile("([0-9a-zA-Z\.\-\_]+)\s+([0-9a-zA-Z\.\+\_]+)(\s+[0-9a-zA-Z\.\_]+)?")  # NOQA
+spec_needing_star_re = re.compile(r"([\w\d\.\-\_]+)\s+((?<![><=])[\w\d\.\-\_]+?(?!\*))(\s+[\w\d\.\_]+)?$")  # NOQA
 spec_ver_needing_star_re = re.compile("^([0-9a-zA-Z\.]+)$")
 
 
-def ensure_valid_spec(spec):
+def ensure_valid_spec(spec, warn=False):
     if isinstance(spec, MatchSpec):
         if (hasattr(spec, 'version') and spec.version and
                 spec_ver_needing_star_re.match(str(spec.version))):
@@ -1251,6 +1258,15 @@ def ensure_valid_spec(spec):
                 spec = spec_needing_star_re.sub(r"\1 \2", spec)
             else:
                 if "*" not in spec:
+                    if match.group(1) != 'python' and warn:
+                        log = get_logger(__name__)
+                        log.warn("Adding .* to spec '{}' to ensure satisfiability.  Please "
+                                 "consider putting {{{{ var_name }}}}.* or some relational "
+                                 "operator (>/</>=/<=) on this spec in meta.yaml, or if req is "
+                                 "also a build req, using {{{{ pin_compatible() }}}} jinja2 "
+                                 "function instead.  See "
+                "https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level"  # NOQA
+                        .format(spec))
                     spec = spec_needing_star_re.sub(r"\1 \2.*", spec)
     return spec
 


Test patch:
diff --git a/tests/test-recipes/variants/14_variant_in_run_and_test/conda_build_config.yaml b/tests/test-recipes/variants/14_variant_in_run_and_test/conda_build_config.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/variants/14_variant_in_run_and_test/conda_build_config.yaml
@@ -0,0 +1,8 @@
+click:
+  - 6
+pytest:
+  - 3.2
+pytest_cov:
+  - 2.3
+pytest_mock:
+  - 1.6
diff --git a/tests/test-recipes/variants/14_variant_in_run_and_test/meta.yaml b/tests/test-recipes/variants/14_variant_in_run_and_test/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/variants/14_variant_in_run_and_test/meta.yaml
@@ -0,0 +1,21 @@
+# in this recipe, both the click variant and the pytest variant are underspecified for conda.
+#   In conda-build <3.0.15, this recipe will fail, with unsatisfiable dependencies.  In 3.0.15,
+#   we add .* to underspecified variant versions in run and test deps, but we also warn people
+#   that it would be better if they used better specs (provide .* themselves, or use >/< expressions
+
+package:
+  name: test_variant_in_run_and_test
+  version: 1.0
+
+requirements:
+  build:
+    - python
+  run:
+    - click  {{ click }}
+
+test:
+  requires:
+    - pytest  {{ pytest }}
+    # neither of these two should have warning messages
+    - pytest-cov  {{ pytest_cov }}.*
+    - pytest-mock  >={{ pytest_mock }}
diff --git a/tests/test_api_build.py b/tests/test_api_build.py
--- a/tests/test_api_build.py
+++ b/tests/test_api_build.py
@@ -892,8 +892,8 @@ def test_run_exports(testing_metadata, testing_config, testing_workdir):
     testing_metadata.meta['requirements']['build'] = ['test_has_run_exports']
     api.output_yaml(testing_metadata, 'meta.yaml')
     m = api.render(testing_workdir, config=testing_config)[0][0]
-    assert 'strong_pinned_package 1.0' in m.meta['requirements']['run']
-    assert 'weak_pinned_package 1.0' in m.meta['requirements']['run']
+    assert 'strong_pinned_package 1.0.*' in m.meta['requirements']['run']
+    assert 'weak_pinned_package 1.0.*' in m.meta['requirements']['run']
 
     #    2. host present.  Use run_exports from host, ignore 'weak' ones from build.  All are
     #           weak by default.
@@ -901,7 +901,7 @@ def test_run_exports(testing_metadata, testing_config, testing_workdir):
     testing_metadata.meta['requirements']['host'] = ['python']
     api.output_yaml(testing_metadata, 'meta.yaml')
     m = api.render(testing_workdir, config=testing_config)[0][0]
-    assert 'weak_pinned_package 2.0' not in m.meta['requirements']['run']
+    assert 'weak_pinned_package 2.0.*' not in m.meta['requirements']['run']
 
     #    3. host present, and deps in build have "strong" run_exports section.  use host, add
     #           in "strong" from build.
@@ -909,10 +909,11 @@ def test_run_exports(testing_metadata, testing_config, testing_workdir):
     testing_metadata.meta['requirements']['host'] = ['test_has_run_exports_implicit_weak']
     api.output_yaml(testing_metadata, 'meta.yaml')
     m = api.render(testing_workdir, config=testing_config)[0][0]
-    assert 'strong_pinned_package 1.0' in m.meta['requirements']['run']
+    assert 'strong_pinned_package 1.0.*' in m.meta['requirements']['run']
     # weak one from test_has_run_exports should be excluded, since it is a build dep
-    assert 'weak_pinned_package 1.0' not in m.meta['requirements']['run']
-    assert 'weak_pinned_package 2.0' in m.meta['requirements']['run']
+    assert 'weak_pinned_package 1.0.*' not in m.meta['requirements']['run']
+    assert 'weak_pinned_package 2.0.*' in m.meta['requirements']['run']
+
 
 @pytest.mark.serial
 def test_ignore_run_exports(testing_metadata, testing_config):
diff --git a/tests/test_source.py b/tests/test_source.py
--- a/tests/test_source.py
+++ b/tests/test_source.py
@@ -7,6 +7,7 @@
 from conda_build import source
 from conda_build.conda_interface import TemporaryDirectory
 from conda_build.source import download_to_cache
+from conda_build.utils import reset_deduplicator
 from .utils import thisdir
 
 
@@ -123,6 +124,7 @@ def test_hoist_different_name(testing_workdir):
     assert not os.path.isdir(testdir)
 
 
+@pytest.mark.serial
 def test_append_hash_to_fn(testing_metadata, caplog):
     relative_zip = 'testfn.zip'
     assert source.append_hash_to_fn(relative_zip, '123') == 'testfn_123.zip'
@@ -136,9 +138,14 @@ def test_append_hash_to_fn(testing_metadata, caplog):
     assert source.append_hash_to_fn(absolute_win_zip, '123') == 'C:\\abc\\testfn_123.zip'
     absolute_win_tar_gz = 'C:\\abc\\testfn.tar.gz'
     assert source.append_hash_to_fn(absolute_win_tar_gz, '123') == 'C:\\abc\\testfn_123.tar.gz'
+    relative_whl = 'setuptools-36.4.0-py2.py3-none-any.whl'
+    assert source.append_hash_to_fn(relative_whl, '123') == 'setuptools-36.4.0-py2.py3-none-any_123.whl'
 
     testing_metadata.meta['source'] = [
         {'folder': 'f1', 'url': os.path.join(thisdir, 'archives', 'a.tar.bz2')}]
+    reset_deduplicator()
     source.provide(testing_metadata)
-    # would be nice if this worked, but broken on Travis.  Works locally.  Catchlog 1.2.2
-    # assert caplog.text.count('No hash (md5, sha1, sha256) provided.') == 1
+    # TODO: Can't seem to get this to work.  Test passes when run by itself, but fails when run in whole
+    #     serial suite.  Some residual state, somehow.  I suspect the deduplicator logic with the logger,
+    #     but attempts to reset it have not been successful.
+    # assert any("No hash (md5, sha1, sha256) provided." in rec.message for rec in caplog.records)
diff --git a/tests/test_subpackages.py b/tests/test_subpackages.py
--- a/tests/test_subpackages.py
+++ b/tests/test_subpackages.py
@@ -73,7 +73,7 @@ def test_run_exports_in_subpackage(testing_metadata):
     p2.meta['requirements']['build'] = ['has_run_exports']
     p2.original_meta = p2.meta.copy()
     p2_final = finalize_metadata(p2)
-    assert 'bzip2 1.0' in p2_final.meta['requirements']['run']
+    assert 'bzip2 1.0.*' in p2_final.meta['requirements']['run']
 
 
 # @pytest.mark.serial
@@ -241,9 +241,10 @@ def test_subpackage_hash_inputs(testing_config):
             assert utils.package_has_file(out, 'info/recipe/build.sh')
 
 
+@pytest.mark.serial
 def test_overlapping_files(testing_config, caplog):
     recipe_dir = os.path.join(subpackage_dir, '_overlapping_files')
+    utils.reset_deduplicator()
     outputs = api.build(recipe_dir, config=testing_config)
     assert len(outputs) == 3
-    # would be nice if this worked, but broken on Travis.  Works locally.  Catchlog 1.2.2
-    # assert caplog.text().count('Exact overlap') == 2
+    assert sum(int("Exact overlap" in rec.message) for rec in caplog.records) == 1
diff --git a/tests/test_variants.py b/tests/test_variants.py
--- a/tests/test_variants.py
+++ b/tests/test_variants.py
@@ -178,3 +178,15 @@ def test_variant_input_with_zip_keys_keeps_zip_keys_list():
                  'MACOSX_DEPLOYMENT_TARGET': '10.9', 'CONDA_BUILD_SYSROOT': '/opt/MacOSX10.9.sdk'}]
     variant_list = variants.dict_of_lists_to_list_of_dicts(variants_)
     assert len(variant_list) == 1
+
+
+@pytest.mark.serial
+def test_ensure_valid_spec_on_run_and_test(testing_workdir, testing_config, caplog):
+    recipe = os.path.join(recipe_dir, '14_variant_in_run_and_test')
+    api.render(recipe, config=testing_config)
+
+    text = caplog.text
+    assert "Adding .* to spec 'click  6'" in text
+    assert "Adding .* to spec 'pytest  3.2'" in text
+    assert "Adding .* to spec 'pytest-cov  2.3'" not in text
+    assert "Adding .* to spec 'pytest-mock  1.6'" not in text