Repository: conda/conda-build

Issue:
[conda-build] issue with package pinning ?
Hi,

Could you please help ?

We are having a strange issue related to conda-build and package pinning:

###### This is the error:

```
raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.
(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"
```

###### Here is the complete output:

```
/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/environ.py:418: UserWarning: The environment variable 'DOCSERVER' is being passed through with value <website>.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/environ.py:418: UserWarning: The environment variable 'DOCSERVER' is being passed through with value website.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
Traceback (most recent call last):
  File "/local/builds/beat/beat.editor/miniconda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 438, in main
    execute(sys.argv[1:])
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 408, in execute
    action(recipe, config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 348, in test_action
    return api.test(recipe, move_broken=False, config=config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/api.py", line 227, in test
    stats=stats)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1835, in test
    config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1805, in construct_metadata_for_test
    m, hash_input = _construct_metadata_for_test_from_recipe(recipedir_or_package, config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1691, in _construct_metadata_for_test_from_recipe
    metadata = finalize_metadata(metadata)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/render.py", line 457, in finalize_metadata
    output = m.get_rendered_output(m.name())
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 2047, in get_rendered_output
    permit_undefined_jinja=permit_undefined_jinja):
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 2037, in get_rendered_outputs_section
    self.parse_until_resolved()
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 980, in parse_until_resolved
    bypass_env_check=bypass_env_check)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 924, in parse_again
    bypass_env_check=bypass_env_check),
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 1494, in _get_contents
    rendered = template.render(environment=env)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/local/builds/beat/beat.editor/conda/meta.yaml", line 11, in top-level template code
    - {{ pin_subpackage(name) }}
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/jinja_context.py", line 313, in pin_subpackage
    raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.
(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"
```

Thanks for your help


Discussion:
Please post the information requested in the template. We also need a minimal reproducible test case.
@msarahan thanks for your quick answer.

Here some extra information:

So basically the system is running on a CI build on a gitlab repository:

It seems to fail when executing this line:
`(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"
`

The example of this failed job can be found here: https://gitlab.idiap.ch/beat/beat.editor/-/jobs/149048

It uses this conda build configuration file:

```
macos_min_version:
  - 10.9
macos_machine:
  - x86_64-apple-darwin13.4.0
MACOSX_DEPLOYMENT_TARGET:
  - 10.9
CONDA_BUILD_SYSROOT:            # [osx]
  - /opt/MacOSX10.9.sdk         # [osx]
# makes autotools verbose
VERBOSE_AT:
  - V=1
# makes cmake verbose
VERBOSE_CM:
  - VERBOSE=1

# the blas implementations that we build against
blas_impl:
  - mkl

pin_run_as_build:
  libboost:
    max_pin: x.x.x
  py_boost:
    max_pin: x.x.x
  boost:
    max_pin: x.x.x

## the dependencies that we build against multiple versions
python:
  - 3.6

zip_keys:
  -                             # [win]
    - vc                        # [win]
    - c_compiler                # [win]
    - cxx_compiler              # [win]
    - fortran_compiler_version  # [win]
    - python                    # [win]


# Here is the version of dependencies are used when building packages (build
# and host requirements). We keep a list of **all of them** here to make sure
# everything goes as expected in our conda build process. For the version of
# packages that are used for testing packages, see the recipe of bob-devel.
# The version here do not necessarily match the versions in bob-devel.

# This version of bob-devel will be used at test time of packages:
bob_devel:
  - 2018.09.24

# This version of beat-devel will be used at test time of packages. Notice it
# uses bob-devel and should have a version that is greater or equal its value
beat_devel:
  - 2018.09.24

# The build time only dependencies (build requirements).
# Updating these to the latest version all the time is OK and a good idea.
# These versions should match the versions inside bob-devel as well (if they
# overlap) so update them in both places.
cmake:
  - 3.12.2
pkg_config:
  - 0.29.2
cython:
  - 0.28.1

# The dependencies that we link against (host requirements).
# Ideally we want to build against the oldest possible version of packages when
# we are linking against them. It is best to keep this in sync with:
# https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml
numpy:
  - 1.14.5
boost:
  - 1.65.1
cyvlfeat:
  - 0.4.6
ffmpeg:
  - 3.4
freetype:
  - 2.8
giflib:
  - 5.1.4
hdf5:
  - 1.10.1
jpeg:
  - 9b
libblitz:
  - 1.0.1
libmatio:
  - 1.5.11
libogg:
  - 1.3.2
libpng:
  - 1.6.32
libsvm:
  - 3.22
libtiff:
  - 4.0.9
mkl:
  - 2018.0.3
openfst:
  - 1.6.1
sox:
  - 14.4.2
speex:
  - 1.2.0
speexdsp:
  - 1.2rc3
sqlite:
  - 3.20.1
vlfeat:
  - 0.9.21
xz:
  - 5.2.3
zlib:
  - 1.2.11

# The dependencies that are needed for runtime only (run requirements).
# These versions **should** match the versions inside bob-devel recipe.
caffe:
  - 1.0
click:
  - 6.7
click_plugins:
  - 1.0.3
coverage:
  - 4.5.1
dlib:
  - 19.7
docopt:
  - 0.6.2
jinja2:
  - 2.10
kaldi:
  - 2017.03.13
matplotlib:
  - 2.2.3
menpo:
  - 0.8.1
menpofit:
  - 0.5.0
mne:
  - 0.15.2
mr_developer:
  - 1.38
nose:
  - 1.3.7
opencv:
  - 3.3.1
pillow:
  - 5.2.0
pyedflib:
  - 0.1.11
pytorch:
  - 0.4.1
pyyaml:
  - 3.13
requests:
  - 2.19.1
schema:
  - 0.6.7
scikit_image:
  - 0.19.2
scipy:
  - 1.1.0
setuptools:
  - 40.2.0
six:
  - 1.11.0
sphinx:
  - 1.7.9
sphinx_rtd_theme:
  - 0.4.1
sqlalchemy:
  - 1.2.11
tabulate:
  - 0.8.2
tensorflow:
  - 1.9.0
torchvision:
  - 0.2.1
zc_buildout:
  - 2.11.2
zc_recipe_egg:
  - 2.0.5

# The dependencies that are needed for runtime only (run requirements) of BEAT
# packages. These versions **should** match the versions inside beat-devel
# recipe.
docker_py:
  - 3.2.1
jsonschema:
  - 2.6.0
oset:
  - 0.1.3
python_graphviz:
  - 0.8.2
pyzmq:
  - 17.0.0
simplejson:
  - 3.13.2
termcolor:
  - 1.1.0
sphinxcontrib_programoutput:
  - 0.11
nodejs:
  - 8.9.3
flask:
  - 0.12.2
flask_cors:
  - 3.0.3
flask_restful:
  - 0.3.6
```


For more information: 
This is the .gitlab.yml file that runs on our CI to launch the job

```
# Definition of global variables (all stages)
variables:
  CONDA_ROOT: "${CI_PROJECT_DIR}/miniconda"


# Definition of our build pipeline order
stages:
  - build
  - browser-tests
  - deploy
  - pypi


# Build targets
.build_template: &build_job
  stage: build
  before_script:
    - mkdir _ci
    - curl --silent "https://gitlab.idiap.ch/bob/bob.admin/raw/master/gitlab/install.sh" > _ci/install.sh
    - chmod 755 _ci/install.sh
    - ./_ci/install.sh _ci master #installs ci support scripts
    - ./_ci/before_build.sh
  script:
    - ./_ci/build.sh
  after_script:
    - ./_ci/after_build.sh
  cache: &build_caches
    paths:
      - miniconda.sh
      - ${CONDA_ROOT}/pkgs/*.tar.bz2
      - ${CONDA_ROOT}/pkgs/urls.txt


.build_linux_template: &linux_build_job
  <<: *build_job
  tags:
    - docker
  image: continuumio/conda-concourse-ci
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - ${CONDA_ROOT}/conda-bld/linux-64/*.tar.bz2
  cache:
    <<: *build_caches
    key: "linux-cache"


.build_macosx_template: &macosx_build_job
  <<: *build_job
  tags:
    - macosx
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - ${CONDA_ROOT}/conda-bld/osx-64/*.tar.bz2
  cache:
    <<: *build_caches
    key: "macosx-cache"


build_linux_27:
  <<: *linux_build_job
  variables:
    PYTHON_VERSION: "2.7"


build_linux_36:
  <<: *linux_build_job
  variables:
    PYTHON_VERSION: "3.6"
    BUILD_EGG: "true"
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - dist/*.zip
      - sphinx
      - ${CONDA_ROOT}/conda-bld/linux-64/*.tar.bz2


build_macosx_27:
  <<: *macosx_build_job
  variables:
    PYTHON_VERSION: "2.7"


build_macosx_36:
  <<: *macosx_build_job
  variables:
    PYTHON_VERSION: "3.6"


# Docker host based testing (must be run inside dind or docker-enabled host)
.browser_test_linux_template: &linux_browser_test_job
  stage: browser-tests
  image: docker.idiap.ch/beat/ci.env.editor:0.0.1r0
  before_script:
    # safe keep artifacts as before_build.sh will erase those...
    - apt-get update > /dev/null
    - apt-get install -y locales > /dev/null
    - echo "en_US UTF-8" > /etc/locale.gen
    - locale-gen en_US.UTF-8
    - export LANG=en_US.UTF-8
    - export LANGUAGE=en_US:en
    - export LC_ALL=en_US.UTF-8
    - mv ${CONDA_ROOT}/conda-bld .
    - ./_ci/install.sh _ci master #updates ci support scripts
    - ./_ci/before_build.sh
    - mv conda-bld ${CONDA_ROOT}
  script:
    - export BEAT_BROWSER_TESTS=true
    - BOB_TEST_ONLY=true ./_ci/build.sh
  after_script:
    - ./_ci/after_build.sh


browser_linux_36:
  <<: *linux_browser_test_job
  variables:
    PYTHON_VERSION: "3.6"
  dependencies:
    - build_linux_36
  tags:
    - docker


# Deploy targets
.deploy_template: &deploy_job
  stage: deploy
  before_script:
    - ./_ci/install.sh _ci master #updates ci support scripts
  script:
    - ./_ci/deploy.sh
  dependencies:
    - build_linux_27
    - build_linux_36
    - build_macosx_27
    - build_macosx_36
  tags:
    - deployer


deploy_beta:
  <<: *deploy_job
  environment: beta
  only:
    - master


deploy_stable:
  <<: *deploy_job
  environment: stable
  only:
    - /^v\d+\.\d+\.\d+([abc]\d*)?$/  # PEP-440 compliant version (tags)
  except:
    - branches


pypi:
  stage: pypi
  environment: pypi
  only:
    - /^v\d+\.\d+\.\d+([abc]\d*)?$/  # PEP-440 compliant version (tags)
  except:
    - branches
  before_script:
    - ./_ci/install.sh _ci master #updates ci support scripts
  script:
    - ./_ci/pypi.sh
  dependencies:
    - build_linux_36
  tags:
    - deployer
```

@msarahan, @183amir: Here is a minimal example that reproduces the issue, using git@github.com:AnacondaRecipes/zlib-feedstock.git:

```sh
$ git clone git@github.com:AnacondaRecipes/zlib-feedstock.git
$ cd zlib-feedstock
$ conda update -n base conda conda-build
# just to ensure we've got the lastest
$ conda build --no-test recipe
# works w/o issues
$ conda build --test recipe
TEST START: recipe
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults

Leaving build/test directories:
  Work:
 /idiap/user/aanjos/conda/conda-bld/work
  Test:
 /idiap/user/aanjos/conda/conda-bld/test_tmp
Leaving build/test environments:
  Test:
source activate  /idiap/user/aanjos/conda/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_
  Build:
source activate  /idiap/user/aanjos/conda/conda-bld/_build_env


Traceback (most recent call last):
  File "/idiap/user/aanjos/conda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 438, in main
    execute(sys.argv[1:])
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 408, in execute
    action(recipe, config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 348, in test_action
    return api.test(recipe, move_broken=False, config=config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/api.py", line 227, in test
    stats=stats)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1835, in test
    config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1805, in construct_metadata_for_test
    m, hash_input = _construct_metadata_for_test_from_recipe(recipedir_or_package, config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1691, in _construct_metadata_for_test_from_recipe
    metadata = finalize_metadata(metadata)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/render.py", line 457, in finalize_metadata
    output = m.get_rendered_output(m.name())
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 2047, in get_rendered_output
    permit_undefined_jinja=permit_undefined_jinja):
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 2037, in get_rendered_outputs_section
    self.parse_until_resolved()
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 980, in parse_until_resolved
    bypass_env_check=bypass_env_check)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 924, in parse_again
    bypass_env_check=bypass_env_check),
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1494, in _get_contents
    rendered = template.render(environment=env)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/remote/idiap.svm/user.active/aanjos/work/bob/zlib-feedstock/recipe/meta.yaml", line 15, in top-level template code
    # mostly OK, but some scary symbol removal.  Let's try for trusting them.
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/jinja_context.py", line 313, in pin_subpackage
    raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.
```

The problem seems to happen when you test only (and not build the package).  This is currently breaking a lot of our infrastructure and issuing (unnecessary) rebuilds, as we cannot just test already built packages.
This is probably a regression issue. This used to work for us before.
I checked - conda-build 3.10.3 is the last to work correctly in this regard.  Changes in 3.10.4 likely created the bug.  All versions following that have it.
I will fix this, but I am deprecating this mode of operation.  Pass tarball paths to conda-build for testing instead.

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -1706,13 +1706,17 @@ def _construct_metadata_for_test_from_recipe(recipe_dir, config):
     config.need_cleanup = False
     config.recipe_dir = None
     hash_input = {}
-    metadata = render_recipe(recipe_dir, config=config, reset_build_id=False)[0][0]
+    metadata = expand_outputs(render_recipe(recipe_dir, config=config, reset_build_id=False))[0][1]
+    log = utils.get_logger(__name__)
+    log.warn("Testing based on recipes is deprecated as of conda-build 3.16.0.  Please adjust "
+             "your code to pass your desired conda package to test instead.")
 
     utils.rm_rf(metadata.config.test_dir)
 
     if metadata.meta.get('test', {}).get('source_files'):
         if not metadata.source_provided:
             try_download(metadata, no_download_source=False)
+
     if not metadata.final:
         metadata = finalize_metadata(metadata)
     return metadata, hash_input
@@ -2082,7 +2086,7 @@ def test(recipedir_or_package_or_metadata, config, stats, move_broken=True):
         test_stats = {}
         utils.check_call_env(cmd, env=env, cwd=metadata.config.test_dir, stats=test_stats)
         log_stats(test_stats, "testing {}".format(metadata.name()))
-        if stats is not None:
+        if stats is not None and metadata.config.variants:
             stats[stats_key(metadata, 'test_{}'.format(metadata.name()))] = test_stats
     except subprocess.CalledProcessError:
         tests_failed(metadata, move_broken=move_broken, broken_dir=metadata.config.broken_dir,
diff --git a/conda_build/variants.py b/conda_build/variants.py
--- a/conda_build/variants.py
+++ b/conda_build/variants.py
@@ -529,8 +529,8 @@ def get_vars(variants, loop_only=False):
     to the matrix dimensionality"""
     special_keys = ('pin_run_as_build', 'zip_keys', 'ignore_version')
     loop_vars = [k for k in variants[0] if k not in special_keys and
-                 (not loop_only or
-                  any(variant[k] != variants[0][k] for variant in variants[1:]))]
+                (not loop_only or
+                any(variant[k] != variants[0][k] for variant in variants[1:]))]
     return loop_vars
 
 


Test patch: