Repository: conda-forge/conda-smithy

Issue:
rerender failure on requirements with jinja2 variables in them
<!--
Thanks for reporting your issue.
Please fill out the sections below.
-->
Issue: I cannot seem to rerender recipes with version requirements that look like `{{ jinja2_var}}.*` or `>={{ jinja2_var }}`.

```bash
(base) localhost:ctng-compiler-activation-feedstock Matt$ conda smithy rerender
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
WARNING conda_build.config:platform(303): Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
WARNING conda_build.config:arch(289): Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING conda_build.metadata:ns_cfg(115): No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from internal_defaults
Adding in variants from /Users/Matt/miniconda3/conda_build_config.yaml
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from /Users/Matt/miniconda3/conda_build_config.yaml
Adding in variants from /Users/Matt/Desktop/ctng-compiler-activation-feedstock/recipe/conda_build_config.yaml
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from /Users/Matt/Desktop/ctng-compiler-activation-feedstock/recipe/conda_build_config.yaml
Adding in variants from argument_variants
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from argument_variants
Traceback (most recent call last):
  File "/Users/Matt/miniconda3/lib/python3.7/site-packages/conda/models/version.py", line 38, in __call__
    return cls._cache_[arg]
KeyError: '.*'
```

<br/>
Environment (<code>conda list</code>):
<details>

```
$ conda list
# packages in environment at /Users/Matt/miniconda3:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                     py_0    conda-forge
anaconda-client           1.7.2                      py_0    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
appnope                   0.1.0                 py37_1000    conda-forge
attrs                     19.3.0                     py_0    conda-forge
babel                     2.8.0                      py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
beautifulsoup4            4.8.2                    py37_0    conda-forge
binutils-meta             1.0.4                         0    conda-forge
black                     19.10b0                  py37_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
blinker                   1.4                        py_1    conda-forge
boost-cpp                 1.71.0               h75728bb_1    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
c-compiler                1.0.4                h1239861_0    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
cctools                   927.0.2              h5ba7a2e_3    conda-forge
certifi                   2019.11.28               py37_0    conda-forge
cffi                      1.13.2           py37h33e799b_0    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
clang                     9.0.1           default_hf57f61e_0    conda-forge
clang_osx-64              9.0.1                h05bbb7f_0    conda-forge
clangxx                   9.0.1           default_hf57f61e_0    conda-forge
clangxx_osx-64            9.0.1                h05bbb7f_0    conda-forge
click                     7.0                        py_0    conda-forge
cloud_sptheme             1.10.0                     py_0    conda-forge
clyent                    1.2.2                      py_1    conda-forge
cmarkgfm                  0.4.2            py37h0b31af3_2    conda-forge
compiler-rt               9.0.1                h6a512c6_0    conda-forge
compiler-rt_osx-64        9.0.1                h6a512c6_0    conda-forge
conda                     4.8.0                    py37_1    conda-forge
conda-build               3.18.11                  py37_0    conda-forge
conda-forge-pinning       2020.01.07                    0    conda-forge
conda-package-handling    1.6.0            py37h0b31af3_1    conda-forge
conda-smithy              3.6.5                      py_0    conda-forge
coverage                  5.0.2            py37h0b31af3_0    conda-forge
cryptography              2.8              py37hafa8578_1    conda-forge
curl                      7.65.3               h22ea746_0    conda-forge
cxx-compiler              1.0.4                h707564b_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
deprecated                1.2.7                      py_0    conda-forge
docutils                  0.15.2                   py37_0    conda-forge
entrypoints               0.3                   py37_1000    conda-forge
expat                     2.2.5             h4a8c4bd_1004    conda-forge
filelock                  3.0.10                     py_0    conda-forge
flake8                    3.7.9                    py37_0    conda-forge
future                    0.18.2                   py37_0    conda-forge
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
git                       2.24.0          pl526hdc91d69_1    conda-forge
gitdb2                    2.0.6                      py_0    conda-forge
gitpython                 3.0.5                      py_0    conda-forge
glob2                     0.7                        py_0    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.8                   py37_1000    conda-forge
imagesize                 1.2.0                      py_0    conda-forge
importlib_metadata        1.3.0                    py37_0    conda-forge
ipython                   7.11.1           py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isodate                   0.6.0                      py_1    conda-forge
jedi                      0.15.2                   py37_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
jsonschema                3.2.0                    py37_0    conda-forge
jupyter_core              4.6.1                    py37_0    conda-forge
keyring                   21.0.0                   py37_0    conda-forge
krb5                      1.16.4               h1752a42_0    conda-forge
ld64                      450.3                h3c32e8a_3    conda-forge
libarchive                3.4.0                h8912c15_2    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.65.3               h16faf7d_0    conda-forge
libcxx                    9.0.1                         1    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libgfortran               4.0.0                         2    conda-forge
libiconv                  1.15              h01d97ff_1005    conda-forge
libidn2                   2.3.0                h0b31af3_0    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
liblief                   0.9.0                h3e78482_1    conda-forge
libllvm9                  9.0.1                ha1b3eb9_0    conda-forge
libopenblas               0.3.7                h3d69b6c_6    conda-forge
libssh2                   1.8.2                hcdc9a53_2    conda-forge
libunistring              0.9.10               h1de35cc_0    conda-forge
libxml2                   2.9.10               h53d96d6_0    conda-forge
llvm-openmp               9.0.1                h40edb58_0    conda-forge
lz4-c                     1.8.3             h6de7cb9_1001    conda-forge
lzo                       2.10              h1de35cc_1000    conda-forge
make                      4.2.1             h1de35cc_2004    conda-forge
markupsafe                1.1.1            py37h0b31af3_0    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mock                      3.0.5                    py37_0    conda-forge
more-itertools            8.0.2                      py_0    conda-forge
msrest                    0.6.10                     py_0    conda-forge
mypy_extensions           0.4.3                    py37_0    conda-forge
nbformat                  5.0.3                      py_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
numpy                     1.17.3           py37hde6bac1_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
openssl                   1.1.1d               h0b31af3_0    conda-forge
packaging                 20.0                       py_0    conda-forge
parso                     0.5.2                      py_0    conda-forge
pathspec                  0.7.0                      py_0    conda-forge
pcre                      8.43                 h4a8c4bd_0    conda-forge
perl                      5.26.2            haec8ef5_1006    conda-forge
pexpect                   4.7.0                    py37_0    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pip                       19.3.1                   py37_0    conda-forge
pkginfo                   1.5.0.1                    py_0    conda-forge
pluggy                    0.13.0                   py37_0    conda-forge
prompt_toolkit            3.0.2                      py_0    conda-forge
psutil                    5.6.7            py37h0b31af3_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
py                        1.8.1                      py_0    conda-forge
py-lief                   0.9.0            py37h0ceac7d_1    conda-forge
pybind11                  2.4.3            py37ha1b3eb9_0    conda-forge
pycodestyle               2.5.0                      py_0    conda-forge
pycosat                   0.6.3           py37h0b31af3_1002    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pycrypto                  2.6.1           py37h0b31af3_1003    conda-forge
pyflakes                  2.1.1                      py_0    conda-forge
pygithub                  1.43.6                   py37_0    conda-forge
pygments                  2.5.2                      py_0    conda-forge
pygount                   1.0.0                    pypi_0    pypi
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                   py37_0    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pyrsistent                0.15.6           py37h0b31af3_0    conda-forge
pysocks                   1.7.1                    py37_0    conda-forge
pytest                    5.3.2                    py37_0    conda-forge
pytest-cov                2.8.1                      py_0    conda-forge
pytest-mock               2.0.0                    py37_0    conda-forge
python                    3.7.6                h5c2c468_1    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-libarchive-c       2.9                      py37_0    conda-forge
python.app                1.2             py37h0b31af3_1201    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.3              py37h0b31af3_0    conda-forge
readline                  8.0                  hcfe32e1_0    conda-forge
readme_renderer           24.0                       py_0    conda-forge
regex                     2020.1.7         py37h0b31af3_0    conda-forge
requests                  2.22.0                   py37_1    conda-forge
requests-oauthlib         1.2.0                      py_0    conda-forge
requests-toolbelt         0.9.1                      py_0    conda-forge
ripgrep                   11.0.2               h01d97ff_3    conda-forge
ruamel.yaml               0.16.5           py37h0b31af3_1    conda-forge
ruamel.yaml.clib          0.2.0            py37h0b31af3_0    conda-forge
ruamel_yaml               0.15.80         py37h0b31af3_1000    conda-forge
setuptools                44.0.0                   py37_0    conda-forge
six                       1.13.0                   py37_0    conda-forge
smmap2                    2.0.5                      py_0    conda-forge
snowballstemmer           2.0.0                      py_0    conda-forge
soupsieve                 1.9.4                    py37_0    conda-forge
sphinx                    2.3.1                      py_0    conda-forge
sphinxcontrib-applehelp   1.0.1                      py_0    conda-forge
sphinxcontrib-devhelp     1.0.1                      py_0    conda-forge
sphinxcontrib-fulltoc     1.2.0                      py_0    conda-forge
sphinxcontrib-htmlhelp    1.0.2                      py_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.2                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.3                      py_0    conda-forge
sqlite                    3.30.1               h93121df_0    conda-forge
swig                      4.0.1                h6de7cb9_0    conda-forge
tapi                      1000.10.8            ha1b3eb9_4    conda-forge
tk                        8.6.10               hbbe82c9_0    conda-forge
toml                      0.10.0                     py_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py37h0b31af3_0    conda-forge
tqdm                      4.41.1                     py_0    conda-forge
traitlets                 4.3.3                    py37_0    conda-forge
twine                     3.1.1                    py37_0    conda-forge
typed-ast                 1.4.0            py37h0b31af3_0    conda-forge
typing_extensions         3.7.4.1                  py37_0    conda-forge
urllib3                   1.25.7                   py37_0    conda-forge
vsts-python-api           0.1.22                     py_0    conda-forge
wcwidth                   0.1.8                      py_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wget                      1.20.1               h33e2efd_0    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
wrapt                     1.11.2           py37h0b31af3_0    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
yaml                      0.2.2                h0b31af3_1    conda-forge
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge
zstd                      1.4.4                he7fca8b_1    conda-forge
```
</details>

<br/>
Details about  <code>conda</code> and system ( <code>conda info</code> ):
<details>

```
$ conda info

     active environment : base
    active env location : /Users/Matt/miniconda3
            shell level : 1
       user config file : /Users/Matt/.condarc
 populated config files : /Users/Matt/.condarc
          conda version : 4.8.0
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __osx=10.14.6
       base environment : /Users/Matt/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/Matt/miniconda3/pkgs
                          /Users/Matt/.conda/pkgs
       envs directories : /Users/Matt/miniconda3/envs
                          /Users/Matt/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.0 requests/2.22.0 CPython/3.7.6 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 501:20
             netrc file : /Users/Matt/.netrc
           offline mode : False
```
</details>



Discussion:

Resolving patch:
diff --git a/conda_smithy/configure_feedstock.py b/conda_smithy/configure_feedstock.py
--- a/conda_smithy/configure_feedstock.py
+++ b/conda_smithy/configure_feedstock.py
@@ -540,6 +540,7 @@ def _render_ci_provider(
     keep_noarchs=None,
     extra_platform_files={},
     upload_packages=[],
+    return_metadata=False,
 ):
     if keep_noarchs is None:
         keep_noarchs = [False] * len(platforms)
@@ -714,7 +715,16 @@ def _render_ci_provider(
         with write_file(platform_target_path) as fh:
             fh.write(template.render(**forge_config))
     # TODO: azure-pipelines might need the same as circle
-    return forge_config
+    if return_metadata:
+        return dict(
+            forge_config=forge_config,
+            metas_list_of_lists=metas_list_of_lists,
+            platforms=platforms,
+            archs=archs,
+            enable_platform=enable_platform,
+        )
+    else:
+        return forge_config
 
 
 def _get_build_setup_line(forge_dir, platform, forge_config):
@@ -884,7 +894,7 @@ def _get_platforms_of_provider(provider, forge_config):
     return platforms, archs, keep_noarchs, upload_packages
 
 
-def render_circle(jinja_env, forge_config, forge_dir):
+def render_circle(jinja_env, forge_config, forge_dir, return_metadata=False):
     target_path = os.path.join(forge_dir, ".circleci", "config.yml")
     template_filename = "circle.yml.tmpl"
     fast_finish_text = textwrap.dedent(
@@ -926,6 +936,7 @@ def render_circle(jinja_env, forge_config, forge_dir):
         keep_noarchs=keep_noarchs,
         extra_platform_files=extra_platform_files,
         upload_packages=upload_packages,
+        return_metadata=return_metadata,
     )
 
 
@@ -983,7 +994,7 @@ def _render_template_exe_files(
         set_exe_file(target_fname, True)
 
 
-def render_travis(jinja_env, forge_config, forge_dir):
+def render_travis(jinja_env, forge_config, forge_dir, return_metadata=False):
     target_path = os.path.join(forge_dir, ".travis.yml")
     template_filename = "travis.yml.tmpl"
     fast_finish_text = textwrap.dedent(
@@ -1022,6 +1033,7 @@ def render_travis(jinja_env, forge_config, forge_dir):
         platform_specific_setup=_travis_specific_setup,
         upload_packages=upload_packages,
         extra_platform_files=extra_platform_files,
+        return_metadata=return_metadata,
     )
 
 
@@ -1039,7 +1051,7 @@ def _appveyor_specific_setup(jinja_env, forge_config, forge_dir, platform):
     forge_config["build_setup"] = build_setup
 
 
-def render_appveyor(jinja_env, forge_config, forge_dir):
+def render_appveyor(jinja_env, forge_config, forge_dir, return_metadata=False):
     target_path = os.path.join(forge_dir, ".appveyor.yml")
     fast_finish_text = textwrap.dedent(
         """\
@@ -1069,6 +1081,7 @@ def render_appveyor(jinja_env, forge_config, forge_dir):
         keep_noarchs=keep_noarchs,
         platform_specific_setup=_appveyor_specific_setup,
         upload_packages=upload_packages,
+        return_metadata=return_metadata,
     )
 
 
@@ -1102,7 +1115,7 @@ def _azure_specific_setup(jinja_env, forge_config, forge_dir, platform):
     )
 
 
-def render_azure(jinja_env, forge_config, forge_dir):
+def render_azure(jinja_env, forge_config, forge_dir, return_metadata=False):
     target_path = os.path.join(forge_dir, "azure-pipelines.yml")
     template_filename = "azure-pipelines.yml.tmpl"
     fast_finish_text = ""
@@ -1127,6 +1140,7 @@ def render_azure(jinja_env, forge_config, forge_dir):
         platform_specific_setup=_azure_specific_setup,
         keep_noarchs=keep_noarchs,
         upload_packages=upload_packages,
+        return_metadata=return_metadata,
     )
 
 
@@ -1155,7 +1169,7 @@ def _drone_specific_setup(jinja_env, forge_config, forge_dir, platform):
     )
 
 
-def render_drone(jinja_env, forge_config, forge_dir):
+def render_drone(jinja_env, forge_config, forge_dir, return_metadata=False):
     target_path = os.path.join(forge_dir, ".drone.yml")
     template_filename = "drone.yml.tmpl"
     fast_finish_text = ""
@@ -1180,27 +1194,54 @@ def render_drone(jinja_env, forge_config, forge_dir):
         platform_specific_setup=_drone_specific_setup,
         keep_noarchs=keep_noarchs,
         upload_packages=upload_packages,
+        return_metadata=return_metadata,
     )
 
 
-def render_README(jinja_env, forge_config, forge_dir):
+def render_README(jinja_env, forge_config, forge_dir, render_info=None):
     if "README.md" in forge_config["skip_render"]:
         logger.info("README.md rendering is skipped")
         return
-    # we only care about the first metadata object for sake of readme
-    metas = conda_build.api.render(
-        os.path.join(forge_dir, "recipe"),
-        exclusive_config_file=forge_config["exclusive_config_file"],
-        permit_undefined_jinja=True,
-        finalize=False,
-        bypass_env_check=True,
-        trim_skip=False,
-    )
 
-    if "parent_recipe" in metas[0][0].meta["extra"]:
-        package_name = metas[0][0].meta["extra"]["parent_recipe"]["name"]
+    render_info = render_info or []
+
+    # pull out relevant metadata from rendering
+    try:
+        metas = conda_build.api.render(
+            os.path.join(forge_dir, "recipe"),
+            exclusive_config_file=forge_config["exclusive_config_file"],
+            permit_undefined_jinja=True,
+            finalize=False,
+            bypass_env_check=True,
+            trim_skip=False,
+        )
+        metas = [m[0] for m in metas]
+    except Exception:
+        # sometimes the above fails so we grab actual metadata
+        done = False
+        metas = []
+        for md in render_info:
+            for _metas, enabled in zip(
+                md["metas_list_of_lists"], md["enable_platform"]
+            ):
+                if enabled and len(_metas) > 0:
+                    metas = _metas
+                    done = True
+                    break
+            if done:
+                break
+
+    if len(metas) == 0:
+        raise RuntimeError(
+            "Could not create any metadata for rendering the README.md!"
+            " This likely indicates a serious bug or a feedstock with no actual"
+            " builds."
+        )
+
+    if "parent_recipe" in metas[0].meta["extra"]:
+        package_name = metas[0].meta["extra"]["parent_recipe"]["name"]
     else:
-        package_name = metas[0][0].name()
+        package_name = metas[0].name()
 
     ci_support_path = os.path.join(forge_dir, ".ci_support")
     variants = []
@@ -1212,17 +1253,17 @@ def render_README(jinja_env, forge_config, forge_dir):
 
     template = jinja_env.get_template("README.md.tmpl")
     target_fname = os.path.join(forge_dir, "README.md")
-    forge_config["noarch_python"] = all(meta[0].noarch for meta in metas)
-    forge_config["package"] = metas[0][0]
+    forge_config["noarch_python"] = all(meta.noarch for meta in metas)
+    forge_config["package"] = metas[0]
     forge_config["package_name"] = package_name
     forge_config["variants"] = sorted(variants)
     forge_config["outputs"] = sorted(
-        list(OrderedDict((meta[0].name(), None) for meta in metas))
+        list(OrderedDict((meta.name(), None) for meta in metas))
     )
     forge_config["maintainers"] = sorted(
         set(
             chain.from_iterable(
-                meta[0].meta["extra"].get("recipe-maintainers", [])
+                meta.meta["extra"].get("recipe-maintainers", [])
                 for meta in metas
             )
         )
@@ -1614,12 +1655,28 @@ def main(
     clear_variants(forge_dir)
     clear_scripts(forge_dir)
 
-    render_circle(env, config, forge_dir)
-    render_travis(env, config, forge_dir)
-    render_appveyor(env, config, forge_dir)
-    render_azure(env, config, forge_dir)
-    render_drone(env, config, forge_dir)
-    render_README(env, config, forge_dir)
+    # the order of these calls appears to matter
+    render_info = []
+    render_info.append(
+        render_circle(env, config, forge_dir, return_metadata=True)
+    )
+    render_info.append(
+        render_travis(env, config, forge_dir, return_metadata=True)
+    )
+    render_info.append(
+        render_appveyor(env, config, forge_dir, return_metadata=True)
+    )
+    render_info.append(
+        render_azure(env, config, forge_dir, return_metadata=True)
+    )
+    render_info.append(
+        render_drone(env, config, forge_dir, return_metadata=True)
+    )
+    # put azure first just in case
+    tmp = render_info[0]
+    render_info[0] = render_info[-2]
+    render_info[-2] = tmp
+    render_README(env, config, forge_dir, render_info)
 
     if os.path.isdir(os.path.join(forge_dir, ".ci_support")):
         with write_file(os.path.join(forge_dir, ".ci_support", "README")) as f:


Test patch: