Repository: conda-forge/conda-smithy

Issue:
smithy rerender doesn't obey recipe_dir from conda-forge.yml
Issue:

If I want the recipe to live in `conda.recipe` instead of `recipe`, I should be able to set `recipe_dir` in `conda-forge.yml`.  However, with 3.6.12 py_0, I hit the following:
```
[I] ➜ conda smithy rerender 
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): README.md rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): LICENSE.txt rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .gitattributes rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .gitignore rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): build-locally.py rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): LICENSE rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .github/CONTRIBUTING.md rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .github/ISSUE_TEMPLATE.md rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .github/PULL_REQUEST_TEMPLATE.md rendering is skipped
INFO conda_smithy.configure_feedstock:copy_feedstock_content(1313): .github/workflows rendering is skipped
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
WARNING conda_build.config:platform(303): Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
WARNING conda_build.config:arch(289): Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
Adding in variants from internal_defaults
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from internal_defaults
Adding in variants from /Users/timothys/opt/miniconda3/conda_build_config.yaml
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from /Users/timothys/opt/miniconda3/conda_build_config.yaml
Adding in variants from config.variant
INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from config.variant
Error: no such directory: /Volumes/cs/cf-ci-skeleton/recipe
```

Looks like there is someplace hardcoded looking for `recipe` instead of taking the value from the config.  I'll submit a PR for this.

<br/>
Environment (<code>conda list</code>):
<details>

```
~/work/cf-ci-skeleton via 🅒 base took 17s 
[I] ➜ conda list
# packages in environment at /Users/timothys/opt/miniconda3:
#
# Name                    Version                   Build  Channel
beautifulsoup4            4.8.2                    py37_0    conda-forge
blinker                   1.4                        py_1    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28       py37hc8dfbb8_1    conda-forge
cffi                      1.14.0           py37h356ff06_0    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
conda                     4.8.2                    py37_0    conda-forge
conda-build               3.18.11                  py37_0    conda-forge
conda-forge-pinning       2020.03.10                    0    conda-forge
conda-package-handling    1.6.0            py37h0b31af3_1    conda-forge
conda-smithy              3.6.12                     py_0    conda-forge
cryptography              2.8              py37hafa8578_1    conda-forge
curl                      7.68.0               h8754def_0    conda-forge
deprecated                1.2.7                      py_0    conda-forge
expat                     2.2.9                h4a8c4bd_2    conda-forge
filelock                  3.0.10                     py_0    conda-forge
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
git                       2.25.0          pl526hdc91d69_0    conda-forge
gitdb                     4.0.2                      py_0    conda-forge
gitpython                 3.1.0                      py_0    conda-forge
glob2                     0.7                        py_0    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.9                        py_1    conda-forge
isodate                   0.6.0                      py_1    conda-forge
jinja2                    2.11.1                     py_0    conda-forge
krb5                      1.16.4               h1752a42_0    conda-forge
libarchive                3.3.3             h8912c15_1007    conda-forge
libcurl                   7.68.0               h709d2b2_0    conda-forge
libcxx                    9.0.1                         1    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libiconv                  1.15              h01d97ff_1005    conda-forge
liblief                   0.9.0                h3e78482_1    conda-forge
libssh2                   1.8.2                hcdc9a53_2    conda-forge
libxml2                   2.9.10               h53d96d6_0    conda-forge
lz4-c                     1.8.3             h6de7cb9_1001    conda-forge
lzo                       2.10              h1de35cc_1000    conda-forge
markupsafe                1.1.1            py37h9bfed18_1    conda-forge
msrest                    0.6.10                     py_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
openssl                   1.1.1d               h0b31af3_0    conda-forge
pcre                      8.44                 h4a8c4bd_0    conda-forge
perl                      5.26.2            haec8ef5_1006    conda-forge
pip                       20.0.2                     py_2    conda-forge
pkginfo                   1.5.0.1                    py_0    conda-forge
psutil                    5.7.0            py37h0b31af3_0    conda-forge
py-lief                   0.9.0            py37h0ceac7d_1    conda-forge
pycosat                   0.6.3           py37h9bfed18_1003    conda-forge
pycparser                 2.20                       py_0    conda-forge
pycrypto                  2.6.1           py37h0b31af3_1003    conda-forge
pygithub                  1.43.6                   py37_0    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pysocks                   1.7.1                    py37_0    conda-forge
python                    3.7.3                h93065d6_1    conda-forge
python-libarchive-c       2.9                      py37_0    conda-forge
python.app                1.2             py37h0b31af3_1201    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.3              py37h0b31af3_0    conda-forge
readline                  8.0                  hcfe32e1_0    conda-forge
requests                  2.23.0                   py37_0    conda-forge
requests-oauthlib         1.2.0                      py_0    conda-forge
ripgrep                   11.0.2               h01d97ff_3    conda-forge
ruamel.yaml               0.16.6           py37h0b31af3_0    conda-forge
ruamel.yaml.clib          0.2.0            py37h0b31af3_0    conda-forge
ruamel_yaml               0.15.80         py37h0b31af3_1000    conda-forge
setuptools                46.0.0           py37hc8dfbb8_2    conda-forge
six                       1.14.0                     py_1    conda-forge
smmap                     3.0.1                      py_0    conda-forge
soupsieve                 1.9.4                    py37_0    conda-forge
sqlite                    3.30.1               h93121df_0    conda-forge
tk                        8.6.10               hbbe82c9_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tqdm                      4.43.0                     py_0    conda-forge
urllib3                   1.25.7                   py37_0    conda-forge
vsts-python-api           0.1.22                     py_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
wrapt                     1.12.1           py37h9bfed18_1    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
yaml                      0.2.2                h0b31af3_1    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge
zstd                      1.4.4                he7fca8b_1    conda-forge

```
</details>

<br/>
Details about  <code>conda</code> and system ( <code>conda info</code> ):
<details>

```
~/work/cf-ci-skeleton via 🅒 base 
[I] ➜ conda info

     active environment : base
    active env location : /Users/timothys/opt/miniconda3
            shell level : 5
       user config file : /Users/timothys/.condarc
 populated config files : /Users/timothys/.condarc
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.3.final.0
       virtual packages : __osx=10.15.3
       base environment : /Users/timothys/opt/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/timothys/opt/miniconda3/pkgs
                          /Users/timothys/.conda/pkgs
       envs directories : /Users/timothys/opt/miniconda3/envs
                          /Users/timothys/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.2 requests/2.23.0 CPython/3.7.3 Darwin/19.3.0 OSX/10.15.3
                UID:GID : 503:20
             netrc file : None
           offline mode : False

```
</details>



Discussion:

Resolving patch:
diff --git a/conda_smithy/configure_feedstock.py b/conda_smithy/configure_feedstock.py
--- a/conda_smithy/configure_feedstock.py
+++ b/conda_smithy/configure_feedstock.py
@@ -447,7 +447,9 @@ def _get_fast_finish_script(
     fast_finish_script = ""
     tooling_branch = "master"
 
-    cfbs_fpath = os.path.join(forge_dir, "recipe", "ff_ci_pr_build.py")
+    cfbs_fpath = os.path.join(
+        forge_dir, forge_config["recipe_dir"], "ff_ci_pr_build.py"
+    )
     if provider_name == "appveyor":
         if os.path.exists(cfbs_fpath):
             fast_finish_script = "{recipe_dir}\\ff_ci_pr_build".format(
@@ -563,7 +565,7 @@ def _render_ci_provider(
             combined_variant_spec,
             _,
         ) = conda_build.variants.get_package_combined_spec(
-            os.path.join(forge_dir, "recipe"), config=config
+            os.path.join(forge_dir, forge_config["recipe_dir"]), config=config
         )
 
         migrated_combined_variant_spec = migrate_combined_spec(
@@ -571,7 +573,7 @@ def _render_ci_provider(
         )
 
         metas = conda_build.api.render(
-            os.path.join(forge_dir, "recipe"),
+            os.path.join(forge_dir, forge_config["recipe_dir"]),
             platform=platform,
             arch=arch,
             ignore_system_variants=True,
@@ -673,7 +675,9 @@ def _render_ci_provider(
         # If the recipe supplies its own upload_or_check_non_existence.py upload script,
         # we use it instead of the global one.
         upload_fpath = os.path.join(
-            forge_dir, "recipe", "upload_or_check_non_existence.py"
+            forge_dir,
+            forge_config["recipe_dir"],
+            "upload_or_check_non_existence.py",
         )
         if os.path.exists(upload_fpath):
             if provider_name == "circle":
@@ -733,15 +737,21 @@ def _get_build_setup_line(forge_dir, platform, forge_config):
     # we use it instead of the global one.
     if platform == "linux":
         cfbs_fpath = os.path.join(
-            forge_dir, "recipe", "run_conda_forge_build_setup_linux"
+            forge_dir,
+            forge_config["recipe_dir"],
+            "run_conda_forge_build_setup_linux",
         )
     elif platform == "win":
         cfbs_fpath = os.path.join(
-            forge_dir, "recipe", "run_conda_forge_build_setup_win.bat"
+            forge_dir,
+            forge_config["recipe_dir"],
+            "run_conda_forge_build_setup_win.bat",
         )
     else:
         cfbs_fpath = os.path.join(
-            forge_dir, "recipe", "run_conda_forge_build_setup_osx"
+            forge_dir,
+            forge_config["recipe_dir"],
+            "run_conda_forge_build_setup_osx",
         )
 
     build_setup = ""
@@ -793,7 +803,7 @@ def _get_build_setup_line(forge_dir, platform, forge_config):
 def _circle_specific_setup(jinja_env, forge_config, forge_dir, platform):
 
     if platform == "linux":
-        yum_build_setup = generate_yum_requirements(forge_dir)
+        yum_build_setup = generate_yum_requirements(forge_config, forge_dir)
         if yum_build_setup:
             forge_config["yum_build_setup"] = yum_build_setup
 
@@ -824,10 +834,10 @@ def _circle_specific_setup(jinja_env, forge_config, forge_dir, platform):
         set_exe_file(target_fname, True)
 
 
-def generate_yum_requirements(forge_dir):
+def generate_yum_requirements(forge_config, forge_dir):
     # If there is a "yum_requirements.txt" file in the recipe, we honour it.
     yum_requirements_fpath = os.path.join(
-        forge_dir, "recipe", "yum_requirements.txt"
+        forge_dir, forge_config["recipe_dir"], "yum_requirements.txt"
     )
     yum_build_setup = ""
     if os.path.exists(yum_requirements_fpath):
@@ -952,7 +962,7 @@ def _travis_specific_setup(jinja_env, forge_config, forge_dir, platform):
     template_files = platform_templates.get(platform, [])
 
     if platform == "linux":
-        yum_build_setup = generate_yum_requirements(forge_dir)
+        yum_build_setup = generate_yum_requirements(forge_config, forge_dir)
         if yum_build_setup:
             forge_config["yum_build_setup"] = yum_build_setup
 
@@ -1091,7 +1101,7 @@ def _azure_specific_setup(jinja_env, forge_config, forge_dir, platform):
     build_setup = _get_build_setup_line(forge_dir, platform, forge_config)
 
     if platform == "linux":
-        yum_build_setup = generate_yum_requirements(forge_dir)
+        yum_build_setup = generate_yum_requirements(forge_config, forge_dir)
         if yum_build_setup:
             forge_config["yum_build_setup"] = yum_build_setup
 
@@ -1128,6 +1138,8 @@ def render_azure(jinja_env, forge_config, forge_dir, return_metadata=False):
         upload_packages,
     ) = _get_platforms_of_provider("azure", forge_config)
 
+    logger.debug("azure platforms retreived")
+
     return _render_ci_provider(
         "azure",
         jinja_env=jinja_env,
@@ -1156,7 +1168,7 @@ def _drone_specific_setup(jinja_env, forge_config, forge_dir, platform):
     build_setup = _get_build_setup_line(forge_dir, platform, forge_config)
 
     if platform == "linux":
-        yum_build_setup = generate_yum_requirements(forge_dir)
+        yum_build_setup = generate_yum_requirements(forge_config, forge_dir)
         if yum_build_setup:
             forge_config["yum_build_setup"] = yum_build_setup
 
@@ -1209,7 +1221,7 @@ def render_README(jinja_env, forge_config, forge_dir, render_info=None):
     # pull out relevant metadata from rendering
     try:
         metas = conda_build.api.render(
-            os.path.join(forge_dir, "recipe"),
+            os.path.join(forge_dir, forge_config["recipe_dir"]),
             exclusive_config_file=forge_config["exclusive_config_file"],
             permit_undefined_jinja=True,
             finalize=False,
@@ -1389,10 +1401,22 @@ def _load_forge_config(forge_dir, exclusive_config_file):
         with open(forge_yml, "r") as fh:
             file_config = list(yaml.safe_load_all(fh))[0] or {}
 
+        # The config is just the union of the defaults, and the overriden
+        # values.
+        for key, value in file_config.items():
+            # Deal with dicts within dicts.
+            if isinstance(value, dict):
+                config_item = config.setdefault(key, value)
+                config_item.update(value)
+            else:
+                config[key] = value
+
         # check for conda-smithy 2.x matrix which we can't auto-migrate
         # to conda_build_config
         if file_config.get("matrix") and not os.path.exists(
-            os.path.join(forge_dir, "recipe", "conda_build_config.yaml")
+            os.path.join(
+                forge_dir, config["recipe_dir"], "conda_build_config.yaml"
+            )
         ):
             raise ValueError(
                 "Cannot rerender with matrix in conda-forge.yml."
@@ -1409,16 +1433,6 @@ def _load_forge_config(forge_dir, exclusive_config_file):
                 " Use conda_build_config.yaml instead"
             )
 
-        # The config is just the union of the defaults, and the overriden
-        # values.
-        for key, value in file_config.items():
-            # Deal with dicts within dicts.
-            if isinstance(value, dict):
-                config_item = config.setdefault(key, value)
-                config_item.update(value)
-            else:
-                config[key] = value
-
     # An older conda-smithy used to have some files which should no longer exist,
     # remove those now.
     old_files = [
@@ -1698,7 +1712,9 @@ def main(
 
     error_on_warn = False if no_check_uptodate else True
     index = conda_build.conda_interface.get_index(channel_urls=["conda-forge"])
+    logger.debug("Beginning Resolving Index")
     r = conda_build.conda_interface.Resolve(index)
+    logger.debug("Index rendered")
 
     # Check that conda-smithy is up-to-date
     check_version_uptodate(r, "conda-smithy", __version__, error_on_warn)
@@ -1728,6 +1744,7 @@ def main(
             )
 
     env = make_jinja_env(forge_dir)
+    logger.debug("env rendered")
 
     copy_feedstock_content(config, forge_dir)
     if os.path.exists(os.path.join(forge_dir, "build-locally.py")):
@@ -1735,24 +1752,30 @@ def main(
     clear_variants(forge_dir)
     clear_scripts(forge_dir)
     set_migration_fns(forge_dir, config)
+    logger.debug("migration fns set")
 
     # the order of these calls appears to matter
     render_info = []
     render_info.append(
         render_circle(env, config, forge_dir, return_metadata=True)
     )
+    logger.debug("circle rendered")
     render_info.append(
         render_travis(env, config, forge_dir, return_metadata=True)
     )
+    logger.debug("travis rendered")
     render_info.append(
         render_appveyor(env, config, forge_dir, return_metadata=True)
     )
+    logger.debug("appveyor rendered")
     render_info.append(
         render_azure(env, config, forge_dir, return_metadata=True)
     )
+    logger.debug("azure rendered")
     render_info.append(
         render_drone(env, config, forge_dir, return_metadata=True)
     )
+    logger.debug("drone rendered")
     # put azure first just in case
     tmp = render_info[0]
     render_info[0] = render_info[-2]


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -48,15 +48,21 @@ def return_to_saved_path():
 
 
 @pytest.fixture(scope="function")
-def config_yaml(testing_workdir):
+def recipe_dirname():
+    return "recipe"
+
+
+@pytest.fixture(scope="function")
+def config_yaml(testing_workdir, recipe_dirname):
     config = {"python": ["2.7", "3.5"], "r_base": ["3.3.2", "3.4.2"]}
-    os.makedirs(os.path.join(testing_workdir, "recipe"))
+    os.makedirs(os.path.join(testing_workdir, recipe_dirname))
     with open(os.path.join(testing_workdir, "config.yaml"), "w") as f:
         f.write("docker:\n")
         f.write("  fallback_image:\n")
         f.write("  - centos:6\n")
     with open(
-        os.path.join(testing_workdir, "recipe", "default_config.yaml"), "w"
+        os.path.join(testing_workdir, recipe_dirname, "default_config.yaml"),
+        "w",
     ) as f:
         yaml.dump(config, f, default_flow_style=False)
         # need selectors, so write these more manually
@@ -87,24 +93,29 @@ def config_yaml(testing_workdir):
     ) as f:
         f.write("echo dummy file")
     with open(
-        os.path.join(testing_workdir, "recipe", "short_config.yaml"), "w"
+        os.path.join(testing_workdir, recipe_dirname, "short_config.yaml"), "w"
     ) as f:
         config = {"python": ["2.7"]}
         yaml.dump(config, f, default_flow_style=False)
     with open(
-        os.path.join(testing_workdir, "recipe", "long_config.yaml"), "w"
+        os.path.join(testing_workdir, recipe_dirname, "long_config.yaml"), "w"
     ) as f:
         config = {"python": ["2.7", "3.5", "3.6"]}
         yaml.dump(config, f, default_flow_style=False)
     with open(os.path.join(testing_workdir, "conda-forge.yml"), "w") as f:
-        config = {"upload_on_branch": "foo-branch"}
+        config = {
+            "upload_on_branch": "foo-branch",
+            "recipe_dir": recipe_dirname,
+        }
         yaml.dump(config, f, default_flow_style=False)
     return testing_workdir
 
 
 @pytest.fixture(scope="function")
-def noarch_recipe(config_yaml, request):
-    with open(os.path.join(config_yaml, "recipe", "meta.yaml"), "w") as fh:
+def noarch_recipe(config_yaml, recipe_dirname, request):
+    with open(
+        os.path.join(config_yaml, recipe_dirname, "meta.yaml"), "w"
+    ) as fh:
         fh.write(
             """
 package:
@@ -124,7 +135,7 @@ def noarch_recipe(config_yaml, request):
         _load_forge_config(
             config_yaml,
             exclusive_config_file=os.path.join(
-                config_yaml, "recipe", "default_config.yaml"
+                config_yaml, recipe_dirname, "default_config.yaml"
             ),
         ),
     )
diff --git a/tests/test_configure_feedstock.py b/tests/test_configure_feedstock.py
--- a/tests/test_configure_feedstock.py
+++ b/tests/test_configure_feedstock.py
@@ -48,6 +48,7 @@ def test_noarch_runs_on_circle(noarch_recipe, jinja_env):
     assert len(os.listdir(matrix_dir)) == 1
 
 
+@pytest.mark.parametrize("recipe_dirname", ["recipe", "custom_recipe_dir"])
 def test_noarch_runs_on_azure(noarch_recipe, jinja_env):
     cnfgr_fdstk.render_azure(
         jinja_env=jinja_env,