Repository: conda-forge/conda-smithy

Issue:
BUG: something non-deterministic going wrong in squished_input_variants/preserve_top_level_loops
This issue first showed up a long time ago: https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/4190

It occurs with regularity while trying to completely skip pypy builds on an already-migrated feedstock. Of course it would be possible to drop the migrator file completely, but I don't know if that doesn't put the migration bot in an inconsistent state somehow, and feels more intrusive than just a (often temporary) skip.

I hit this again today in https://github.com/conda-forge/scipy-feedstock/pull/274, where we really need to skip PyPy builds for now. 

The first thing to note is that debugging this is hard because the rerender changes based on random print statements or other changes I've made to my local smithy checkout (or even the `ordering:` length in the pypy migrator). This points to some indeterminism (e.g. dict/set order).

So far I was able to narrow it down to the following: things go wrong in
https://github.com/conda-forge/conda-smithy/blob/a04dba97f7f204c6df401bb2cda9c68644c8ad42/conda_smithy/configure_feedstock.py#L626-L627
Before that line, we have
```
used_key_values['python_impl']=['cpython', 'cpython', 'cpython', 'cpython']
```
whereas after it we get (for the failing case)
```
used_key_values['python_impl']=('pypy', 'cpython', 'cpython', 'cpython', 'cpython', 'cpython')
```
This happens against a background of the following values:
```
top_level_loop_vars={'target_platform', 'numpy', 'python', 'python_impl'}
all_used_vars={'docker_image', 'MACOSX_DEPLOYMENT_TARGET', 'libblas', 'channel_sources', 'macos_machine', 'cdt_name', 'liblapack', 'channel_targets', 'BUILD', 'numpy', 'python', 'pin_run_as_build', 'MACOSX_SDK_VERSION', 'python_impl', 'c_compiler', 'target_platform', 'macos_min_version', 'c_stdlib', 'cxx_compiler', 'zip_keys', 'cdt_arch', 'build_number_decrement', 'libcblas'}
preserve_top_level_loops={'python_impl'}
```

Given the definition
https://github.com/conda-forge/conda-smithy/blob/a04dba97f7f204c6df401bb2cda9c68644c8ad42/conda_smithy/configure_feedstock.py#L579

it's possible that `python_impl` should be detected as used (if only from `skip: true  # [python_impl == "pypy"]`) but isn't. Beyond that, I've kind of saturated on smithy code for now 😅  Any help, @beckermr? 🙏 


Discussion:
cc @mbargull (in case you have any insights here)
As usual to debug this it is likely important to set the python hash seed. :/
Too much stuff was changed on the branch for scipy above for me to be able to debug. @h-vetinari do you have a cleaner branch I can look at?
> do you have a cleaner branch I can look at?

The problem should reproduce if you simply add `skip: true  # [python_impl == "pypy"]` on the main branch of the scipy feedstock, but it seems you already were able to track something down? Thanks a lot! 🙏

Resolving patch:
diff --git a/conda_smithy/configure_feedstock.py b/conda_smithy/configure_feedstock.py
--- a/conda_smithy/configure_feedstock.py
+++ b/conda_smithy/configure_feedstock.py
@@ -428,7 +428,7 @@ def _get_used_key_values_by_input_order(
         )
     )
 
-    return used_key_values
+    return used_key_values, zipped_keys
 
 
 def _merge_deployment_target(container_of_dicts, has_macdt):
@@ -617,14 +617,16 @@ def _collapse_subpackage_variants(
         "top_level_loop_vars {}".format(pprint.pformat(top_level_loop_vars))
     )
 
-    used_key_values = _get_used_key_values_by_input_order(
+    used_key_values, used_zipped_vars = _get_used_key_values_by_input_order(
         squished_input_variants,
         squished_used_variants,
         all_used_vars,
     )
 
     for k in preserve_top_level_loops:
-        used_key_values[k] = squished_input_variants[k]
+        # we do not stomp on keys in zips since their order matters
+        if k not in used_zipped_vars:
+            used_key_values[k] = squished_input_variants[k]
 
     _trim_unused_zip_keys(used_key_values)
     _trim_unused_pin_run_as_build(used_key_values)
@@ -859,6 +861,84 @@ def migrate_combined_spec(combined_spec, forge_dir, config, forge_config):
     return combined_spec
 
 
+def _conda_build_api_render_for_smithy(
+    recipe_path,
+    config=None,
+    variants=None,
+    permit_unsatisfiable_variants=True,
+    finalize=True,
+    bypass_env_check=False,
+    **kwargs,
+):
+    """This function works just like conda_build.api.render, but it returns all of metadata objects
+    regardless of whether they produce a unique package hash / name.
+
+    When conda-build renders a recipe, it returns the metadata for each unique file generated. If a key
+    we use at the top-level in a multi-output recipe does not explicitly impact one of the recipe outputs
+    (i.e., an output's recipe doesn't use that key), then conda-build will not return all of the variants
+    for that key.
+
+    This behavior is not what we do in conda-forge (i.e., we want all variants that are not explicitly
+    skipped even if some of the keys in the variants are not explicitly used in an output).
+
+    The most robust way to handle this is to write a custom function that returns metadata for each of
+    the variants in the full exploded matrix that involve a key used by the recipe anywhere,
+    except the ones that the recipe skips.
+    """
+
+    from conda.exceptions import NoPackagesFoundError
+
+    from conda_build.exceptions import DependencyNeedsBuildingError
+    from conda_build.render import finalize_metadata, render_recipe
+    from conda_build.config import get_or_merge_config
+
+    config = get_or_merge_config(config, **kwargs)
+
+    metadata_tuples = render_recipe(
+        recipe_path,
+        bypass_env_check=bypass_env_check,
+        no_download_source=config.no_download_source,
+        config=config,
+        variants=variants,
+        permit_unsatisfiable_variants=permit_unsatisfiable_variants,
+    )
+    output_metas = []
+    for meta, download, render_in_env in metadata_tuples:
+        if not meta.skip() or not config.trim_skip:
+            for od, om in meta.get_output_metadata_set(
+                permit_unsatisfiable_variants=permit_unsatisfiable_variants,
+                permit_undefined_jinja=not finalize,
+                bypass_env_check=bypass_env_check,
+            ):
+                if not om.skip() or not config.trim_skip:
+                    if "type" not in od or od["type"] == "conda":
+                        if finalize and not om.final:
+                            try:
+                                om = finalize_metadata(
+                                    om,
+                                    permit_unsatisfiable_variants=permit_unsatisfiable_variants,
+                                )
+                            except (
+                                DependencyNeedsBuildingError,
+                                NoPackagesFoundError,
+                            ):
+                                if not permit_unsatisfiable_variants:
+                                    raise
+
+                        # remove outputs section from output objects for simplicity
+                        if not om.path and (
+                            outputs := om.get_section("outputs")
+                        ):
+                            om.parent_outputs = outputs
+                            del om.meta["outputs"]
+
+                        output_metas.append((om, download, render_in_env))
+                    else:
+                        output_metas.append((om, download, render_in_env))
+
+    return output_metas
+
+
 def _render_ci_provider(
     provider_name,
     jinja_env,
@@ -976,7 +1056,7 @@ def _render_ci_provider(
             channel_sources = migrated_combined_variant_spec.get(
                 "channel_sources", [""]
             )[0].split(",")
-            metas = conda_build.api.render(
+            metas = _conda_build_api_render_for_smithy(
                 os.path.join(forge_dir, forge_config["recipe_dir"]),
                 platform=platform,
                 arch=arch,


Test patch:
diff --git a/tests/test_configure_feedstock.py b/tests/test_configure_feedstock.py
--- a/tests/test_configure_feedstock.py
+++ b/tests/test_configure_feedstock.py
@@ -2,6 +2,7 @@
 import logging
 import os
 import re
+import shutil
 import textwrap
 
 import pytest
@@ -1015,7 +1016,7 @@ def test_remote_ci_setup(config_yaml):
 
 
 @pytest.mark.parametrize(
-    "squished_input_variants,squished_used_variants,all_used_vars,used_key_values",
+    "squished_input_variants,squished_used_variants,all_used_vars,expected_used_key_values",
     [
         (
             dict(
@@ -1934,13 +1935,96 @@ def test_get_used_key_values_by_input_order(
     squished_input_variants,
     squished_used_variants,
     all_used_vars,
-    used_key_values,
+    expected_used_key_values,
 ):
-    assert (
+    used_key_values, _ = (
         configure_feedstock._get_used_key_values_by_input_order(
             squished_input_variants,
             squished_used_variants,
             all_used_vars,
         )
-        == used_key_values
     )
+    assert used_key_values == expected_used_key_values
+
+
+def test_conda_build_api_render_for_smithy(testing_workdir):
+    import conda_build.api
+
+    _thisdir = os.path.abspath(os.path.dirname(__file__))
+    recipe = os.path.join(_thisdir, "recipes", "multiple_outputs")
+    dest_recipe = os.path.join(testing_workdir, "recipe")
+    shutil.copytree(recipe, dest_recipe)
+    all_top_level_builds = {
+        ("1.5", "9.5"),
+        ("1.5", "9.6"),
+        ("1.6", "9.5"),
+        ("1.6", "9.6"),
+    }
+
+    cs_metas = configure_feedstock._conda_build_api_render_for_smithy(
+        dest_recipe,
+        config=None,
+        variants=None,
+        permit_unsatisfiable_variants=True,
+        finalize=False,
+        bypass_env_check=True,
+        platform="linux",
+        arch="64",
+    )
+    # we have a build matrix with 4 combinations and we get two outputs
+    # plus a top-level build to give us 4 * (2 + 1) = 12 total
+    assert len(cs_metas) == 12
+
+    # we check that we get all combinations
+    top_level_builds = set()
+    for meta, _, _ in cs_metas:
+        for variant in meta.config.variants:
+            top_level_builds.add(
+                (
+                    variant.get("libpng"),
+                    variant.get("libpq"),
+                )
+            )
+        variant = meta.config.variant
+        top_level_builds.add(
+            (
+                variant.get("libpng"),
+                variant.get("libpq"),
+            )
+        )
+    assert len(top_level_builds) == len(all_top_level_builds)
+    assert top_level_builds == all_top_level_builds
+    cb_metas = conda_build.api.render(
+        dest_recipe,
+        config=None,
+        variants=None,
+        permit_unsatisfiable_variants=True,
+        finalize=False,
+        bypass_env_check=True,
+        platform="linux",
+        arch="64",
+    )
+    # conda build will only give us the builds for the unique file names
+    # and collapses the top-level build matrix expansion
+    # thus we only get 3 outputs
+    assert len(cb_metas) == 3
+
+    # we check that we get a subset of all combinations
+    top_level_builds = set()
+    for meta, _, _ in cb_metas:
+        for variant in meta.config.variants:
+            top_level_builds.add(
+                (
+                    variant.get("libpng"),
+                    variant.get("libpq"),
+                )
+            )
+        variant = meta.config.variant
+        top_level_builds.add(
+            (
+                variant.get("libpng"),
+                variant.get("libpq"),
+            )
+        )
+    assert len(top_level_builds) < len(all_top_level_builds)
+    assert top_level_builds.issubset(all_top_level_builds)