Repository: conda/conda-build

Issue:
KeyError when using libmamba solver with conda-build
### What happened?

A customer has seen this traceback when using conda-build 3.26.1 with conda-libmamba-solver:

```
Traceback (most recent call last):
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/cli/main.py", line 64, in main_subshell
exit_code = do_call(args, parser)
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 143, in do_call
result = plugin_subcommand.action(getattr(args, "_args", args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/plugin.py", line 10, in build
execute(*args, **kwargs)
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/cli/main_build.py", line 568, in execute
outputs = api.build(
^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/api.py", line 253, in build
return build_tree(
^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/build.py", line 3804, in build_tree
packages_from_this = build(
^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/build.py", line 2474, in build
output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 923, in expand_outputs
for output_dict, m in deepcopy(_m).get_output_metadata_set(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/metadata.py", line 2574, in get_output_metadata_set
conda_packages = finalize_outputs_pass(
^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/metadata.py", line 934, in finalize_outputs_pass
fm = finalize_metadata(
^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 637, in finalize_metadata
build_unsat, host_unsat = add_upstream_pins(
^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 452, in add_upstream_pins
build_deps, build_unsat, extra_run_specs_from_build = _read_upstream_pin_files(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 440, in _read_upstream_pin_files
extra_run_specs = get_upstream_pins(m, actions, env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 420, in get_upstream_pins
loc, dist = execute_download_actions(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 387, in execute_download_actions
_loc = os.path.join(pkg_dir, index[pkg].fn)
~~~~~^^^^^
KeyError: Dist(channel='https://conda.anaconda.xxx.xxx/anaconda', dist_name='python-3.8.17-h955ad1f_0', name='python', fmt='.tar.bz2', version='3.8.17', build_string='h955ad1f_0', build_number=0, base_url=None, platform=None)
```

### Conda Details

<details>
<summary><code>conda info</code></summary>

```
active environment : None
user config file : /root/.condarc
populated config files : /root/conda-build/.condarc
conda version : 23.7.3
conda-build version : 3.26.1
python version : 3.11.4.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.26=0
__linux=3.10.0=0
__unix=0=0
base environment : /root/conda-build (writable)
conda av data dir : /root/conda-build/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.xxxxx/arepo/linux-64
https://conda.anaconda.xxxxx/arepo/noarch
https://conda.anaconda.xxxxx/anaconda-patched/linux-64
https://conda.anaconda.xxxxx/anaconda-patched/noarch
https://conda.anaconda.xxxxx/anaconda/linux-64
https://conda.anaconda.xxxxx/anaconda/noarch
https://conda.anaconda.xxxxx/r-patched/linux-64
https://conda.anaconda.xxxxx/r-patched/noarch
https://conda.anaconda.xxxxx/r/linux-64
https://conda.anaconda.xxxxx/r/noarch
https://conda.anaconda.xxxxx/msys2-patched/linux-64
https://conda.anaconda.xxxxx/msys2-patched/noarch
https://conda.anaconda.xxxxx/msys2/linux-64
https://conda.anaconda.xxxxx/msys2/noarch
package cache : /root/conda-build/pkgs
/root/.conda/pkgs
envs directories : /root/conda-build/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/23.7.3 requests/2.31.0 CPython/3.11.4 Linux/3.10.0-1160.95.1.el7.x86_64 amzn/2 glibc/2.26 solver/libmamba conda-libmamba-solver/23.7.0 libmambapy/1.4.1
UID:GID : 0:0
netrc file : None
offline mode : False
```

</details>

<details>
<summary><code>conda config</code></summary>

```
==> /root/conda-build/.condarc <==
allow_softlinks: False
auto_update_conda: False
notify_outdated_conda: False
aggressive_update_packages:
- anaconda-patched::ca-certificates
- anaconda-patched::certifi
- arepo::condarc
- arepo::anaconda-client-config
add_anaconda_token: True
channel_alias: https://conda.anaconda.xxxxx/
channels:
- defaults
default_channels:
- arepo
- anaconda-patched
- anaconda
- r-patched
- r
- msys2-patched
- msys2
show_channel_urls: True
report_errors: False
shortcuts: False
solver: libmamba

==> cmd_line <==
debug: False
json: False
```

</details>

<details>
<summary><code>conda list</code></summary>

```
# packages in environment at /root/conda-build:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
anaconda-client 1.12.0 py311h06a4308_0 defaults
anaconda-client-config 2019.08.21 0 defaults
attrs 22.1.0 py311h06a4308_0 defaults
backports 1.1 pyhd3eb1b0_0 defaults
backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0 defaults
backports.tempfile 1.0 pyhd3eb1b0_1 defaults
backports.weakref 1.0.post1 py_1 defaults
beautifulsoup4 4.12.2 py311h06a4308_0 defaults
boltons 23.0.0 py311h06a4308_0 defaults
brotlipy 0.7.0 py311h5eee18b_1002 defaults
bzip2 1.0.8 h7b6447c_0 defaults
c-ares 1.19.0 h5eee18b_0 defaults
ca-certificates 2023.05.30 h06a4308_0 defaults
certifi 2023.7.22 py311h06a4308_0 defaults
cffi 1.15.1 py311h5eee18b_3 defaults
chardet 4.0.0 py311h06a4308_1003 defaults
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
click 8.0.4 py311h06a4308_0 defaults
clyent 1.2.2 py311h06a4308_1 defaults
conda 23.7.3 py311h06a4308_0 defaults
conda-build 3.26.1 py311h06a4308_0 defaults
conda-index 0.2.3 py311h06a4308_0 defaults
conda-libmamba-solver 23.7.0 py311h06a4308_0 defaults
conda-package-handling 2.2.0 py311h06a4308_0 defaults
conda-package-streaming 0.9.0 py311h06a4308_0 defaults
conda-standalone 22.11.1 h06a4308_1 defaults
conda-verify 3.4.2 py_1 defaults
conda-wrap 0.2.0 py_0 defaults
condarc 2023.03.1 0 defaults
constructor 3.4.5 py311h06a4308_0 defaults
cryptography 41.0.2 py311h22a60cf_0 defaults
defusedxml 0.7.1 pyhd3eb1b0_0 defaults
filelock 3.9.0 py311h06a4308_0 defaults
fmt 9.1.0 hdb19cb5_0 defaults
future 0.18.3 py311h06a4308_0 defaults
glob2 0.7 pyhd3eb1b0_0 defaults
icu 73.1 h6a678d5_0 defaults
idna 3.4 py311h06a4308_0 defaults
jinja2 3.1.2 py311h06a4308_0 defaults
jsonpatch 1.32 pyhd3eb1b0_0 defaults
jsonpointer 2.1 pyhd3eb1b0_0 defaults
jsonschema 4.17.3 py311h06a4308_0 defaults
jupyter_core 5.3.0 py311h06a4308_0 defaults
krb5 1.20.1 h143b758_1 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
libarchive 3.6.2 h6ac8c49_2 defaults
libcurl 8.2.1 h251f7ec_0 defaults
libedit 3.1.20221030 h5eee18b_0 defaults
libev 4.33 h7f8727e_1 defaults
libffi 3.4.4 h6a678d5_0 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
liblief 0.12.3 h6a678d5_0 defaults
libmamba 1.4.1 h2dafd23_1 defaults
libmambapy 1.4.1 py311h2dafd23_1 defaults
libnghttp2 1.52.0 h2d74bed_1 defaults
libsolv 0.7.22 he621ea3_0 defaults
libssh2 1.10.0 hdbd6064_2 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
libuuid 1.41.5 h5eee18b_0 defaults
libxml2 2.10.4 hf1b16e4_1 defaults
lz4-c 1.9.4 h6a678d5_0 defaults
markupsafe 2.1.1 py311h5eee18b_0 defaults
more-itertools 8.12.0 pyhd3eb1b0_0 defaults
nbformat 5.7.0 py311h06a4308_0 defaults
ncurses 6.4 h6a678d5_0 defaults
openssl 3.0.10 h7f8727e_2 defaults
packaging 23.1 py311h06a4308_0 defaults
patch 2.7.6 h7b6447c_1001 defaults
patchelf 0.17.2 h6a678d5_0 defaults
pcre2 10.37 he7ceb23_1 defaults
pip 23.2.1 py311h06a4308_0 defaults
pkginfo 1.9.6 py311h06a4308_0 defaults
platformdirs 3.10.0 py311h06a4308_0 defaults
pluggy 1.0.0 py311h06a4308_1 defaults
psutil 5.9.0 py311h5eee18b_0 defaults
py-lief 0.12.3 py311h6a678d5_0 defaults
pybind11-abi 4 hd3eb1b0_1 defaults
pycosat 0.6.4 py311h5eee18b_0 defaults
pycparser 2.21 pyhd3eb1b0_0 defaults
pyopenssl 23.2.0 py311h06a4308_0 defaults
pyrsistent 0.18.0 py311h5eee18b_0 defaults
pysocks 1.7.1 py311h06a4308_0 defaults
python 3.11.4 h955ad1f_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
python-fastjsonschema 2.16.2 py311h06a4308_0 defaults
python-libarchive-c 2.9 pyhd3eb1b0_1 defaults
pytz 2022.7 py311h06a4308_0 defaults
pyyaml 6.0 py311h5eee18b_1 defaults
readline 8.2 h5eee18b_0 defaults
reproc 14.2.4 h295c915_1 defaults
reproc-cpp 14.2.4 h295c915_1 defaults
requests 2.31.0 py311h06a4308_0 defaults
requests-toolbelt 1.0.0 py311h06a4308_0 defaults
ruamel.yaml 0.17.21 py311h5eee18b_0 defaults
setuptools 68.0.0 py311h06a4308_0 defaults
setuptools-scm 7.1.0 py311h06a4308_0 defaults
six 1.16.0 pyhd3eb1b0_1 defaults
soupsieve 2.4 py311h06a4308_0 defaults
sqlite 3.41.2 h5eee18b_0 defaults
tk 8.6.12 h1ccaba5_0 defaults
toolz 0.12.0 py311h06a4308_0 defaults
tqdm 4.65.0 py311h92b7b1e_0 defaults
traitlets 5.7.1 py311h06a4308_0 defaults
typing-extensions 4.7.1 py311h06a4308_0 defaults
typing_extensions 4.7.1 py311h06a4308_0 defaults
tzdata 2023c h04d1e81_0 defaults
urllib3 1.26.16 py311h06a4308_0 defaults
wheel 0.38.4 py311h06a4308_0 defaults
xz 5.4.2 h5eee18b_0 defaults
yaml 0.2.5 h7b6447c_0 defaults
yaml-cpp 0.7.0 h295c915_1 defaults
zlib 1.2.13 h5eee18b_0 defaults
zstandard 0.19.0 py311h5eee18b_0 defaults
zstd 1.5.5 hc292b87_0 defaults
```

</details>


Discussion:
@jaimergp Have you see this before?
Nop. It looks like there's some kind of private mirror involved? I assume this is conda-libmamba-solver 23.7.0 too?

conda-build hacks its way to build an `index` dictionary that maps `Dist` objects to `PackageRecord`s. I think we bypass that mechanism partially, so maybe we are not rebuilding it as we reload outputs or something? Are they able to share their recipe, or an equivalent reproducer? I can take a look once I have a way to debug it :D 
It is conda-libmamba-solver 23.7.0.

They get the error if they use conda-build is version 3.26.1 with libmamba solver, but the build works with classic w/ 3.26.1 conda-build. They have not yet tried libmamba with older conda-build.

Not sure if this helps but:

<details>
<summary><code>Dockerfile</code></summary>

```dockerfile
FROM xxxxxx/amazonlinux:2 as base

COPY conda-build-0.0.0-Linux-x86_64.sh /tmp/conda-build.sh
RUN /tmp/conda-build.sh -b
RUN mkdir -p /root/conda-build/pkgs

FROM xxxxxx/amazonlinux:2
COPY --from=base /root/conda-build /root/conda-build
RUN source ~/conda-build/etc/profile.d/conda.sh && conda init
SHELL ["/bin/bash", "-lc"]
```

</details>

<details>
<summary><code>compose.yaml</code></summary>

```yaml
services:
image: conda/conda-build-test
build:
context: .
dockerfile: ../Dockerfile
command: /root/conda-build/bin/conda-build /test
volumes:
- ./test/test:z,ro
```

</details>

This is how they run test and get the error...

```bash
cd test
DOCKER_BUILDKIT=0 docker compose build test
docker compose run test
```
I just merged a PR in conda-forge/staged-recipes that switches from `conda mambabuild` to `conda build` with `conda-libmamba-solver` and it seems to work correctly. See latest comments https://github.com/conda-forge/staged-recipes/issues/23923 for some examples. Those are using `conda=23.7.3`, `conda-build=3.26.1` and `conda-libmamba-solver=23.7.0`, so there's no intrinsic incompatibility between those.

That leads me to thinking that the error is triggered by something in their system. Can I get more details on:

- Which packages are distributed in that custom `conda-build-0.0.0-Linux-x86_64.sh` installer?
- What's in the target `test/` directory? Is there a chance we can see those recipes?

The traceback hints at a problem with `get_upstream_pins`. This call happens early in the `conda build` process. Is this happening right away after calling `conda build` or after successfully processing some packages?

Thanks!
@jaimergp I've added more info to the issue description above to help with debugging!
This is happening right after the calling conda-build.
It seems like something with channel aliases may be involved. After lots of testing with different condarc configurations, a minimal condarc where conda-build works has been found along with several minimal configurations that cause the error.


This minimal condarc works:
-----------------------------------

```
channels:
- https://conda.anaconda.xxxxx/anaconda
solver: libmamba
```

> Note: No channel Alias.

------------------------------------
All following condarc combinations DO NOT work.
-----------------------------------

```
channel_alias: https://conda.anaconda.xxxxx/
channels:
- https://conda.anaconda.xxxxx/anaconda
solver: libmamba
```

-------------------------------
```
channel_alias: https://conda.anaconda.xxxxx/
channels:
  - anaconda
solver: libmamba
```
-------------------------------


In the following case, there is no channel alias but modified default channels

-------------------------------
```
default_channels:
- https://conda.anaconda.xxxxx/anaconda
channels:
- defaults
solver: libmamba
```

------------------------------------------------------
```
channel_alias: https://conda.anaconda.xxxxx/
default_channels:
- anaconda
channels:
- defaults
solver: libmamba
```
---------------------------------------
I've been trying to reproduce this with publicly available resources that mimic the reported configuration but I've had no luck despite matching almost everything 1:1. Details below.

* My base OS is a Docker container for linux-aarch64 (my machine is Apple Silicon). It should make no difference. It can be started like this:

```
$ docker run -it --rm --platform=linux/aarch64 ghcr.io/conda/conda-ci:main-linux-python3.10-conda-forge bash
```

* I've configured the Docker container to report the following info, matching the versions in the first message:

```
$ conda info
                                                                                                                                                                                                                              
     active environment : base                                                                                                                                                                                                
    active env location : /opt/conda                                                                                                                                                                                          
            shell level : 1                                                                                                                                                                                                   
       user config file : /home/test_user/.condarc                                                                                                                                                                            
 populated config files : /opt/conda/.condarc                                                                                                                                                                                 
          conda version : 23.7.3                                                                                                                                                                                              
    conda-build version : 3.26.1.dev37                                                                                                                                                                                        
         python version : 3.10.12.final.0                                                                                                                                                                                     
       virtual packages : __archspec=1=aarch64                                                                                                                                                                                
                          __glibc=2.31=0                                                                                                                                                                                      
                          __linux=5.15.49=0
                          __unix=0=0
       base environment : /opt/conda  (read only)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-aarch64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/test_user/.conda/pkgs
       envs directories : /home/test_user/.conda/envs
                          /opt/conda/envs
               platform : linux-aarch64
             user-agent : conda/23.7.3 requests/2.31.0 CPython/3.10.12 Linux/5.15.49-linuxkit-pr debian/11 glibc/2.31 solver/libmamba conda-libmamba-solver/23.7.0 libmambapy/1.4.1
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False
```

* The image `.condarc` results in the following config. Note I am using the only public Anaconda mirror _I_ know.

```
==> /opt/conda/.condarc <==
channel_alias: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
channels:
  - defaults
default_channels:
  - conda-forge
solver: libmamba
```

* I will try to build the [following test recipe](https://github.com/conda/conda-libmamba-solver/tree/main/tests/data/conda_build_recipes/jedi) (it also uses `python`, which is the package raising `KeyError`).

* I then ran `conda build <test-recipe>/` and it builds successfully:

```
$ conda-build tests/data/conda_build_recipes/jedi
/opt/conda/bin/conda-build:8: DeprecationWarning: conda_build.cli.main_build.main is deprecated and will be removed in 4.0.0. Use `conda build` instead.
  sys.exit(main())
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Attempting to finalize metadata for jedi
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
BUILD START: ['jedi-0.17.2-py310_1.tar.bz2']
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done

## Package Plan ##

  environment location: /home/test_user/conda-bld/jedi_1697102244495/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla


The following NEW packages will be INSTALLED:

    _openmp_mutex:         4.5-2_gnu                 
    bzip2:                 1.0.8-hf897c2e_4          
    ca-certificates:       2023.7.22-hcefe29a_0      
    ld_impl_linux-aarch64: 2.40-h2d8c526_0           
    libffi:                3.4.2-h3557bc0_5          
    libgcc-ng:             13.2.0-hf8544c7_2         
    libgomp:               13.2.0-hf8544c7_2         
    libnsl:                2.0.0-h31becfc_1          
    libsqlite:             3.43.2-h194ca79_0         
    libuuid:               2.38.1-hb4cce97_0         
    libzlib:               1.2.13-h31becfc_5         
    ncurses:               6.4-h2e1726e_0            
    openssl:               3.1.3-h31becfc_0          
    pip:                   23.2.1-pyhd8ed1ab_0       
    python:                3.10.12-hbbe8eec_0_cpython
    readline:              8.2-h8fc344f_1            
    setuptools:            68.2.2-pyhd8ed1ab_0       
    tk:                    8.6.13-h194ca79_0         
    tzdata:                2023c-h71feb2d_0          
    wheel:                 0.41.2-pyhd8ed1ab_0       
    xz:                    5.2.6-h9cdd2b7_0          

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... done
Source cache directory is: /home/test_user/conda-bld/src_cache
Found source in cache: jedi-0.17.2_86ed7d9b75.tar.gz
Extracting download
source tree in: /home/test_user/conda-bld/jedi_1697102244495/work
export PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
export BUILD_PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_build_env
export SRC_DIR=/home/test_user/conda-bld/jedi_1697102244495/work
Processing $SRC_DIR
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: jedi
  Building wheel for jedi (setup.py): started
  Building wheel for jedi (setup.py): finished with status 'done'
  Created wheel for jedi: filename=jedi-0.17.2-py2.py3-none-any.whl size=1424265 sha256=c1dabafe95aca4a06558a4534eb7843df50600e1d4e926ced739635375422e1b
  Stored in directory: /tmp/pip-ephem-wheel-cache-q52fgcmk/wheels/d4/ef/c8/938d955f0557f0fd6af3245376304ee5ce33619bf24850a065
Successfully built jedi
Installing collected packages: jedi
Successfully installed jedi-0.17.2

Resource usage statistics from building jedi:
   Process count: 4
   CPU time: Sys=0:00:00.7, User=0:00:00.9
   Memory: 82.4M
   Disk usage: 28.4K
   Time elapsed: 0:00:04.1


Packaging jedi
Packaging jedi-0.17.2-py310_1
compiling .pyc files...
number of files: 1765
Fixing permissions
INFO :: Time taken to mark (prefix)
        0 replacements in 0 files was 0.14 seconds
Package verification results:
-----------------------------
/tmp/tmpkvgw7lz9/jedi-0.17.2-py310_1.tar.bz2: C1115 Found invalid license "None" in info/index.json
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:linux-aarch64 listdir
DEBUG:conda_index.index.sqlitecache:linux-aarch64 save fs state
DEBUG:conda_index.index:linux-aarch64 find packages to extract
DEBUG:conda_index.index:linux-aarch64 extract 1 packages
DEBUG:conda_index.index.sqlitecache:Found jedi-0.17.2-py310_1.tar.bz2 in cache
INFO:conda_index.index:linux-aarch64 cached 962 KB from 1 packages at 5.19 GB/second
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:noarch listdir
DEBUG:conda_index.index.sqlitecache:noarch save fs state
DEBUG:conda_index.index:noarch find packages to extract
DEBUG:conda_index.index:noarch extract 0 packages
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index:Subdir: linux-aarch64 Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:linux-aarch64 Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json to /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json
INFO:conda_index.index:linux-aarch64 Applying patch instructions
INFO:conda_index.index:linux-aarch64 Writing patched repodata
DEBUG:conda_index.index:linux-aarch64 write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata.json to /home/test_user/conda-bld/linux-aarch64/repodata.json
INFO:conda_index.index:linux-aarch64 Building current_repodata subset
DEBUG:conda_index.index:linux-aarch64 build current_repodata
INFO:conda_index.index:linux-aarch64 Writing current_repodata subset
DEBUG:conda_index.index:linux-aarch64 write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/current_repodata.json to /home/test_user/conda-bld/linux-aarch64/current_repodata.json
INFO:conda_index.index:linux-aarch64 Writing index HTML
DEBUG:conda_index.index:linux-aarch64 write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/index.html to /home/test_user/conda-bld/linux-aarch64/index.html
DEBUG:conda_index.index:linux-aarch64 finish
INFO:conda_index.index:Completed linux-aarch64
INFO:conda_index.index:Subdir: noarch Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:noarch Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata_from_packages.json to /home/test_user/conda-bld/noarch/repodata_from_packages.json
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
DEBUG:conda_index.index:noarch write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata.json to /home/test_user/conda-bld/noarch/repodata.json
INFO:conda_index.index:noarch Building current_repodata subset
DEBUG:conda_index.index:noarch build current_repodata
INFO:conda_index.index:noarch Writing current_repodata subset
DEBUG:conda_index.index:noarch write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/current_repodata.json to /home/test_user/conda-bld/noarch/current_repodata.json
INFO:conda_index.index:noarch Writing index HTML
DEBUG:conda_index.index:noarch write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/index.html to /home/test_user/conda-bld/noarch/index.html
DEBUG:conda_index.index:noarch finish
INFO:conda_index.index:Completed noarch
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
INFO:conda_index.index:Channeldata subdir: linux-aarch64
DEBUG:conda_index.index:linux-aarch64 read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:linux-aarch64 channeldata finished
INFO:conda_index.index:Channeldata subdir: noarch
DEBUG:conda_index.index:noarch read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:noarch channeldata finished
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/index.html to /home/test_user/conda-bld/index.html
DEBUG:conda_index.index:write channeldata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/channeldata.json to /home/test_user/conda-bld/channeldata.json
TEST START: /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:linux-aarch64 listdir
DEBUG:conda_index.index.sqlitecache:linux-aarch64 save fs state
DEBUG:conda_index.index:linux-aarch64 find packages to extract
DEBUG:conda_index.index:linux-aarch64 extract 0 packages
INFO:conda_index.index:linux-aarch64 cached 0 B from 0 packages at 0 B/second
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:noarch listdir
DEBUG:conda_index.index.sqlitecache:noarch save fs state
DEBUG:conda_index.index:noarch find packages to extract
DEBUG:conda_index.index:noarch extract 0 packages
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index:Subdir: linux-aarch64 Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:linux-aarch64 Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json to /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json
INFO:conda_index.index:linux-aarch64 Applying patch instructions
INFO:conda_index.index:linux-aarch64 Writing patched repodata
DEBUG:conda_index.index:linux-aarch64 write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata.json to /home/test_user/conda-bld/linux-aarch64/repodata.json
INFO:conda_index.index:linux-aarch64 Building current_repodata subset
DEBUG:conda_index.index:linux-aarch64 build current_repodata
INFO:conda_index.index:linux-aarch64 Writing current_repodata subset
DEBUG:conda_index.index:linux-aarch64 write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/current_repodata.json to /home/test_user/conda-bld/linux-aarch64/current_repodata.json
INFO:conda_index.index:linux-aarch64 Writing index HTML
DEBUG:conda_index.index:linux-aarch64 write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/index.html to /home/test_user/conda-bld/linux-aarch64/index.html
DEBUG:conda_index.index:linux-aarch64 finish
INFO:conda_index.index:Completed linux-aarch64
INFO:conda_index.index:Subdir: noarch Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:noarch Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata_from_packages.json to /home/test_user/conda-bld/noarch/repodata_from_packages.json
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
DEBUG:conda_index.index:noarch write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata.json to /home/test_user/conda-bld/noarch/repodata.json
INFO:conda_index.index:noarch Building current_repodata subset
DEBUG:conda_index.index:noarch build current_repodata
INFO:conda_index.index:noarch Writing current_repodata subset
DEBUG:conda_index.index:noarch write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/current_repodata.json to /home/test_user/conda-bld/noarch/current_repodata.json
INFO:conda_index.index:noarch Writing index HTML
DEBUG:conda_index.index:noarch write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/index.html to /home/test_user/conda-bld/noarch/index.html
DEBUG:conda_index.index:noarch finish
INFO:conda_index.index:Completed noarch
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
INFO:conda_index.index:Channeldata subdir: linux-aarch64
DEBUG:conda_index.index:linux-aarch64 read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:linux-aarch64 channeldata finished
INFO:conda_index.index:Channeldata subdir: noarch
DEBUG:conda_index.index:noarch read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:noarch channeldata finished
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/index.html to /home/test_user/conda-bld/index.html
DEBUG:conda_index.index:write channeldata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/channeldata.json to /home/test_user/conda-bld/channeldata.json
Adding in variants from /tmp/tmp86g7_0xu/info/recipe/conda_build_config.yaml
Renaming work directory '/home/test_user/conda-bld/jedi_1697102244495/work' to '/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64'
shutil.move(work)=/home/test_user/conda-bld/jedi_1697102244495/work, dest=/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64)
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... done

## Package Plan ##

  environment location: /home/test_user/conda-bld/jedi_1697102244495/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol


The following NEW packages will be INSTALLED:

    _openmp_mutex:         4.5-2_gnu                       
    bzip2:                 1.0.8-hf897c2e_4                
    ca-certificates:       2023.7.22-hcefe29a_0            
    jedi:                  0.17.2-py310_1             local
    ld_impl_linux-aarch64: 2.40-h2d8c526_0                 
    libffi:                3.4.2-h3557bc0_5                
    libgcc-ng:             13.2.0-hf8544c7_2               
    libgomp:               13.2.0-hf8544c7_2               
    libnsl:                2.0.0-h31becfc_1                
    libsqlite:             3.43.2-h194ca79_0               
    libuuid:               2.38.1-hb4cce97_0               
    libzlib:               1.2.13-h31becfc_5               
    ncurses:               6.4-h2e1726e_0                  
    openssl:               3.1.3-h31becfc_0                
    parso:                 0.7.1-pyh9f0ad1d_0              
    pip:                   23.2.1-pyhd8ed1ab_0             
    python:                3.10.12-hbbe8eec_0_cpython      
    python_abi:            3.10-4_cp310                    
    readline:              8.2-h8fc344f_1                  
    setuptools:            68.2.2-pyhd8ed1ab_0             
    tk:                    8.6.13-h194ca79_0               
    tzdata:                2023c-h71feb2d_0                
    wheel:                 0.41.2-pyhd8ed1ab_0             
    xz:                    5.2.6-h9cdd2b7_0                

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol
export SRC_DIR=/home/test_user/conda-bld/jedi_1697102244495/test_tmp
import: 'jedi'
import: 'jedi.api'
import: 'jedi.common'
import: 'jedi.inference'
import: 'jedi.inference.compiled'
import: 'jedi.inference.compiled.subprocess'
import: 'jedi.inference.gradual'
import: 'jedi.inference.value'
import: 'jedi.plugins'
import: 'jedi'
import: 'jedi.api'
import: 'jedi.common'
import: 'jedi.inference'
import: 'jedi.inference.compiled'
import: 'jedi.inference.compiled.subprocess'
import: 'jedi.inference.gradual'
import: 'jedi.inference.value'
import: 'jedi.plugins'

Resource usage statistics from testing jedi:
   Process count: 2
   CPU time: Sys=0:00:00.0, User=0:00:00.0
   Memory: 10.5M
   Disk usage: 16B
   Time elapsed: 0:00:02.0


TEST END: /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
Renaming work directory '/home/test_user/conda-bld/jedi_1697102244495/work' to '/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64_main_build_loop'
shutil.move(work)=/home/test_user/conda-bld/jedi_1697102244495/work, dest=/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:


# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "jedi-0.17.2-py310_1": {
    "recipe": {}
  }
}


####################################################################################
Resource usage summary:

Total time: 0:01:08.0
CPU usage: sys=0:00:00.7, user=0:00:00.9
Maximum memory usage observed: 82.4M
Total disk usage observed (not including envs): 28.4K


####################################################################################
Source and build intermediates have been left in /home/test_user/conda-bld.
There are currently 3 accumulated.
To remove them, you can run the ```conda build purge``` command
```

FWIW, I also tried the most recent versions of every component (conda 23.9.0, conda-build 23.7.0, conda-libmamba-solver 23.9.1, libmambapy 1.5.1) and it works too.

I'll keep trying other combinations but I would have expected to reproduce this more easily.
One thing to notice is that the line raising `KeyError` is part of this code block:

https://github.com/conda/conda-build/blob/c2bfe2b44ad34b06d8930aa4239995367f3eaf12/conda_build/render.py#L372-L391

"Vile hack" doesn't give me too much confidence and is also a "last resort" strategy, part of the `run_exports` / pins metadata calculation. Since I matched the config but had to come up with a demo recipe that might not representative of the code paths being triggered in `conda-build`... is there a chance we can get some insight into what kind of recipes are being built? Namely:

* Is it single output or does it have multiple outputs? In other words, does it contain and `outputs:` key?
* What kind of dependencies are present in `build` and `host` environments? Things like compilers or compiled libraries that might be exporting pins to `run`?
* Is it using Jinja functions like `pin_subpackage` or `pin_compatible`?

Thanks!
> I've been trying to reproduce this with publicly available resources that mimic the reported configuration but I've had no luck despite matching almost everything 1:1. Details below.

@jaimergp from my experience with this bug, you should test in an isolated network environment where you can talk to a private conda-forge mirror, but nothing else.  If you have general internet access, you probably won't trigger the bug.

Edit: If network connectivity is indeed a factor, maybe we could integrate a tools like https://gitlab.freedesktop.org/hadess/resolvconf-override/ or https://github.com/pagekite/libunaccept into the test suite to test to ensure we test for regressions in this space.  Monkey-patching the socket module is another option.
After some fiddling it does reproduce with the proposed configuration above, but to hit the buggy code path you need a clean cache via `conda clean -pty`; otherwise the tarball is found earlier and it doesn't try to download it on the spot. I have found a simple fix that I'm going to submit to the CI to see if it breaks anything.

Resolving patch:
diff --git a/conda_build/render.py b/conda_build/render.py
--- a/conda_build/render.py
+++ b/conda_build/render.py
@@ -384,7 +384,7 @@ def execute_download_actions(m, actions, env, package_subset=None, require_files
             with utils.LoggingContext():
                 pfe.execute()
             for pkg_dir in pkgs_dirs:
-                _loc = os.path.join(pkg_dir, index[pkg].fn)
+                _loc = os.path.join(pkg_dir, index.get(pkg, pkg).fn)
                 if os.path.isfile(_loc):
                     pkg_loc = _loc
                     break


Test patch: