fix broken Docker image build
State: closed

- [X] closes #45807

This PR adds a missing flag to successfully build the pandas Docker image.

Labels:
name: Build
description: Library building on various platforms

Comments:
I have not tested the environment itself, but I could build the docker image properly.
```bash
docker build --tag pandas-pr45807-env .
Successfully built ba38c2d0969f
Successfully tagged pandas-pr45807-env:latest
```

I think a test to test the build of this image could avoid the same mistake.
sure would welcome a test of this. 
@jreback  
> sure would welcome a test of this.

I expect that the the test is a shell script which runs only `docker build --tag pandas-dev-env .` (maybe with `--no-cache`?)

Can you give me a pointer where I should create such a test?
> Can you give me a pointer where I should create such a test?

Maybe the easiest thing is to create a "no-op" Github action (in `code-checks.yml`) that builds the Dockerfile: https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action 
@mroeschke I created a new job in the mentioned `code-checks.yml`. 

I have run the new build job twice and both failed:
- [first run](https://github.com/pandas-dev/pandas/runs/5169893922?check_suite_focus=true#step:3:1051)
- [second run](https://github.com/pandas-dev/pandas/runs/5170008127?check_suite_focus=true#step:3:1050)

Both failed with the same `CondaHTTPError: HTTP 404 NOT FOUND` errors:

> CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-2.0.11-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.4-pyh9f0ad1d_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/idna-3.3-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/pip-22.0.3-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/pycparser-2.21-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/pyopenssl-22.0.0-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/requests-2.27.1-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/tqdm-4.62.3-pyhd8ed1ab_0.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/urllib3-1.26.8-pyhd8ed1ab_1.conda>
CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.conda>

Is there something wrong?

Another problem which I currently see is that in the Dockerfile the [main branch](https://github.com/pandas-dev/pandas/blob/main/Dockerfile#L27-L31) is used to build the Docker image. Therefore, changes in a PR outside of the Dockerfile are only taken into account after they are merged into the main branch. This means that even if the job runs successfully for a PR, it can fail after the merge into the main branch.
For the 404, maybe make sure `-c conda-forge` is passed when installing the deps from `environment.yml`.

I'm guessing this Dockerfile shouldn't change much ideally so I think it's a good first pass test for this PR
@mroeschke 
For me it looks like that the `conda-forge` channel is used (as configured). The problem seems to be that it tries to download non-existing `conda` files instead of the existing `tar.bz2` files.

What I tried in the meantime:

- I added `conda config --set use_only_tar_bz2 true`
	-> flag was added to `/root/.condarc`: https://github.com/pandas-dev/pandas/runs/5183038474?check_suite_focus=true#step:4:569
	-> results into the same problem: 11 non-downloadable conda files (it seems the flag isn't taken into account)
	
- I updated conda before installing mamba: `conda update -n base conda`
	-> results into the same problem: but 10 non-downloadable conda files instead of 11
	
Since I am not familiar with conda and mamba, I would need help at this point. I have enabled additional debug output. Here is the output of the last failed build:

- [conda info + conda config](https://github.com/pandas-dev/pandas/runs/5184695517?check_suite_focus=true#step:4:546)
- [mamba env update with debug output](https://github.com/pandas-dev/pandas/runs/5184695517?check_suite_focus=true#step:4:670)

Maybe you can find something. Building the Dockerfile via `docker build ...` works on my local machine.
I also recognized that the `Dockerfile` and the `ci/setup_env.sh` use different steps to configure the setup.
For now, I think it's best to just make the Dockerfile changes here and open a new issue about testing the Dockerfile in the CI (i.e. revert the Github Action job changes).

I don't use the Dockerfile for local development so I am not sure where the issue might be
I digged a little bit deeper, to identify why the problem didn't happen locally. 

Starting with `quay.io/condaforge/miniforge3:4.11.0-2` (released 2022-02-11) I see the strange `download non-existant .conda URLs` behavior. The older versions work.

Here is a reduced Dockerfile to demo the problem:
```
# first non working base image
FROM quay.io/condaforge/miniforge3:4.11.0-2

# last working base image
# FROM quay.io/condaforge/miniforge3:4.11.0-0

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive

# Configure apt and install packages
RUN apt-get update \
    && apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
    #
    # Verify git, process tools, lsb-release (common in install instructions for CLIs) installed
    && apt-get -y install git iproute2 procps iproute2 lsb-release \
    #
    # cleanup
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=dialog

COPY environment.yml environment.yml

RUN conda install -y mamba
RUN mamba env update -n base -f environment.yml
```

I will pin the base image and check if the test runs successfully.

I also found an issue that describes the problem that occurs when I use the latest base image: [Mamba tries to download non-existant .conda URLs from conda-meta](https://github.com/mamba-org/mamba/issues/1497)
@mroeschke The PR is ready for review from my side.
Excellent! Thanks @rendner 
Thank you @rendner! I was having trouble getting docker to build, but I assumed it was on my end, so just gave up because I didn't want to dig into the weeds. Thank you for doing this, it works now for me!
The latest version `quay.io/condaforge/miniforge3:4.11.0-4` (released Feb 26, 2022) works again. 

This seems to be the commit which temporary solves the problem: https://github.com/conda-forge/miniforge/commit/5429a0fe6e37cc996327be9e5496df2d1f011cba

@mroeschke 
We could unpin the version of the base image or wait until it is fully resolved. What do you think?

*Note:*
After unpinning the base image, users who have the broken image `quay.io/condaforge/miniforge3:4.11.0-2` as the latest version locally must delete the image or use build option [--pull](https://docs.docker.com/engine/reference/commandline/build/#options) to be able to build successfully.
Sure. As long as our CI check can replicate a Docker build with an image that works correctly, updating the version should be fine