Repository: astronomer/astro-provider-venv (language: python)

Issue:
Snowpark method for airflow-extensions is broken.
**Describe the bug**

This method was using `python3.8:slim` image which is based on bookworm debian. Recently the new image for python has bumped the glibc version to 2.36. We can check that by `ldd --version`. Whereas our base runtime images are built on debian11 which only has glibc version 2.31.

<!--
A clear and concise description of what the bug is.
On errors, please provide the output of the console
-->

```
 => ERROR [stage-0 13/17] RUN /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark                                                    0.1s
------
 > [stage-0 13/17] RUN /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark:
#0 0.115 + /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark
#0 0.116 /usr/local/bin/python3.8: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/local/bin/python3.8)
#0 0.116 /usr/local/bin/python3.8: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /usr/local/bin/../lib/libpython3.8.so.1.0)
#0 0.116 /usr/local/bin/python3.8: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/local/bin/../lib/libpython3.8.so.1.0)
#0 0.116 /usr/local/bin/python3.8: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/local/bin/../lib/libpython3.8.so.1.0)
#0 0.116 /usr/local/bin/python3.8: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/local/bin/../lib/libpython3.8.so.1.0)
------
Dockerfile:31
--------------------
  29 |     RUN mkdir -p /home/astro/.venv/snowpark
  30 |     COPY nre_req.txt /home/astro/.venv/snowpark/requirements.txt
  31 | >>> RUN /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark
  32 |     ENV ASTRO_PYENV_snowpark /home/astro/.venv/snowpark/bin/python
  33 |     RUN --mount=type=cache,target=/home/astro/.cache/pip /home/astro/.venv/snowpark/bin/pip --cache-dir=/home/astro/.cache/pip install -r /home/astro/.venv/snowpark/requirements.txt
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -e -u -x -c /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark" did not complete successfully: exit code: 1
```

**What CLI Version did you experience this bug?**

Version 1.16.1. But Cli version doesn't matter in this scenario.
<!--
Run the command `astro version` to get the cli version
-->

**This CLI bug is related to which Astronomer Platform?**

- [ ] Astro
- [ ] Software
- [x] None/Unknown

**What Operating System is the above CLI installed on?**

**🪜 Steps To Reproduce**

Follow the [documentation](https://docs.astronomer.io/learn/external-python-operator#step-2-configure-your-astro-project)
<!--
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
-->




Discussion / hints:
A solution would be to use `python:<version>-bullseye` as it matches the base OS with runtime images.

Working solution:

```
FROM quay.io/astronomer/astro-runtime:8.4.0
USER root
COPY --link --from=python:3.8-bullseye /usr/local/bin/*3.8* /usr/local/bin/
COPY --link --from=python:3.8-bullseye /usr/local/include/python3.8* /usr/local/include/python3.8
COPY --link --from=python:3.8-bullseye /usr/local/lib/pkgconfig/*3.8* /usr/local/lib/pkgconfig/
COPY --link --from=python:3.8-bullseye /usr/local/lib/*3.8*.so* /usr/local/lib/
COPY --link --from=python:3.8-bullseye /usr/local/lib/python3.8 /usr/local/lib/python3.8
RUN /sbin/ldconfig /usr/local/lib
RUN ln -s /usr/local/include/python3.8 /usr/local/include/python3.8m
USER astro
RUN mkdir -p /home/astro/.venv/snowpark
COPY snowpark-requirements.txt /home/astro/.venv/snowpark/requirements.txt
RUN /usr/local/bin/python3.8 -m venv /home/astro/.venv/snowpark
```
Seems like the following works:

```
# syntax=quay.io/astronomer/airflow-extensions:v1
FROM quay.io/astronomer/astro-runtime:8.4.0
PYENV 3.9.16 googleads requirements.txt
```

Because 3.9.16 was based on bullseye but new image 3.9.17 is based on bookworm.
And tags `3.9-slim` is equivalent to `3.9.17-slim-bookworm`, `3.9-slim-bookworm`, `3.9.17-slim`.

Hmmm, so switching it to `3.8-slim-bullseye` would work -- except for when we upgrade the runtime images to use Bookworm (which we likely will soon. I'll chat with the team and see if that would be a Runtime 9.0 or if we'll do it in-stream for the existing releases etc.)
Would it not be possible to automate selection of tags by checking `/etc/os-release` of base image on fly ?
@abhishekbhakat Not trivially -- at the time we build the dockerfile instructions we don't actually have access to the image or any of the files in it -- it's just how docker builders work.

Fix patch:
diff --git a/buildkit/internal/transform/transforms.go b/buildkit/internal/transform/transforms.go
index ac01593..68ce38c 100644
--- a/buildkit/internal/transform/transforms.go
+++ b/buildkit/internal/transform/transforms.go
@@ -16,11 +16,11 @@ import (
 
 const (
 	pythonEnvTemplate = `USER root
-COPY --link --from=python:{{.PythonVersion}}-slim /usr/local/bin/*{{.PythonMajorMinor}}* /usr/local/bin/
-COPY --link --from=python:{{.PythonVersion}}-slim /usr/local/include/python{{.PythonMajorMinor}}* /usr/local/include/python{{.PythonMajorMinor}}
-COPY --link --from=python:{{.PythonVersion}}-slim /usr/local/lib/pkgconfig/*{{.PythonMajorMinor}}* /usr/local/lib/pkgconfig/
-COPY --link --from=python:{{.PythonVersion}}-slim /usr/local/lib/*{{.PythonMajorMinor}}*.so* /usr/local/lib/
-COPY --link --from=python:{{.PythonVersion}}-slim /usr/local/lib/python{{.PythonMajorMinor}} /usr/local/lib/python{{.PythonMajorMinor}}
+COPY --link --from=python:{{.PythonVersion}}-{{.PythonFlavour}} /usr/local/bin/*{{.PythonMajorMinor}}* /usr/local/bin/
+COPY --link --from=python:{{.PythonVersion}}-{{.PythonFlavour}} /usr/local/include/python{{.PythonMajorMinor}}* /usr/local/include/python{{.PythonMajorMinor}}
+COPY --link --from=python:{{.PythonVersion}}-{{.PythonFlavour}} /usr/local/lib/pkgconfig/*{{.PythonMajorMinor}}* /usr/local/lib/pkgconfig/
+COPY --link --from=python:{{.PythonVersion}}-{{.PythonFlavour}} /usr/local/lib/*{{.PythonMajorMinor}}*.so* /usr/local/lib/
+COPY --link --from=python:{{.PythonVersion}}-{{.PythonFlavour}} /usr/local/lib/python{{.PythonMajorMinor}} /usr/local/lib/python{{.PythonMajorMinor}}
 RUN /sbin/ldconfig /usr/local/lib
 # hack for python <= 3.7
 RUN ln -s /usr/local/include/python{{.PythonMajorMinor}} /usr/local/include/python{{.PythonMajorMinor}}m
@@ -36,6 +36,8 @@ ENV ASTRO_PYENV_{{.Name}} /home/astro/.venv/{{.Name}}/bin/python
 	argCommand        = "ARG"
 	pyenvCommand      = "PYENV"
 	astroRuntimeImage = "quay.io/astronomer/astro-runtime"
+
+	defaultImageFlavour = "slim-bullseye"
 )
 
 var (
@@ -51,6 +53,7 @@ type Transformer struct {
 type virtualEnv struct {
 	Name             string
 	PythonVersion    string
+	PythonFlavour    string
 	PythonMajorMinor string
 	RequirementsFile string
 }
@@ -167,6 +170,8 @@ func parsePyenvDirective(s string) (*virtualEnv, error) {
 	}
 	env := &virtualEnv{
 		PythonVersion: tokens[1],
+		// For now we just hardcode this -- will add an option later
+		PythonFlavour: defaultImageFlavour,
 		Name:          tokens[2],
 	}
 	if len(tokens) > 3 {


Test patch:
diff --git a/buildkit/internal/transform/transforms_test.go b/buildkit/internal/transform/transforms_test.go
index 21bd589..699942f 100644
--- a/buildkit/internal/transform/transforms_test.go
+++ b/buildkit/internal/transform/transforms_test.go
@@ -22,11 +22,11 @@ ARG baseimage
 FROM ${baseimage}
 `
 	expectedDockerfile := `USER root
-COPY --link --from=python:3.8-slim /usr/local/bin/*3.8* /usr/local/bin/
-COPY --link --from=python:3.8-slim /usr/local/include/python3.8* /usr/local/include/python3.8
-COPY --link --from=python:3.8-slim /usr/local/lib/pkgconfig/*3.8* /usr/local/lib/pkgconfig/
-COPY --link --from=python:3.8-slim /usr/local/lib/*3.8*.so* /usr/local/lib/
-COPY --link --from=python:3.8-slim /usr/local/lib/python3.8 /usr/local/lib/python3.8
+COPY --link --from=python:3.8-slim-bullseye /usr/local/bin/*3.8* /usr/local/bin/
+COPY --link --from=python:3.8-slim-bullseye /usr/local/include/python3.8* /usr/local/include/python3.8
+COPY --link --from=python:3.8-slim-bullseye /usr/local/lib/pkgconfig/*3.8* /usr/local/lib/pkgconfig/
+COPY --link --from=python:3.8-slim-bullseye /usr/local/lib/*3.8*.so* /usr/local/lib/
+COPY --link --from=python:3.8-slim-bullseye /usr/local/lib/python3.8 /usr/local/lib/python3.8
 RUN /sbin/ldconfig /usr/local/lib
 
 RUN ln -s /usr/local/include/python3.8 /usr/local/include/python3.8m
@@ -38,11 +38,11 @@ ENV ASTRO_PYENV_venv1 /home/astro/.venv/venv1/bin/python
 RUN --mount=type=cache,target=/home/astro/.cache/pip /home/astro/.venv/venv1/bin/pip --cache-dir=/home/astro/.cache/pip install -r /home/astro/.venv/venv1/requirements.txt
 COPY foo bar
 USER root
-COPY --link --from=python:3.10-slim /usr/local/bin/*3.10* /usr/local/bin/
-COPY --link --from=python:3.10-slim /usr/local/include/python3.10* /usr/local/include/python3.10
-COPY --link --from=python:3.10-slim /usr/local/lib/pkgconfig/*3.10* /usr/local/lib/pkgconfig/
-COPY --link --from=python:3.10-slim /usr/local/lib/*3.10*.so* /usr/local/lib/
-COPY --link --from=python:3.10-slim /usr/local/lib/python3.10 /usr/local/lib/python3.10
+COPY --link --from=python:3.10-slim-bullseye /usr/local/bin/*3.10* /usr/local/bin/
+COPY --link --from=python:3.10-slim-bullseye /usr/local/include/python3.10* /usr/local/include/python3.10
+COPY --link --from=python:3.10-slim-bullseye /usr/local/lib/pkgconfig/*3.10* /usr/local/lib/pkgconfig/
+COPY --link --from=python:3.10-slim-bullseye /usr/local/lib/*3.10*.so* /usr/local/lib/
+COPY --link --from=python:3.10-slim-bullseye /usr/local/lib/python3.10 /usr/local/lib/python3.10
 RUN /sbin/ldconfig /usr/local/lib
 
 RUN ln -s /usr/local/include/python3.10 /usr/local/include/python3.10m