Package: astro-provider-venv
Apache Airflow virtual envs made easyMaking it easy to run tasks in isolated python virtual environments (venv) in Dockerfiles.
  Maintained with ❤️ by Astronomer.Let's say you want to be able to run an Airflow task against Snowflake's Snowpark -- which requires Python 3.8.With the addition of the ExternalPythonOperator in Airflow 2.4 this is possible, but managing the build process to get clean, quick Docker builds can take a lot of plumbing.This repo provides a nice packaged solution to it, that plays nicely with Docker image caching.SynopsisCreate a requirements.txt fileFor example,snowpark-requirements.txtsnowflake-snowpark-python[pandas]

# To get credentials out of a connection we need these in the venv too sadly
apache-airflow
psycopg2-binary
apache-airflow-providers-snowflakeUse our custom Docker build frontend# syntax=quay.io/astronomer/airflow-extensions:v1FROMquay.io/astronomer/astro-runtime:7.2.0-basePYENV3.8snowparksnowpark-requirements.txtNote: That first# syntax=comment is important, don't leave it out!Read more about thenewPYENVinstructionUse it in a DAGfrom__future__importannotationsimportsysfromairflowimportDAGfromairflow.decoratorsimporttaskfromairflow.utils.timezoneimportdatetimewithDAG(dag_id="astro_snowpark",schedule=None,start_date=datetime(2022,1,1),catchup=False,tags=["example"],)asdag:@taskdefprint_python():print(f"My python version is{sys.version}")@task.venv("snowpark")defsnowpark_task():fromairflow.providers.snowflake.hooks.snowflakeimportSnowflakeHookfromsnowflake.snowparkimportSessionprint(f"My python version is{sys.version}")hook=SnowflakeHook("snowflake_default")conn_params=hook._get_conn_params()session=Session.builder.configs(conn_params).create()tables=session.sql("show tables").collect()print(tables)df_table=session.table("sample_product_data")print(df_table.show())returndf_table.to_pandas()@taskdefanalyze(df):print(f"My python version is{sys.version}")print(df.head(2))print_python()>>analyze(snowpark_task())RequirementsThis needs Apache Airflow 2.4+ for theExternalPythonOperatorto work.Requirements for building Docker imagesThis needs thebuildkitbackend for Docker.It is enabled by default for Docker Desktop users; Linux users will need to enable it:To set the BuildKit environment variable when running the docker build command, run:DOCKER_BUILDKIT=1dockerbuild.To enable docker BuildKit by default, set daemon configuration in /etc/docker/daemon.json feature to true and restart the daemon. If the daemon.json file doesn’t exist, create new file called daemon.json and then add the following to the file.{"features":{"buildkit":true}}And restart the Docker daemon.The syntax extension also currently expects to find apackages.txtandrequirements.txtin the Docker context directory (these can be empty by default).ReferencePYENVDocker instructionThePYENVcommand adds a Python Virtual Environment, running on the specified Python version to the docker image, and optionally install packages from a requirements.txtIt has the following syntax:PYENV<python-version><venv-name>[<reqs-file>]The requirements file is optional, so one can install a bare Python environment with something like:PYENV3.10venv1@task.venvdecoratorTODO! Write the decorator, then fill out docs!In This Repobuildkit/This contains the cusotm  Docker BuildKit frontend (see thisblogfor details) adds a new custom commandPYENVthat can be used inside Dockerfiles to install new Python versions and virtual environments with custom dependencies.provider/This contains an Apache Airflow provider that providers the@task.venvdecorator.The Gory Detailsa.k.a. How do I do this all manually?The# syntaxline tells buildkit to user our Build frontend to process the Dockerfile into instructions.The example Dockerfile above gets converted into roughly following instructionsUSERrootCOPY--link--from=python:3.8-slim/usr/local/bin/*3.8*/usr/local/bin/COPY--link--from=python:3.8-slim/usr/local/include/python3.8*/usr/local/include/python3.8COPY--link--from=python:3.8-slim/usr/local/lib/pkgconfig/*3.8*/usr/local/lib/pkgconfig/COPY--link--from=python:3.8-slim/usr/local/lib/*3.8*.so*/usr/local/lib/COPY--link--from=python:3.8-slim/usr/local/lib/python3.8/usr/local/lib/python3.8RUN/sbin/ldconfig/usr/local/libRUNln-s/usr/local/include/python3.8/usr/local/include/python3.8mUSERastroRUNmkdir-p/home/astro/.venv/snowparkCOPYreqs/venv1.txt/home/astro/.venv/snowpark/requirements.txtRUN/usr/local/bin/python3.8-mvenv--system-site-packages/home/astro/.venv/snowparkENVASTRO_PYENV_snowpark/home/astro/.venv/snowpark/bin/pythonRUN--mount=type=cache,target=/home/astro/.cache/pip/home/astro/.venv/snowpark/bin/pip--cache-dir=/home/astro/.cache/pipinstall-r/home/astro/.venv/snowpark/requirements.txtThe final part of this puzzle from the Airflow operator is to look up the path topythonin the created venv using theASTRO_PYENV_*environment variable:@task.external_python(python=os.environ["ASTRO_PYENV_snowpark"])defsnowpark_task():...