Python Packages in Artifact Registry
By: Lukas Karlsson
Tags: Gcp, Cloud Functions, Python, Artifact Registry, Serverless

Today, Google announced a public preview of support for Python packages (using the Pypi repository format) in Artifact Registry. With this feature, you can create a private repository in your project where you can publish your packages, instead of pushing them to the public repository at Pypi.org.

Uploading Python Packages with Cloud Build

Google recommends authenticating to Artifact Registry PyPi repositories using the Python keyring library. Here is an example of how you can do this in your cloudbuild.yaml :



steps:

- name: python:3.9

entrypoint: /bin/bash

dir: package

args:

- -c

- |

pip install --upgrade pip setuptools twine wheel

pip install keyring

pip install keyrings.google-artifactregistry-auth

mkdir -p ~/.config/pip

printf "[distutils]

index-servers =

${_REPO}



[${_REPO}]

repository:

printf "[global]

index-url =

python setup.py sdist bdist_wheel

twine upload --repository-url ---steps:- name: python:3.9entrypoint: /bin/bashdir: packageargs:- -c- |pip install --upgrade pip setuptools twine wheelpip install keyringpip install keyrings.google-artifactregistry-authmkdir -p ~/.config/pipprintf "[distutils]

index-servers =

${_REPO}



[${_REPO}]

repository: https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/

" > ~/.pypircprintf "[global]

index-url = https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/simple/

" > ~/.config/pip/pip.confpython setup.py sdist bdist_wheeltwine upload --repository-url https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/ dist/*

This Cloud Build config handles the following steps:

install setuptools , twine , wheel , keyring and the Artifacts Registry backend

, , , and the Artifacts Registry backend create the ~/.config/pip directory

directory populate the pip.conf

build the wheels and sdists

upload to the registry

and it assumes the PROJECT_ID of the Pypi repository is the same project where Cloud Build is running and depends on the following substitutions:

_REPO : the name of the Pypi repository ( mypypi for example)

: the name of the Pypi repository ( for example) _REGION : the region

Your Cloud Build GitHub Trigger could look something like this:

---

description: Upload Package to Artifact Registry PyPi Repository filename: package/cloudbuild.yaml github:

name: my-package

owner: my-org

push:

branch: ^main$



includedFiles:

- package/setup.py name: upload-package substitutions:

_REGION: us-east4

_REPO: mypypi

This way, when you push a new commit to the main branch in GitHub with changes to the package/setup.py file (since you need to update the version anytime you push to a repository), the Cloud Build job will run and will update the package in the Artifact Registry Repository.

Using Python Packages with Cloud Functions

What if you want to use these private Artifact Registry packages in your Cloud Functions? Google provides instructions for how to install a package, so you could use that to install the package in your local environment or to include it in one of container images which you could then use with Cloud Run. But what about Cloud Functions?

The first undocumented trick you need to know is there is support for authenticating with an oauth2accesstoken by adding the following line to the beginning of your requirements.txt .

--extra-index-url https://oauth2accesstoken:{{ACCESS_TOKEN}}@{{REGION}}-pypi.pkg.dev/{{PROJECT}}/{{REPO}}/simple/

where {{ACCESS_TOKEN}} is replaced by the output of:

gcloud auth print-access-token

and {{REGION}} , {{PROJECT}} and {{REPO}} describe the location of your Artifact Registry Pypi repository.

In order to get this working in a cloudbuild.yaml , I did a couple of other things.

Renamed my requirements.txt to requirements.txt.tmpl and added the templated line above with the {{}}’s . Added requirements.txt to my .gitignore so that the file with the access token never gets committed to GitHub. Added !requirements.txt to my .gcloudignore so that the requirements.txt file does get pushed when I run gcloud functions deploy . Added a shell script to prepare the requirements.txt file called prepare-requirements.sh (see below). Added a step to the cloudbuild.yaml that deploys my Cloud Function

Here is an example of the prepare-requirements.sh shell script that turns the requirements.txt.tmpl into the requirements.txt with the --extra-index-url populated.

#!/bin/bash ACCESS_TOKEN="$(gcloud auth print-access-token)"

REGION="${REGION:="us-east4"}"

REPO="${REPO:="mypypi"}"

PROJECT="${PROJECT:="broad-gaia-karlsson"}" sed -e "s/{{ACCESS_TOKEN}}/${ACCESS_TOKEN}/" \

-e "s/{{PROJECT}}/${PROJECT}/" \

-e "s/{{REGION}}/${REGION}/" \

-e "s/{{REPO}}/${REPO}/" \

requirements.txt.tmpl > requirements.txt echo "Created requirements.txt from requirements.txt.tmpl"

Then, in my cloudbuild.yaml for deploying the Cloud Function, I added a step at the beginning to prepare the requirements.txt .

---

steps:

- name: gcr.io/google.com/cloudsdktool/cloud-sdk:alpine

dir: functions/my-fancy-function

entrypoint: bash

args:

- ../../scripts/prepare-requirements.sh

env:

- PROJECT=${PROJECT_ID}

- REGION=${_REGION}

- REPO=${_REPO} - name: gcr.io/google.com/cloudsdktool/cloud-sdk:alpine

dir: functions/my-fancy-function

entrypoint: gcloud

args:

- functions

- deploy

- my-fancy-function

- --memory=1024MB

- --project=${PROJECT_ID}

- --region=us-east4

- --runtime=python38

- --timeout=300

- --trigger-http

- -q

Finally, I created a Cloud Build GitHub Trigger to deploy the Cloud Function:

---

description: Deploy my-fancy-function filename: functions/my-fancy-function/cloudbuild.yaml github:

name: my-package

owner: my-org

push:

branch: ^main$



includedFiles:

- functions/my-fancy-function/** name: deploy-my-fancy-function substitutions:

_REGION: us-east4

_REPO: mypypi

Now, when I push a new commit to the main branch in GitHub, if it includes changes to this Cloud Function, the Cloud Build Job will run and will populate the requirements.txt and deploy the function.

Conclusion

Support for Pypi packages in Artifact Registry is currently in the Preview launch stage, so it is very possible that this process will change, almost certainly to make it easier to do than the current keychain approach. In the meantime, you should be able to start using private Pypi repositories with your Cloud Functions today!

Check back here for updates as things change!