Package: proxpi
proxpiPyPI caching mirrorHost a proxy PyPI mirror server with cachingCache the index (project list and projects' file list)Cache the project filesSupport multiple indicesSet index cache times-to-live (individually for each index)Set files cache max-size on diskManually invalidate index cacheInstallationIf not using DockerpipinstallproxpiInstallcoloredlogsas well to get coloured loggingUsageStart serverDockerUses aGunicornWSGI serverdockerrun-p5000:5000epicwink/proxpiWithout arguments, runs with 2 threads. If passing arguments, make sure to bind to an
exported address (or all with0.0.0.0) on port 5000 (ie--bind 0.0.0.0:5000).ComposeAlternatively, useDocker ComposedockercomposeupLocalFLASK_APP=proxpi.serverflaskrunSeeflask run --helpfor more information on address and port binding, and certificate
specification to use HTTPS. Alternatively, bring your own WSGI server.Use proxyUse PIP's index-URL flag to install packages via the proxypipinstall--index-urlhttp://127.0.0.1:5000/index/simplejsonCache invalidationEither head tohttp://127.0.0.1:5000/in the browser, or run:curl-XDELETEhttp://127.0.0.1:5000/cache/simplejson
curl-XDELETEhttp://127.0.0.1:5000/cache/listIf you need to invalidate a locally cached file, restart the server: files should never
change in a package index.Environment variablesPROXPI_INDEX_URL: index URL, default:https://pypi.org/simple/PROXPI_INDEX_TTL: index cache time-to-live in seconds,
default: 30 minutes. Disable index-cache by setting this to 0PROXPI_EXTRA_INDEX_URLS: extra index URLs (comma-separated)PROXPI_EXTRA_INDEX_TTLS: corresponding extra index cache times-to-live in seconds
(comma-separated), default: 3 minutes, cache disabled when 0PROXPI_CACHE_SIZE: size of downloaded project files cache (bytes), default 5GB.
Disable files-cache by setting this to 0PROXPI_CACHE_DIR: downloaded project files cache directory path, default: a new
temporary directoryPROXPI_BINARY_FILE_MIME_TYPE=1: force file-response content-type to"application/octet-stream"instead of letting Flask guess it. This may be needed
if your package installer (eg Poetry) mishandles responses with declared encoding.Considerations with CIproxpiwas designed with three goals (particularly for continuous integration (CI)):to reduce load on PyPI package servingto reducepip installtimesnot require modification to the current workflowSpecifically,proxpiwas designed to run for CI services such asTravis,Jenkins,GitLab CI,Azure PipelinesandGitHub Actions.proxpiworks by caching index requests (ie which versions, wheel-types, etc are
available for a given project, the index cache) and the project files themselves (to a
local directory, the package cache). This means they will cache identical requests after
the first request, and will be useless for just onepip install.Cache persistenceAs a basic end-user of these services, for at least most of these services you won't be
able to keep aproxpiserver running between multiple invocations of your project(s)
CI pipeline: CI invocations are designed to be independent. This means the best that you
can do is start the cache for just the current job.A more advanced user of these CI services can bring their own runner (personally, my
needs are for running GitLab CI). This means you can runproxpion a fully-controlled
server (egEC2instance), and proxy PyPI requests (during
apipcommand) through the local cache. See the instructionsbelow.Hopefully, in the future these CI services will all implement their own transparent
caching for PyPI. For example, Azure already hasAzure Artifactswhich
provides much more functionality thanproxpi, but won't reducepip installtimes for
CI services not using Azure.GitLab CI instructionsThis implementation leverages the index URL configurable ofpipand Docker networks.
This is to be run on a server you have console access to.Create a Docker bridge networkdockernetworkcreategitlab-runner-networkStart a GitLab CI Docker runner usingtheir documentationRun theproxpiDocker containerdockerrun\--detach\--networkgitlab-runner-network\--volumeproxpi-cache:/var/cache/proxpi\--envPROXPI_CACHE_DIR=/var/cache/proxpi\--nameproxpiepicwink/proxpi:latestYou don't need to expose a port (the-pflag) as we'll be using an internal
Docker network.Setpip's index URL to theproxpiserver by setting it in the runner environment.
Setrunners[0].docker.network_modetogitlab-runner-network.
AddPIP_INDEX_URL=http://proxpi:5000/index/andPIP_TRUSTED_HOST=proxpitorunners.environmentin the GitLab CI runner configuration TOML. For example, you
may end up with the following configuration:[[runners]]name="awesome-ci-01"url="https://gitlab.com/"token="SECRET"executor="docker"environment=["DOCKER_TLS_CERTDIR=/certs","PIP_INDEX_URL=http://proxpi:5000/index/","PIP_TRUSTED_HOST=proxpi",][[runners.docker]]network_mode="gitlab-runner-network"...This is designed to not require any changes to the GitLab CI project configuration (iegitlab-ci.yml), unless it already sets the index URL for some reason (if that's the
case, you're probably already using a cache).Another option is to set up a proxy, but that's more effort than the above method.