Package: pip-accel
The pip-accel program is a wrapper forpip, the Python package manager. It
accelerates the usage of pip to initializePython virtual environmentsgiven
one or morerequirements files. It does so by combining the following two
approaches:Source distribution downloads are cached and used to generate alocal index
of source distribution archives. If all your dependencies are pinned to
absolute versions whose source distribution downloads were previously
cached, pip-accel won’t need a network connection at all! This is one of the
reasons why pip can be so slow: given absolute pinned dependencies available
in the download cache it will still scanPyPIand distribution websites.Binary distributionsare used to speed up the process of installing
dependencies with binary components (likeM2CryptoandLXML). Instead of
recompiling these dependencies again for every virtual environment we
compile them once and cache the result as a binary*.tar.gzdistribution.In addition, since version 0.9 pip-accel contains a simple mechanism that
detects missing system packages when a build fails and prompts the user whether
to install the missing dependencies and retry the build.The pip-accel program is currently tested on cPython 2.6, 2.7, 3.4 and 3.5 and
PyPy (2.7). The automated test suite regularly runs on Ubuntu Linux (Travis
CI) as well as Microsoft Windows (AppVeyor). In addition to these platforms
pip-accel should work fine on most UNIX systems (e.g. Mac OS X).ContentsStatusUsageConfigurationHow fast is it?Alternative cache backendsStoring the binary cache on Amazon S3Using S3 compatible storage servicesCaching of setup requirementsDependencies on system packagesIntegrating with toxControl flow of pip-accelContactLicenseStatusPaylogicuses pip-accel to quickly and reliably initialize virtual
environments on its farm of continuous integration slaves which are constantly
running unit tests (this was one of the original use cases for which pip-accel
was developed). We also use it on our build servers.When pip-accel was originally developedPyPIwas sometimes very unreliable
(PyPI wasn’tbehind a CDNback then). Because of the CDN, PyPI is much more
reliable nowadays however pip-accel still has its place:The CDN doesn’t help for distribution sites, which are as unreliably as they
have always been.By using pip-accel you can make Python deployments completely independent
from internet connectivity.Because pip-accel caches compiled binary packages it can still provide a nice
speed boost over using plain pip.UsageThe pip-accel command supports all subcommands and options supported by pip,
however it is of course only useful for thepip installsubcommand. So for
example:$pip-accelinstall-rrequirements.txtAlternatively you can also run pip-accel as follows, but note that this
requires Python 2.7 or higher (it specifically doesn’t work on Python 2.6):$python-mpip_accelinstall-rrequirements.txtIf you pass a-vor--verboseoption then pip and pip-accel will both
use verbose output. The-qor--quietoption is also supported.Based on the user running pip-accel the following file locations are used by
default:Root userAll other usersPurpose/var/cache/pip-accel~/.pip-accelUsed to store the source/binary indexesThis default can be overridden by defining the environment variablePIP_ACCEL_CACHE.ConfigurationFor most users the default configuration of pip-accel should be fine. If you do
want to change pip-accel’s defaults you do so by setting environment variables
and/or adding configuration options to a configuration file. This is because
pip-accel shares its command line interface with pip and adding support for
command line options specific to pip-accel is non trivial and may end up
causing more confusion than it’s worth :-). For an overview of the available
configuration options and corresponding environment variables please refer to
thedocumentation of the pip_accel.config module.How fast is it?To give you an idea of how effective pip-accel is, below are the results of a
test to build a virtual environment for one of the internal code bases ofPaylogic. This code base requires more than 40 dependencies including several
packages that need compilation with SWIG and a C compiler:ProgramDescriptionDurationPercentagepipDefault configuration444 seconds100% (baseline)pipWith download cache (first run)416 seconds94%pipWith download cache (second run)318 seconds72%pip-accelFirst run397 seconds89%pip-accelSecond run30 seconds7%Alternative cache backendsBundled with pip-accel are a local cache backend (which stores distribution
archives on the local file system) and anAmazon S3backend (see below).Both of these cache backends are registered with pip-accel using a generic
pluggable cache backend registration mechanism. This mechanism makes it
possible to register additional cache backends without modifying pip-accel. If
you are interested in the details please refer to pip-accel’ssetup.pyscript and the two simple Python modules that define the bundled backends.If you’ve written a cache backend that you think may be valuable to others,
please feel free to open an issue or pull request on GitHub in order to get
your backend bundled with pip-accel.Storing the binary cache on Amazon S3You can configure pip-accel to store its binary cache files in anAmazon S3bucket. In this case Amazon S3 is treated as a second level cache, only used if
the local file system cache can’t satisfy a dependency. If the dependency is
not found in the Amazon S3 bucket, the package is built and cached locally (as
usual) but then also saved to the Amazon S3 bucket. This functionality can be
useful for continuous integration build worker boxes that are ephemeral and
don’t have persistent local storage to store the pip-accel binary cache.To get started you need to install pip-accel as follows:$pipinstall'pip-accel[s3]'The[s3]part enables the Amazon S3 cache backend by installing theBotopackage. Once installed you can use the following environment variables to
configure the Amazon S3 cache backend:$PIP_ACCEL_S3_BUCKETThe name of the Amazon S3 bucket in which binary distribution archives should
be cached. This environment variable is required to enable the Amazon S3 cache
backend.$PIP_ACCEL_S3_PREFIXThe optional prefix to apply to all Amazon S3 keys. This enables name spacing
based on the environment in which pip-accel is running (to isolate the binary
caches of ABI incompatible systems).The user is currently responsible for
choosing a suitable prefix.$PIP_ACCEL_S3_READONLYIf this option is set pip-accel will skip uploading to the Amazon S3 bucket.
This means pip-accel will use the configured Amazon S3 bucket to “warm up”
your local cache but it will never write to the bucket, so you can use read
only credentials. Of course you will need to run at least one instance of
pip-accel that does have write permissions, so this setup is best suited to
teams working around e.g. a continuous integration (CI) server, where the CI
server primes the cache and developers use the cache in read only mode.You can also set these options from a configuration file, please refer to thedocumentation of the pip_accel.config module. You will also need to set AWS
credentials, either in a.boto fileor in the$AWS_ACCESS_KEY_IDand$AWS_SECRET_ACCESS_KEYenvironment variables (refer to the Boto
documentation for details).Using S3 compatible storage servicesIf you want to point pip-accel at anS3 compatible storage servicethat isnotAmazon S3 you canoverride the S3 API URLusing a configuration option
or environment variable. For example the pip-accel test suite first installs
and startsFakeS3and then setsPIP_ACCEL_S3_URL=http://localhost:12345to
point pip-accel at the FakeS3 server (in order to test the Amazon S3 cache
backend without actually having to pay for an Amazon S3 bucket :-). For more
details please refer to the documentation of theAmazon S3 cache backend.Caching of setup requirementsSince version 0.38 pip-accel instructs setuptools to cache setup requirements
in a subdirectory of pip-accel’s data directory (see theeggs_cacheoption) to
avoid recompilation of setup requirements. This works by injecting a symbolic
link called.eggsinto unpacked source distribution directories before pip
or pip-accel runs the setup script.The use of the.eggsdirectory was added in setuptools version 7.0 which is
why pip-accel now requires setuptools 7.0 or higher to be installed. This
dependency was added because the whole point of pip-accel is to work well out
of the box, shielding the user from surprising behavior like setup requirements
slowing things down and breaking offline installation.Dependencies on system packagesSince version 0.9 pip-accel contains a simple mechanism that detects missing
system packages when a build fails and prompts the user whether to install the
missing dependencies and retry the build. Currently only Debian Linux and
derivative Linux distributions are supported, although support for other
platforms should be easy to add. This functionality currently works based on
configuration files that define dependencies of Python packages on system
packages. This means the results should be fairly reliable, but every single
dependency needs to be manually defined…Here’s what it looks like in practice:2013-06-16 01:01:53 wheezy-vm INFO Building binary distribution of python-mcrypt (1.1) ..
2013-06-16 01:01:53 wheezy-vm ERROR Failed to build binary distribution of python-mcrypt! (version: 1.1)
2013-06-16 01:01:53 wheezy-vm INFO Build output (will probably provide a hint as to what went wrong):

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DVERSION="1.1" -I/usr/include/python2.7 -c mcrypt.c -o build/temp.linux-i686-2.7/mcrypt.o
mcrypt.c:23:20: fatal error: mcrypt.h: No such file or directory
error: command 'gcc' failed with exit status 1

2013-06-16 01:01:53 wheezy-vm INFO python-mcrypt: Checking for missing dependencies ..
2013-06-16 01:01:53 wheezy-vm INFO You seem to be missing 1 dependency: libmcrypt-dev
2013-06-16 01:01:53 wheezy-vm INFO I can install it for you with this command: sudo apt-get install --yes libmcrypt-dev
Do you want me to install this dependency? [y/N] y
2013-06-16 01:02:05 wheezy-vm INFO Got permission to install missing dependency.

The following extra packages will be installed:
  libmcrypt4
Suggested packages:
  mcrypt
The following NEW packages will be installed:
  libmcrypt-dev libmcrypt4
0 upgraded, 2 newly installed, 0 to remove and 68 not upgraded.
Unpacking libmcrypt4 (from .../libmcrypt4_2.5.8-3.1_i386.deb) ...
Unpacking libmcrypt-dev (from .../libmcrypt-dev_2.5.8-3.1_i386.deb) ...
Setting up libmcrypt4 (2.5.8-3.1) ...
Setting up libmcrypt-dev (2.5.8-3.1) ...

2013-06-16 01:02:13 wheezy-vm INFO Successfully installed 1 missing dependency.
2013-06-16 01:02:13 wheezy-vm INFO Building binary distribution of python-mcrypt (1.1) ..
2013-06-16 01:02:14 wheezy-vm INFO Copying binary distribution python-mcrypt-1.1.linux-i686.tar.gz to cache as python-mcrypt:1.1:py2.7.tar.gz.Integrating with toxYou can tellToxto use pip-accel using a small shell script that first uses
pip to install pip-accel, then uses pip-accel to bootstrap the virtual
environment. You can find details about this inissue #30 on GitHub.Control flow of pip-accelThe way pip-accel works is not very intuitive but it is very effective. Below
is an overview of the control flow. Once you take a look at the code you’ll
notice that the steps below are all embedded in a loop that retries several
times. This is mostly because of step 2 (downloading the source
distributions).Runpip install--download=...--no-index-rrequirements.txtto unpack
source distributions available in the local source index. This is the first
step because pip-accel should acceptrequirements.txtfiles as input but
it will manually install dependencies from cached binary distributions
(without using pip or easy_install):If the command succeeds it means all dependencies are already available as
downloaded source distributions. We’ll parse the verbose pip output of step
1 to find the direct and transitive dependencies (names and versions)
defined inrequirements.txtand use them as input for step 3.
Go to step 3.If the command fails it probably means not all dependencies are available
as local source distributions yet so we should download them. Go to step 2.Runpip install--download=...-rrequirements.txtto download missing
source distributions to the download cache:If the command fails it means that pip encountered errors while scanningPyPI, scanning a distribution website, downloading a source distribution
or unpacking a source distribution. Usually these kinds of errors are
intermittent so retrying a few times is worth a shot. Go to step 2.If the command succeeds it means all dependencies are now available as
local source distributions; we don’t need the network anymore! Go to step 1.Runpython setup.py bdist_dumb--format=gztarfor each dependency that
doesn’t have a cached binary distribution yet (taking version numbers into
account). Go to step 4.Install all dependencies from binary distributions based on the list of
direct and transitive dependencies obtained in step 1. We have to do these
installations manually because easy_install nor pip support binary*.tar.gzdistributions.ContactIf you have questions, bug reports, suggestions, etc. please create an issue on
theGitHub project page. The latest version of pip-accel will always be
available on GitHub. The internal API documentation ishosted on Read The
Docs.LicenseThis software is licensed under theMIT licensejust likepip(on which
pip-accel is based).© 2016 Peter Odding andPaylogicInternational.