Package: npm-accel
The npm-accel program is a wrapper fornpm(theNode.jspackage manager) that
was created to optimizeone specific use case: Creating anode_modulesdirectory from apackage.jsonfile as quickly as possible.It works on the assumption that you build node_modules directories more
frequently then you change the contents of package.json files, because it
computes a fingerprint of the dependencies and uses that fingerprint as a cache
key, to cache the complete node_modules directory in a tar archive.The program is intended to be used in environments that always or frequently
start with an empty node_modules directory and need to populate the complete
directory from scratch (e.g. continuous integration builds and deployments).
I’m specifically not claiming that you will see any speed improvements if
you’re updating existing node_modules directories.The npm-accel program is currently tested on Python 2.7, 3.5, 3.6, 3.7, 3.8 and
PyPy (yes, it’s written in Python, deal with it 😉). It’s intended to work on
UNIX systems like Linux and Mac OS X and specifically won’t work on Windows
(seesupported operating systemsfor details).InstallationUsageStatusPerformanceSupported operating systemsFuture improvementsContactLicenseInstallationThenpm-accelpackage is available onPyPIwhich means installation
should be as simple as:$pipinstallnpm-accelThere’s actually a multitude of ways to install Python packages (e.g. theper
user site-packages directory,virtual environmentsor just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions 😉.Alternatively if you are running Arch Linux, npm-accel isavailable in AUR.UsageThere are two ways to use thenpm-accelpackage: As the command line programnpm-acceland as a Python API. For details about the Python API please
refer to the API documentation available onRead the Docs. The command line
interface is described below.Usage:npm-accel [OPTIONS] [DIRECTORY]The npm-accel program is a wrapper for npm (the Node.js package manager) that
optimizes one specific use case: Building a “node_modules” directory from a
“package.json” file as quickly as possible.It works on the assumption that you build “node_modules” directories more
frequently then you change the contents of “package.json” files, because it
computes a fingerprint of the dependencies and uses that fingerprint as a
cache key, to cache the complete “node_modules” directory in a tar archive.Supported options:OptionDescription-p,--productionDon’t install modules listed in “devDependencies”.-i,--installer=NAMESet the installer to use. Supported values forNAMEare “npm”, “yarn”,
“pnpm” and “npm-cache”. When yarn is available it will be selected as the
default installer, otherwise the default is npm.-u,--updateDon’t read from the cache but do write to the cache. If you suspect a cache
entry to be corrupt you can use--updateto ‘refresh’ the cache entry.-n,--no-cacheDisallow writing to the cache managed by npm-accel (reading is still
allowed though). This option does not disable internal caching
performed by npm, yarn, pnpm and npm-cache.-c,--cache-directory=DIRSet the pathname of the directory where the npm-accel cache is stored.-l,--cache-limit=COUNTSet the maximum number of tar archives to preserve. When the cache
directory contains more thanCOUNTarchives the least recently used
archives are removed. Defaults to 20.The environment variable$NPM_ACCEL_CACHE_LIMITprovides a convenient
way to customize this option in CI and build environments.-b,--benchmarkBenchmark and compare the following installation methods:npm installyarnpnpmnpm-accelnpm-cacheThe first method performs no caching (except for the HTTP caching that’s
native to npm) while the other four methods each manage their own cache
(that is to say, the caching logic of npm-accel is only used in step 4).Warning: Benchmarking wipes the caches managed by npm, yarn, pnpm,
npm-accel and npm-cache in order to provide a fair comparison (you
can override this in the Python API but not on the command line).-r,--remote-host=SSH_ALIASOperate on a remote system instead of the local system. TheSSH_ALIASargument gives the SSH alias of the remote host.-v,--verboseIncrease logging verbosity (can be repeated).-q,--quietDecrease logging verbosity (can be repeated).--versionReport the version of npm-accel.-h,--helpShow this message and exit.StatusThe npm-accel project was developed and published in September 2016 because I
got fed up waiting fornpm installto finish, specifically in the context
of continuous integration builds and deployments (where you frequently start
with an emptynode_modulesdirectory). It was developed in about a week
without much prior knowledge aboutNode.jsornpm, which explains why it’s
written in Python 😛.The project was initially published with 95% test coverage and I was careful
not to repeat the bugs I encountered innpm-cacheandnpm-fast-installwhile
evaluating those tools 🙂.At the time of writing (early 2020) npm-accel has been in active use at my
employer for more than three years and it has worked wonderfully well to speed
up our continuous integration builds and deployments. But don’t take my word
for it, try it out and see if it helps your use case!PerformanceThe following table lists the output ofnpm-accel--benchmarkagainst a
private code base with 128 dependencies listed in the package.json file
(resulting in a 895 MiB node_modules directory):ApproachIterationElapsed timePercentagenpm install1 of 21 minute and 16.56 seconds100%npm install2 of 21 minute and 4.99 seconds85%yarn1 of 21 minute and 6.33 seconds87%yarn2 of 241.7 seconds54%pnpm1 of 255.24 seconds72%pnpm2 of 241.28 seconds54%npm-accel1 of 21 minute and 12.1 seconds94%npm-accel2 of 23.25 seconds4%npm-cache1 of 22 minutes and 14.41 seconds176%npm-cache2 of 222.51 seconds29%Some notes about this benchmark:Each of the installation methods is run twice. The first run starts with
empty cache directories and is intended to “prime the cache”. The second run
is intended to use the cache and should be able to do so quite effectively,
given that the package.json file does not change between the two runs.During the benchmark, the caching performed by npm-accel is only used in the
eight row of the table above. This is because the original point of the
benchmark (for me) was to find out whether it was even worth it to develop
and publish npm-accel. That is to say, if it wouldn’t have given a speed
improvement it wasn’t worth my time, nor yours 😛.Supported operating systemsThe npm-accel program was developed to work on UNIX systems like Linux and Mac
OS X. It requires several external commands to be installed (e.g.mkdir,mv,rm,tarandwhich).I’ve tried to keep all of the external command invocations compatible with the
Linux and BSD variants of commands liketar, that is to say npm-accel uses
only short options and skips the more esoteric features even if they would be
useful. If you find that I failed in this respect, please feel free to report
this as a bug.For posterity: It was a conscious decision (for several reasons) to use thetarprogram instead of manipulating tar archives via Python’starfile
module.Future improvementsDealing with optionalDependenciesI’ve never seenoptionalDependenciesin the wild but encountered them
while browsing through thepackage.jsondocumentation. Maybe these should be
part of the computed cache keys aswell?ContactThe latest version ofnpm-accelis available onPyPIandGitHub. The
documentation is hosted onRead the Docs. For bug reports please create an
issue onGitHub. If you have questions, suggestions, etc. feel free to send me
an e-mail atpeter@peterodding.com.LicenseThis software is licensed under theMIT license.© 2020 Peter Odding.