isaacs/npm — Issue #1920: Parallelization of install scripts causing crash on VPS
Author: justmoon
Created: 2011-12-22 11:33:06+00:00
State: open

I was trying to `npm install -g bitcoinjs` on a MediaTemple (ve) Server, but ran into random errors like (paraphrasing) "could not allocate virtual memory" and "could not start thread". Most of the time one of the node-waf processes just fails without error message. While the error seems to be somewhat random, the installation failed reliably every time I tried it.

Eventually I tried installing some of the dependencies that contain C++ code first, one-by-one and then installing bitcoinjs last which worked fine.

Here is the log for first trying to install it in one go, then individually: https://gist.github.com/1492987

What I believe is happening is that NPM tries to compile all of the dependencies at once. However gcc is pretty heavy (node-waf may even start multiple threads itself, I'm not sure), so what ends up happening is that memory or simply the allowed number of threads is exhausted.

Note that the installation works fine on machines with a regular kernel that doesn't have those hard resource limits some VPSes have. I still suspect that parallelizing large numbers of node-waf processes or more generally install scripts is a bad idea there as well, slowing down the installation rather than speeding it up.