Node_Modules Did Nothing Wrong.
By: Chris Ackerman
Tags: Nodejs, Commonjs, NPM, Node Modules, Yarn Package Manager

NPM is planning on integrating Tink. Yarn 2 introduced PnP. The two biggest package managers in the ECMAScript ecosystem, are both starting to steer away from the node_modules directory. At least Tink is keeping the ability to “unwind” for backwards compatibility, and NPM is trying to play nice with the community (See “Play Nicer with Yarn”).

At the heart of this is “the node_modules problem”. This is a short title encompassing several pain points.

The size of the average node_modules directory.

directory. The complexity of the node_modules filesystem.

filesystem. That node_modules is on disk, and disk IO is slow.

Other problems are generally related to those problems.

But are these really problems?

I would argue that they are not.

First, disk size isn’t that precious, and getting less so. Packages in a project might get up to around 100MB or more, but in the world of terabyte size storage, that’s 0.01% of your capacity. You could literally have 100s of projects installed or in development before it became noticeable.

Second, file system complexity is only an inconvenience in a small subset of activities. Installing packages can be helped by caching and optimizing package managers to do the least work required. Copying a project can be optimized by removing node_modules , copying, and then reinstalling.

Third, disk IO is slow, but with solid state drives and (again) package manager optimizations, it’s not that slow. At work, I can reinstall all of the packages in our most complex mono-repo in about 3 minutes, which I really only do 2 or 3 times a week.

I’m really not sure where the teams are that are suffering from slow app startup times. Whoever they are, bundling, a virtual file system, or a RAM disk, should sort you right out in your highly tuned environment.

Why should we keep node_modules?

Because it’s simple, agnostic, transparent, incredibly flexible, and widely supported.

Anything that can read the filesystem can implement the algorithm. You don’t need to extract archives, read a Javascript manifest, or hack an API. It doesn’t matter what or how the files got to their places in the filesystem, as long as they can be found by following the algorithm.

Both Tink and PnP work in Node by overriding the fs API. Tink has a JSON manifest, and PnP has a Javascript “manifest”. PnP’s manifest actually includes its own injection code. Just executing your entry script with Node won’t work anymore. You have to use Tink or Yarn as a bootstrap, or run Node with a --require argument. This means that your package manager just became part of your runtime stack.

Compilers and bundlers will also have to support each (diverging) resolution system. I’m sure they will, but it adds complexity for what I don’t think is any real benefit.

Losing node_modules makes dependency code and resolution less transparent and harder to debug. If I need to right now, I can go dig into the node_modules directories to see what’s what. That’s something I actually do several times a day. Doing the same thing in Tink or PnP means extra steps, instrumentation, IDE support, reading generated code, looking into archive files, unrolling, and/or leaving my project directory.

And let’s talk about workspaces and linking…

Which I think are another elephant in the room driving these changes.

Workspaces are a way to develop multiple projects together, without having to publish one package before using it in another package. To do this, they make heavy use of linking (filesystem symbolic links). Which seemed like a good idea when NPM came up with the pattern early on, but they have given people headaches ever since. Because symlinks turn your nice package resolution tree into super fun maze with trapdoors.

I have to imagine that as the Workspaces pattern has become more popular, NPM and Yarn have both been trying to find a way to fix the problems related to symlinks. And what better way than to just stop using the filesystem?

Well, how about just not using symlinks? Granted, it seems like the obvious and simple solution when you need to use one unpublished package as a dependency of another. But, as with a lot of simple and obvious things, it’s not really the only or the most correct solution.

Final word.

There are a lot of blogs and tech writers out there saying things like “about time”, and “logical next step”.

But, I’ve been using Node nearly since its inception. I’ve also carefully evaluated and adopted Typescript, Babel, Webpack, Yarn, and many other innovations as they came along, in both a professional and personal capacity. And for contrast, I’ve worked outside the ECMAScript ecosystem in C[++|#], Java, Go, and Python.

To all of the people leaping to praise the Yarn and NPM roadmaps: I respectfully disagree with you.