ngds/data-archiver — pull request #17 by asonnenschein (2014-05-29 05:55:35+00:00)
Title: Flow control

A major effort to tame asynchronous processes. Your operating system has a limit of how many files can be open at a single time. Normally when automating I/O tasks, this isn't a problem because your program operates on a single file at a time. In the asynchronous I/O world, it's possible that you'll have thousands of files open at the same time -- which will push the open files limit and crash your program.

So, the solution then is to modularize the flow of data through the program. This means splitting chunks of data into groups and triggering those groups synchronously, but still running all processes within the groups asynchronously. Easy, right? Well it would be if Node.js had no bugs...

Either an "end" or an "error" message will be emitted when a stream of data is finished piping to an output file. Both of these events should tell the process to then close that file. For "end" events, this always works as it should. For "error" events, there is a known bug called the "file descriptor memory leak" which happens when a stream process has emitted a termination command but has not terminated -- so it stays open in memory. When dealing with streaming data over a network, we're always going to have lots of errors that we cannot predict and in Node-flavored file I/O it means that we'll hit the too many open files error.

The code in this merge fixes all of these open file errors and does pretty much everything (including compressing data into zips) except interaction with S3.