konklone/upload-anything — pull request #1 by konklone (2014-10-12 06:27:40+00:00)
Title: First demo: read large files without stressing memory

![ssstttrrreeeaaammmiiinnnggg](https://cloud.githubusercontent.com/assets/4592/4605287/372a951c-51d8-11e4-9788-03ed7bdfde3a.png)

It works! This is me dropping a 1GB file (`1gb.txt`, visible in the bottom-right) into the center of the page, and then printing output every 1MB. The contents of each chunk (using the default `8128`-byte chunk size) aren't used, they are allowed to fade into garbage collection.

There seemed to be some memory increase during the upload operation, memory which eventually dissipated after the upload was done. But it was at a roughly 50:1 ratio (500MB streamed, 10MB more memory), and could easily have been normal V8 work and not actually correlated.

At around the ~850MB mark, the console stopped printing out information

The next step is: testing this out with a 200GB file and watching the memory more closely. No PR for that, most likely.

After that, is tackling the S3 upload directly, using [@maxogden's notes](https://github.com/DamonOehlman/filestream/issues/9#issuecomment-58468336) and the [AWS docs](http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html) and maybe the [AWS JS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-examples.html) as a guide.