konklone/upload-anything — pull request #6 by konklone (2014-10-13 09:42:28+00:00)
Title: Project now supports 10GB files

![yay](https://cloud.githubusercontent.com/assets/4592/4611627/3244f16a-52bd-11e4-9d5d-1597844b342a.png)

The work all ended up on the master branch, but to mark the moment, the project now properly handles **backpressure**, and properly installs s3-upload-stream from the web.

This brings the supported file size of this project from **1GB** to **10GB**!

* On `s3-upload-stream`, I've [patched its browser support](https://github.com/nathanpeck/s3-upload-stream/pull/21) (PR still outstanding) so that it can be installed directly, without freezing or massaging.
* On `filereader-stream`, I've [added backpressure management](https://github.com/maxogden/filereader-stream/pull/6) (PR still outstanding) so that it will only continue reading in file data as there is reasonable buffer to send data into. This relies on `s3-upload-stream` default high watermark of 4MB.

Put together, this project is now pulling its two stream libraries from the web, and can handle files that are larger than the available memory of the machine doing this work, while still holding somewhere around ~10MB in memory at any given time.

Next steps:

* It should be easy to get to 100GB, by overriding the part size and allowing each part to be 10-20MB (instead of the default 5MB). S3 has a limit of 10,000 parts in any multipart upload.
* It will be tougher to get to 1TB+, because that makes the part size (100MB) unmanageable by the client. I gave it a brief shot with 200MB parts and my browser tab crashed on the first `xhr.send()`. As of right now, with S3's limitations I think this will require a server-side proxy. Even a weak server should be capable of making a sustained 100MB HTTP upload (or 500MB, to hit Amazon's max of 5TB) to S3. A server-side proxy could take data streamed however (e.g. Websockets), because multi-part chunking would have to take place entirely server-side.
* The Internet Archive would be a nice next goal, but they don't support CORS yet. I'll ask them to update this.