nathanoehlman/knox-mpu — pull request #29 by dustMason (2014-05-30 21:15:21+00:00)
Title: Upload doesn't start until input stream is done?

I've been puzzling through an implementation of knox-mpu today and have come up against something confusing. I'm building a tool which will stream a bucket full of files through a transform stream to create a tar archive, which will then be streamed back to the bucket.

Here's what I've got at the moment: https://gist.github.com/dustMason/735a9ecae57c65cb9685

I'm confused by the fact that no uploads begin until the `end` event is triggered on the stream given to MuliPartUpload, which doesn't seem to jive with my understanding of how streams should work in general. I played around with https://github.com/visionmedia/batch which is what looks like is being used under the hood. Since the only time Batch's `end` method is called appears in `_completeUploads`, no uploading occurs until then. And `_completeUploads` is only called upon the `end` event of the input stream.

Thus it feels like there is no benefit to being able to pass a stream to MultiPartUpload at all :frowning:

I messed around with calling Batch's `end` in the `partReady` function after adding each upload to the batch, but that results in strange behavior due to `end` being called repeatedly.

Would you consider using something like https://github.com/caolan/async#queue instead? I would be up for making a pull request to get this going.