aws/aws-sdk-js — pull request #109 by Rafe (2013-05-01 23:30:32+00:00)
Title: Fix readableStream unexpected end while waiting response

Hello,

recently while using createReadStream() to pipe response in AWS.S3, under Node v0.10.1
I found the stream will hangout unexpected and only return partial result:

var buffers = [];
var s3 = new AWS.S3();
var stream = s3.getObject(params).createReadStream();
stream.on('readable', function(){
var data;
while (data = stream.read()){
buffers.push(data);
};
});
stream.on('end', function(){
fs.writeFileSync('./test.jpg', Buffer.concat(buffers));
}); // result in partial image

After checking the source of createReadStream(),
I found the stream._read method is missing a push method when httpStream.read() return false values, which result to the unexpected end:

stream._read = function() {
var data;
/*jshint boss:true*/
while (data = httpStream.read()) {
stream.push(data);
}
};

What happened here, is because when the read method on stream is called, it will set a reading flag to true, until the stream.push is called, it will block next read call until the previous read finished. Therefore when httpStream is still waiting for response, the readStream will block itself because of the missing push call.

Further details is in the source of [issac/readable-stream](https://github.com/issac/readable-stream)
[https://github.com/isaacs/readable-stream/blob/master/lib/_stream_readable.js#L286](https://github.com/isaacs/readable-stream/blob/master/lib/_stream_readable.js#L286)

So the pipe is also working now under 0.10.x,

var out = fs.createWriteStream('./nyancat.jpg');
s3.getObject(params).createReadStream().pipe(out);

Let me know is this solves the problem, also I made some hack on helper to emulate empty read call. Probably will have better way to emulate it.