taskcluster/docker-worker — pull request #29 by jonasfj (2014-04-14 20:28:02+00:00)
Title: Using new time-chunked-stream module to avoid backpressure from azure li...

...velog by having a lot of small commits

So... the azure blob stream was getting a lot of small chunks... each took about a second to upload and commit...
This cause backpressure when lots of small chunks was written, because node.js doesn't merge buffered chunks... Instead it's writes buffered chunks as they came in...

A fix could be to implement from buffering logic in azure blob stream. I opted to implement an intermediate duplex stream that would buffer data and output it all data collected every 5 seconds...

I suspect that all forms of live logging should have this kind of buffering logic... even if they upload-chunk overhead is much smaller than it is with azure... Uploading a log one line at the time is simply not feasible and the risk of back-pressure from request overhead is too big.