LearnBoost/mongoose — Issue #668: Add a 'complete' event for streams
Author: jeffutter
Created: 2012-01-02 08:00:48+00:00
State: open

I have an app with a few levels of nested streamed queues. " For all these records, remove all the related records" type things.

To manage flow control i'm using the nested streams 'close' callback (and some async library functions) indicate "These records are done updating you can now update the parent record" The problem i'm running into is that 'close' fires when the last record passes through the stream... not when it's 'data' is done.

This causes a problem in the following scenario: Sub stream fires a task that takes a bit of time (Creating a record) and then adds that record's id to an array in it's parent element (the parent stream). What ends up happening is 'done' gets called before the child record is created and the parent saves before the child is added.

I'm not familiar with EventEmitter to know if this would be a viable solution. However would it be possible to add a 'complete' event to the stream indicating that all records have passed through and all of their 'data' events have completed.

here's a gist with the problem code:
https://gist.github.com/548a2c12932811d7ece8

The problem i'm hitting is in the: paymentStream.on('data', function (payment) {...}
it should be appending it's _id to the billClosed above it, but the 'done' event is getting fired before the record is created and saved.

here's a gist with a simple workaround that i'm sure is not ideal:
https://gist.github.com/fa9a903f4168a7b5acc2 (notice the que=[] bit)

Sorry for the lenghty gists but it may be necessary to understand the nesting problem here.