mpi-sws-rse/datablox — Issue #17: Need throttling for push/pull connections
Author: jfischer
Created: 2011-12-10 22:55:31+00:00
State: open

I tried running all the blocks of the file_index.json example on my laptop. It drove up the cpu and memory usage to the point where the UI froze and I had to hard-reset my laptop. This will also be an issue in distributed scenarios where multiple producers (e.g. a crawler) feed into one consumer. The solution is to add throttling to push/pull connections.

We need to discuss the design options. Here's a few ideas:
* Does zeromq actually queue messages? If so, can we get queue size statistics?
* Could look at cpu / memory usage on the consumer side. If it exceeds some threshold (e.g. above 80%), request that produces back off a bit.
* Messages would include timestamps. Consumer would send periodic acknowledgement message to consumer indicating the wait time and processing time for messages. If it is too large, producer would back off.