addthis/hydra — pull request #50 by tea-dragon (2014-03-21 16:01:53+00:00)
Title: Prefetch meshy2

planning to do a bit of cleanup still but, summary is:

# mesh2 source
- stream files can be initialized and put onto the preOpen queue without blocking (that was a known side effect of gzip wrapping). If combined with the mesh parameter for 'prefetch', this can allow them to begin buffering as well. Although in that case, you probably also want to set multi bundle read to a higher value. This works best if your source files fit into your mesh stream receive buffer (typically about 5MB but configurable). Future work there could be a setting to block until at least one bundle is read and then 'multi-read' until data isn't available, but at that point you are probably very close to a more robust entirely non-blocking setup anyway.
- selecting which replica to read from is done outside of a lock. To keep the selection logic etc out of ASFDS, I moved the lock into StreamSourceMeshy.
- mesh host score cache has a configurable timeout for status look ups, and way more metrics and logging.
- linked blocking queue instead of array blocking queue in ASFDS. The array variant uses a global lock, and I ran into an issue where larger numbers of source worker threads (16 or 32 thereabouts) could cause so much lock contention trying to put things onto the queue that the single thread in task feeder had trouble getting bundles even though the queue was full. This is probably at least partially due to the way we are using it though - i'll skip the details, but this was a quick hack/fix because the linked queue has a separate push/pull lock so the source workers only fight each other.

# task feeder
- changed some of the internals of the task feeder, especially logic related to work stealing. We don't generally employ it, and even though I think it might be better now, and my testing was not super extensive, it still didn't help. So I don't feel strongly about those changes.