DigitalPebble/storm-crawler — pull request #14 by jakekdodd (2014-10-22 18:16:50+00:00)
Title: added a simplified fetcher bolt

This pull request adds a simplified, single-threaded fetcher bolt. The SimpleFetcherBolt is a stripped-down version of the full FetcherBolt that removes fetch items and the item queues. Use of this bolt requires the user to implement queues externally--also necessitating that crawl-delay politeness constraints are handled externally.

For crawl topologies that utilize external queues (RabbitMQ, Kafka, Redis, etc), this will make the topology easier to reason about, and can (potentially) achieve greater throughput and performance tuning.