DigitalPebble/storm-crawler — pull request #46 by jakekdodd (2014-12-19 16:14:24+00:00)
Title: Improved robots.txt handling

This PR adds a new mechanism for robots.txt caching and filtering.

Among the changes are:

- A new thread-safe, in-memory singleton cache for robots rules
- A RobotsURLFilter
- Moves the getCacheKey() method to the cache implementation because it's used in several locations
- The ParserBolt ensures that the robots rules are loaded into the shared cache, which means conditionally refetching and parsing the domain's robots.txt file