DigitalPebble/storm-crawler — pull request #40 by jakekdodd (2014-12-17 16:50:02+00:00)
Title: Updated robots rules caching + a RobotsURLFilter

This PR adds: an interface for robots caches; an in-memory, thread-safe cache implementation; and a basic URLFilter for robots rules.