Updated robots rules caching + a RobotsURLFilter

This PR adds: an interface for robots caches; an in-memory, thread-safe cache implementation; and a basic URLFilter for robots rules.