DigitalPebble/storm-crawler — pull request #12 by jakekdodd (2014-09-22 14:47:11+00:00)
Title: added regex filtering

Added a regex URL filter. Adapted almost line-for-line from Nutch 1.9.

Only quirk is that the user must specify a file (in urlfilters.json) from which to load regex rules, rather than including the rules themselves in the JSON config. This is because regular expressions typically contain characters that must be escaped within JSON, so it would be a big headache for users to include their regex rules directly within the ``params`` node.