DigitalPebble/storm-crawler — pull request #20 by jakekdodd (2014-11-04 16:38:45+00:00)
Title: Added a URL normalizing regex filter

This PR adds a new URLFilter, the RegexURLNormalizer. This is a normalizing filter; its purpose is to match and replace substrings of URLs.

The filter is configured through an XML document, specified by the regexNormalizerFile param in ``urlfilters.json``. The PR also includes a ``default-regex-normalizers.xml`` config file. All of the example normalizers are commented out in this file, so the behavior of storm-crawler won't change until the user modifies this file.