scrapinghub/splash — pull request #75 by kmike (2014-02-11 23:24:41+00:00)
Title: [WIP] Request filters and more versatile logging

I failed to split it into several PRs because code became too intertwined.

This PR adds two new features: network request filters and logging verbosity option.

Verbosity works this way:

1. Splash can be started with `-v / --verbosity` option that should have a value in range 0-3.
2. Verbosity=3 means "log everything", including download progress of individual files from a web page.
3. Verbosity=2 means "verbose logging", including download requests to individual files from a web page.
4. Verbosity=1 roughly means existing logging.
5. Verbosity=0 turns off some existing messages (not all). Maybe make splash silent when verbosity=0?

To add a request filter user should do the following:

1. Create a folder and put one or more `.txt` files into it. Files should contain rules in Adblock Plus format, one rule per line. Files from https://easylist.adblockplus.org/en/ should work. Let's say we've created `filter1.txt`, `filter2.txt` and `filter3.txt` files.
2. Start splash with `--filters-path` option pointing to this folder.
3. Use `filters` GET argument with comma-separated list of filter names that should be enabled for a request, without `.txt` extensions. For example, `filters=filter3,filter1`.

All filter files are parsed when splash starts; parsed/compiled filters are kept in memory. Rules based on HTML are not supported (they are silently discarded). Rules that use options other than 'domain' are also discarded.

I think we can add support for custom options for limiting requests by headers (e.g. content-type and content-size). Supporting existing Adblock Plus options like `third-party` or `image` also looks useful - see https://adblockplus.org/en/filters.

Splash works without adblockparser installed if `--filters-path` is not used.

Discarding `woff` and `ttf` files using a filter fixes https://github.com/scrapinghub/splash/issues/16 for me.

Questions:

1. What about adding support for "default" filters in "default.txt" files? They will be active if no other filters are selected via GET parameter.
2. What about switching proxy blacklist/whitelist to Adblock Plus rules format (they are currently regexes)?
3. Should `-v0` make splash silent?

TODO:

* [ ] debian packaging fixes
* [ ] README
* [ ] there is naming confusion between "request filters" as middleware and "filters" as adblock rules