openrightsgroup/Blocking-Middleware — pull request #65 by dantheta (2014-12-28 21:56:36+00:00)
Title: Features/robots txt rebase

This branch adds a new queueing daemon, which will retrieve the robots.txt for a target domain, check that our probes are allowed, and pass the request on to the regular ISP queues, for openrightsgroup/cmp-issues#90

A second content test checks the mime type and content length of the target and rejects if the content type is not text/* (we can expand this) or the content-length is too great. openrightsgroup/cmp-issues#114