processor.web: Refactor url filter and robots.txt concerns into processor.rule