FIX: RobotRules tracks Allow: rules and clears the disallow list when an Allow is encountered!
UpdateRobots handler turning off timeout by default, too confusing