FIX: ROBOTS checking and updating: fix robots.txt checking and updating for https hosts
FIX: Move some RedisRobotExclusionFilter rules to RobotExclusionFilter (needs another look perhaps)