FIX: UpdateRobots request handler canonicalizes urls to http://www (for now)
as that's what's checked in redis first