CDX/robots: add ignoreRobots option to AuthToken, to explicitly skip robots checking
REFACTOR: Improve robots redis check to support a non-ArcResource response.. allow for handling of plain robots txt
(wrapped in RobotsTxtResource)