internetarchive/wayback — pull request #33 by kngenie (2014-07-10 00:54:56+00:00)
Title: robots.txt-exclusion is not disabled for embeds

EmbeddedCDXServerIndex has a code that allows us to turns off robots.txt exclusion check for embedded resources (JavaScript, CSS, images, etc) for better replay of captures. It turned out that this is not working at all.

Problem is
* `CDXServer` uses `PrivTokenAuthChecker` for internal requests from Wayback. It is only relevant to CDX Server API calls.
* `PrivTokenAuthChecker.isAllUrlAccessAllowed()` resets `ignoreRobots` flag set by EmbeddedCDXServerIndex for embed resource (this is a confusing side-effect)

Fixed by:
* `PrivTokenAuthChecker` no longer changes `AuthToken.ignoreRobots` flag. Now this flag is just for internal CDX queries.
* Disable robots.txt exclusion check by consulting both `AuthToken.ignoreRobots` flag and auth token name, in `createAccessFilter()` method in `WaybackAuthChecker` and `WaybackAPAuthChecker`.

Possible Incompatibility
* This change can break existing third-party sub-class of `PrivTokenAuthChecker` (ignoreRobots flag no longer work).