add test case catching WWM-119 (it fails).
	EmbeddexCDXServerIndexTest is sorta wrong place to have this test, but it is much harder to have this test in wayback-cdx-server - clean up after refactoring.
BUGFIX: do not overwrite AuthToken.ignoreRobots flag in PrivTokenAuthChecker.isAllUrlAccessAllowed(AuthToken). it re-enables robots.txt exclusion turned-off for embeds. Run a separate check for ignore-robots token in createAccessFilter().
This is an interim fix until we refactor permission control system.
Fixes WWM-119.
Merge pull request #33 from kngenie/WWM-119

robots.txt-exclusion is not disabled for embeds