Further code cleanup.
Use proxies and checker session in robots.txt.