Automated tracking for Alexa's top 100 websites robots.txt