Robots.txt updated for crawlers