added exclusion of robots.txt