Updated robots.txt detection algorithm