Added some exclusions for robots.txt