Added support for robots.txt adherance