Added Check to ensure robots.txt exists