Added robots.txt check