robots.txt check added