Implemented robots.txt check