complete check_robots_txt method