'Robots.txt' must be checked before the url request.