The 403 Forbidden HTTP status code, as well as other 4xx
HTTP status codes, is interpreted as the robots.txt file doesn't exist. This means
that crawlers will generally assume that they can crawl all URLs of the website. In
order to block crawling of the website, the robots.txt must be returned with a
200 OK HTTP status code, and must contain an appropriate
disallow rule.