whoops, actually fixed the crawl delay problem when robots.txt is not able to be retreived

Signed-off-by: mep <miethpo@gmail.com>