better error message for urls we are not allowed to crawl
write files with utf-8 encoding