Do not download robots.txt twice for HTTP and HTTPS if --protocol-directories is not given