Remove rdoctask as it is no longer supported and raised an exception
Remove exception handling around robots.txt retrieval as it gave the caller nothing useful and simply printed a message to stdout