More robust error handling and obeys robots.txt