Added error handling when using the crawler to avoid fatal errors