Added error handling when using the crawler to avoid fatal errors

I'm not yet fixing all testsuite failures in this PR. I simply fix the fatal error case to let the testsuite run fully