improves exception handling for single-document crawling