Better managing of exceptions during the crawl