Add site status checks before attempting a crawl