Added known crawlers to the crawler list and requerying of whois XML if there is an error