add everything to do with dns crawling