Crawl Map is a programme that downloads and indexes content from all across the internet, often known as a spider or a search engine bot. A bot like this has the purpose of learning about (nearly) every webpage on the internet so that information may be obtained when needed.