A crawler is a program that browses the Internet to compile a data index. Its goal is to acquire information by crawling like a spider into every web page on the internet. Spider or Spider-bot is another name for it.