A web crawler is a program that scans the internet to find and index web pages. It’s also called a spider or bot. These programs follow links between pages, reading content and gathering data for search engines.