The component of a search engine that indexes web sites automatically. A search engine’s crawler (also called a spider or robot), copies web page source code into its index database and follows links to other web pages.