Search engines work by crawling over 1 billion websites with bots (sometimes called spiders or web crawlers) . These bots navigate the web by following internal or external links to find new pages. Once found, they are downloaded, evaluated and added to the search engines index accordingly.