Search engines work by searching hundreds of billions of pages using their own website crawlers. These website crawlers are often known as “search engine bots” or “spiders.” Search engines navigate the web by downloading web pages and following links on these pages to discover new pages that have been made available.