Spiders, the automated robots that search engines send out, use links to navigate their way through billions of pages of content. The bot starts with a few pages, and then follows links on those pages to find new URLs. Links act as a highway system for the internet. Each exit that the spider takes leads it to new content that it can index, like webpages, images, videos, PDFs, etc. Once the pages have been found, the code is deciphered and selected pieces move forward to be stored in an index.