Like spiders, crawlers are bots that crawl the internet searching for fresh content. This is the discovery phase when webpages, images, videos, or any content is found.
When a crawler finds a new URL through a webpage, it adds it to the search engine's index for processing.