Search engines crawl or visit sites by passing between the links on pages. However, if you have a new website without links connecting your pages to others, you can ask search engines to crawl your site by submitting your URL on Google Search Console.
Crawlers act as explorers in a new land.
They’re always looking for discoverable links on pages and jotting them down on their map once they understand their features. But website crawlers can only sift through public pages on websites, and the private pages that they can’t crawl are labeled the “dark web.”
Web crawlers, while they’re on the page, gather information about the page like the copy and meta tags. Then, the crawlers store the pages in the index so Google’s algorithm can sort them for their contained words to later fetch and rank for users.