Web crawlers are essential search engine tools, so their specific components are considered proprietary information. They contribute to distinguishing search services and define search experience — your experience on Google is different from your experience on Yandex or Bing, for instance. Moreover, your search experience on your own website may vary from that of your competitors depending on how up-to-date, accurate, and relevant the information presented in your search results are. So though different web crawlers work differently, be they Internet or enterprise crawler bots, they share standard architecture and have similar capabilities. They receive a seed URL as input. From there, they can access more URLs along the crawl frontier, which is composed of a list of URLs that have yet to be visited by a crawl bot. Based on a set of policies or crawl rules, such as politeness (what the bot can index) and revisit policies (how often it can crawl), the crawler will continue to visit new URLs. From there, it must have the capacity to render the URL's information, download it at a high speed, index it, and store it in the engine for future use. While Internet crawlers enable a web-wide search, enterprise web crawlers allow your content to be searchable on your site. Some of their capabilities include: Full visibility into crawl activity so you can keep track of crawl performance Programmability, which gives you control of the bot with flexible APIs Easy to use user interfaces