Robots.txt file: Websites use a robots.txt file to specify which pages or directories should not be crawled by search engines.
Meta tags (nofollow, noindex): Meta tags like nofollow instruct search engines not to follow links on a page, while noindex tells them not to include the page in search results.
HTTP status codes (403, 404): Websites return HTTP status codes like 403 (Forbidden) or 404 (Not Found) to indicate that certain pages or resources are inaccessible for crawling.
Crawl-delay directive: The crawl-delay directive in robots.txt allows website owners to define a delay between requests made by search engine bots to control the rate of crawling.
CAPTCHA or authentication: Websites implement CAPTCHA challenges or require user authentication to prevent automated crawlers from accessing restricted content.