A robots.txt file guides search engine crawlers on which pages to index and which to ignore. This helps manage crawl efficiency and ensure that important content is prioritized.