The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site and can help manage crawl budget.