It’s an instruction manual for google web crawlers. It forms a part of technical SEO. A robots.txt file tells a search engine the URLs it can exclude indexing. It is used to exclude pages that may not ever rank like login, register pages and thus, optimize the crawl budge. It is found in the root domain folder (www.yourwebsite.com/robots.txt). Google search console provides details of indexed pages.
The following are the elements in a robots.txt file.
User-agent (eg: Google bot, robots.txt)
Allow
Disallow
Regex (*) (any crawlers, follow these rules)