A robot.txt file simply functions as a request that is ignored by specific robots as well as the directories in their search. The robot.txt files are used to provide privacy for many search engine results for the fact that content or information might prove misleading to the visitor.