In simplest of terms, if you really wish to make sure that your website gets ranked higher during search engine rankings then you must create robots.txt file for your website. With the help of this file the search engine crawl your website and correct information is passed on which proves to be helpful in ranking.