Genre: Instruction

How do I test my robots . txt file ? You can use the robots . txt analysis tool in Google Sitemaps to : Check specific URLs to see if your robots . txt file allows or blocks them . See if Googlebot had trouble parsing any lines in your robots . txt file . Test changes to your robots . txt file . If you do n't currently use a robots . txt file , you can create one and then test it with the tool before you upload it to your site . the robots . txt analysis tool displays the text of your cached robots . txt file . You can enter a list of URLs and check to make sure that file restricts or allows access as you expect . You can also modify the displayed robots . txt file and then enter a list of URLs to check so that you can see how changes to your robots . txt file would change Googlebot's access to specific pages . Once you are happy with your revised robots . txt file , make the changes on the version on your site . Note that it may take up to a day for the Googlebot to retrieve the latest version . You can always check when we last downloaded your robots . txt file at the top of the robots . txt analysis page . You may also be interested in ... How do I use the robots . txt file analysis tool ? What do the robots . txt file analysis results means ? Why did the robots . txt analysis tool display a syntax not understood error ?