There’s a range of errors that can affect your robots.txt file, as well as some ‘best practice’ warnings that you might see when you check your file. These, are things that can affect your SEO and should be fixed. Warnings are less important and act as advice on how to improve your robots.txt file.
Errors you may see include:
Invalid URL – You’ll see this error if your robots.txt file is missing completely
Potential wildcard error – Although technically a warning rather than an error, if you see this message it’s usually because your robots.txt file contains a wildcard (*) in the Disallow field (e.g. Disallow: /*.rss). This is a best practice issue – Google allows wildcards in the Disallow field but it’s not a recommended practice.
Generic and specific user-agents in the same block of code – This is a syntax error in your robots.txt file and should be corrected to avoid problems with crawling your website.
Warnings you may see include:
Allow: / – Using the allow order isn’t going to damage your ranking or affect your website, but it’s not standard practice. Major robots including Google and Bing will accept this directive, but not all crawlers do – and generally speaking, it’s best to make your robots.txt file compatible with all crawlers, not just the big ones.
Field name capitalization – While field names are not necessarily case sensitive, some crawlers may require capitalization, therefore it’s a good idea to capitalize field names for specific user-agents
Sitemap support – Many robots.txt files include the details of the sitemap for the website, but this is not considered to be best practice. Google and Bing, however, both support this feature.