Update robots.txt comments

clarify comments and update permanently moved URL
Merge pull request #4222 from rivendale2010/patch-1

Update robots.txt comments