Update robots.txt comments

clarify comments and update permanently moved URL