openspending/openspending — pull request #768 by nickstenning (2014-02-17 17:08:21+00:00)
Title: robots.txt: fix possible syntax error

GoogleBot has been noisily complaining about OpenSpending's robots.txt
for some time. Further investigation suggests that "Allow: *" is not
syntactically valid. See

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

(the section entitled "Formal syntax / definition"), which implies that
the right-hand side arguments of "Allow:" and "Disallow:" must always
start with a "/".

Given that this is now a fully permissive robots.txt, a blank file will
serve the same purpose, and is unlikely to be syntactically invalid.