Open up robots.txt a bit more, conservatively, and allow the IA to crawl anything we've got.
Merge branch 'robots-tweaking' into beta

* robots-tweaking:
  Open up robots.txt a bit more, conservatively, and allow the IA to crawl anything we've got.