bug 866873 - robots.txt accordingly
Merge pull request #333 from peterbe/bug866873-robotstxt-accordingly

bug 866873 - robots.txt accordingly