Revert "add robots.txt which allows to crawl only Google"

This reverts commit 823a8754935ae733d4bbac8b5d5d8d33080d6d28.