[TASK] Allow access from search bots in robots.txt