Allow all site for bots
Merge pull request #1447 from geoadmin/gjn_adapt_robots

Allow whole site for bots