Allow all site for bots

Recent changes in google webmaster tools and google crawlers allows for javarendering for bots (see [1] and [2]. This in turn means that we need to allow all resources to be crawled by our robots.txt definition.

In general, we need to monitor if we could probably get rid of our snapshot service alltogether...

[1] http://www.googlewebmastercentral.blogspot.ch/2014/05/understanding-web-pages-better.html
[2] http://googlewebmastercentral.blogspot.ch/