geoadmin/mf-geoadmin3 — pull request #1048 by gjn (2014-02-04 07:48:51+00:00)
Title: Adding robots file and make it deploy target dependant

Currently, we serve a generic robots.txt file which prevents search bots from crawling our web page (see map.geo.admin.ch/robots.txt). This means that even if we search for the official site description in the the meta tag of the site, we won't get any results in google (see https://www.google.ch/?gws_rd=cr&ei=AJvwUri-FqmCyQPkzIGwAw#q=Maps+of+Switzerland+%3A+The+official+geoportal+of+the+Swiss+Confederation+provides+digital+data+which+can+be+viewed%2C+printed+out%2C+ordered+&safe=off)

This PR adds a custom robots.txt which will enable crawling. Note that if the deploy target is not prod, the robots file we serve still prevents the search engines to crawl it. This is in order to _not_ index our dev and int instances.