alphagov/spotlight — pull request #613 by alexmuller (2014-06-25 09:23:08+00:00)
Title: Send a robots.txt directive to disallow indexing

We'd like to get people comfortable pushing this app to anywhere they want, including platform-as-a-service and software-as-a-service providers.

This will help us to prototype things and experiment more freely.

But we don't want those copies of the app to be indexed by search engines, so send a robots.txt file.

This duplicates [the robots.txt file we send from Puppet][1], but also applies to services beyond our control.

This does not affect GOV.UK, because the only parts of the site that are routed to this application are below /performance.

[1]: https://github.com/alphagov/pp-puppet/blob/jenkins-performanceplatform_puppet-616/modules/performanceplatform/manifests/proxy_vhost.pp#L170-179