mysociety/popit — pull request #704 by struan (2014-12-11 16:36:53+00:00)
Title: Allow instances to set robots.txt to Disallow all

Add a no spider option to the about page that when enabled serves up a
robots.txt file that disallows all. If not set then serves up the normal
robots.txt

Also adds middleware to serve up robots.txt which means moving the file
to views as otherwise the static serving kicks in first and we need to
process this later on in order to get access to the instance settings.

This does also have the side effect that we have separate robots.txt
files for the hosting and instance apps.

Fixes #692