mozilla/bedrock — pull request #1156 by jgmize (2013-08-13 05:44:26+00:00)
Title: Fix Bug 903089 - robots.txt changes

In production, add the following line to robots.txt:

disallow: /*/newsletter/existing/*

In dev, staging, etc:

user-agent: *
disallow: *

settings.SITE_URL is used to determine if we are in production.

In etc/httpd/global.conf, add a RewriteRule to serve robots.txt from
Django.

Add robots.txt to settings.SUPPORTED_NONLOCALES to eliminate redirect.