So, I think we need a setting like `PROD_URL` in `settings/base.py` that is the same among environments, because the canonical URLs should be the production URLs regardless of the site.

Unfortunately, we **cannot prevent** Google from indexing pages on dev sites by using `robots.txt`. It's useless. [Here's the search results](https://www.google.com/search?q=site:www-dev.allizom.org+OR+site:www.allizom.org&filter=0). The solution is 

1. using meta noindex tag (or HTTP header) on each page, or 
2. using canonical URL on each page that points the production URL

I had used the latter technique on www.mozilla.jp, and it worked well as expected. I also recommend the latter, because Google can aggregate PageRank among sites including dev sites. The former technique will just lose PageRank on dev sites. I'll file a bug to remove `robots.txt` once this PR is merged. SEO consultants may also give advices.