Repository: mdn/kuma

Issue:
Legacy namespace documents are indexed/followed in production
**Summary**
Legacy namespace documents, like https://developer.mozilla.org/en-US/docs/User:anonymous:uitest, are given a `robots` `meta` tag of `index, follow`. This may not be important, but I just wanted to record it since it's a deviation from the Wiki view. I discovered this when running the new and improved headless integration tests against production ([the test `test_user_document`](https://github.com/mdn/kuma/blob/b94d91211ed959121b019fc31bf6c63087cac5ff/tests/headless/test_endpoints.py#L126) fails).

**Steps To Reproduce (STR)**
1. Visit https://developer.mozilla.org/en-US/docs/User:anonymous:uitest
2. View the source HTML

**Actual behavior**
<img width="416" alt="image" src="https://user-images.githubusercontent.com/3743693/67247317-d36ae780-f415-11e9-96d0-56a2ac2be996.png">

**Expected behavior**
```
...
<meta name="robots" content="noindex, nofollow">
...
```

**Additional context**
See the [new document template](https://github.com/mdn/kuma/blob/b94d91211ed959121b019fc31bf6c63087cac5ff/kuma/wiki/jinja2/wiki/react_document.html#L15) as opposed to the [Wiki document template](https://github.com/mdn/kuma/blob/b94d91211ed959121b019fc31bf6c63087cac5ff/kuma/wiki/jinja2/wiki/document.html#L41).



Discussion:
The good news is that it's still not mentioned in [the sitemap](https://developer.mozilla.org/sitemaps/en-US/sitemap.xml) but I suspect Googlebot might pick these up because there's no reason not to. 

P1?
Yeah, not just that. We are also asking Google to index pages that haven't been localized yet, like https://developer.mozilla.org/de/docs/Mozilla/Add-ons/WebExtensions/API/downloads/BooleanDelta

Can we see how these rules changed? And yeah, I'd pull that into the next sprint as a P1
Hmm... I scraped that one and ran it locally. In the read-only side I get `<meta name="robots" content="noindex, nofollow">` just the same as I get with `wiki.mdn.localhost:8000`. 
Working on it. 

Resolving patch:
diff --git a/kuma/wiki/views/document.py b/kuma/wiki/views/document.py
--- a/kuma/wiki/views/document.py
+++ b/kuma/wiki/views/document.py
@@ -810,7 +810,9 @@ def react_document(request, document_slug, document_locale):
     slug_dict = split_slug(document_slug)
 
     # Is there a document at this slug, in this locale?
-    doc, _ = _get_doc_and_fallback_reason(document_locale, document_slug)
+    doc, fallback_reason = _get_doc_and_fallback_reason(
+        document_locale,
+        document_slug)
 
     if doc is None:
         # We can throw a 404 immediately if the request type is HEAD.
@@ -819,7 +821,7 @@ def react_document(request, document_slug, document_locale):
             raise Http404
 
         # Check if we should fall back to default locale.
-        fallback_doc, _, redirect_url = _default_locale_fallback(
+        fallback_doc, fallback_reason, redirect_url = _default_locale_fallback(
             request, document_slug, document_locale)
         if fallback_doc is not None:
             doc = fallback_doc
@@ -861,6 +863,28 @@ def react_document(request, document_slug, document_locale):
     doc_api_data = document_api_data(doc)
     document_data = doc_api_data['documentData']
 
+    def robots_index():
+        if fallback_reason:
+            return False
+
+        if not doc.html:
+            return False
+
+        if doc.is_experiment:
+            return False
+
+        if doc.has_legacy_namespace:
+            return False
+
+        if request.get_host() not in settings.ALLOW_ROBOTS_WEB_DOMAINS:
+            return False
+
+        return True
+
+    robots_meta_content = (
+        'index, follow' if robots_index() else 'noindex, nofollow'
+    )
+
     # Bundle it all up and, finally, return.
     context = {
         'document_data': document_data,
@@ -869,6 +893,7 @@ def react_document(request, document_slug, document_locale):
         # to be bundled up into the json object above instead.
         'seo_summary': seo_summary,
         'seo_parent_title': seo_parent_title,
+        'robots_meta_content': robots_meta_content,
     }
     response = render(request, 'wiki/react_document.html', context)
 


Test patch: