Repository: mdn/kuma

Issue:
T - Update robots.tx (Warnings have spiked on Search Console)
**Summary**
We are seeing a large spike in warnings on Google Search Console. The spike started on December 22, 2019.

<img width="952" alt="Coverage" src="https://user-images.githubusercontent.com/557852/72343521-5fb44000-36cf-11ea-8f1f-12955b98e34d.png">

**Steps To Reproduce (STR)**
1. Go to Search Console
2. Open Coverage Report
3. open "Indexed, though blocked by robots.txt"


**Actual behavior**
It looks like a bunch of URLs are blocked from indexing by Google that should not be blocked. Examples are:
* https://developer.mozilla.org/it/docs/Web/API/GlobalEventHandlers/onresize$translate
* https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/end%24edit
* https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidateInit/sdpMid$translate?tolocale=de

When I test these URLs in Search Console's robots.txt tester it tells me that those URLs are allowed, and they should be. $translate, or $edit are not in robots.txt


**Expected behavior**
Those URLs should not be blocked, so that Google can crawl the pages, discover the no-index meta tag and remove them from its index.


**Additional context**
I assume that something is broken when it comes to handling the $ chracter in the URL. Maybe we just remove everything with the $url from our robots.txt. As far as I can tell, almost all of those URLs now live under wiki.developer.mozilla.org anyway.


Discussion:
Granted that the Wiki editing pages should perhaps technically not be *blocked* but surely they shouldn't be indexed, no?

The `<meta name="robots" content="noindex, nofollow">` comes from the fact that `wiki.developer.mozilla.org` is *not* in `settings.ALLOW_ROBOTS_WEB_DOMAINS`.


The whole wiki domain is blocked from crawling.

Re indexing: If you block a page in robots.txt, you make it impossible for Google to see its content. That means it will index the naked URL that it found somewhere. If you allowed it to crawl the page (allowed in robots.txt) Google could see that you don't want the page to be indexed and remove it from the index.
Some more context from Raphael: The weird behavior we are seeing is because those URLs are forwarded to wiki.developer.mozilla.org which has blocked all crawling via robots. txt

So, the solution to this issue would be to remove that block from robots.txt on the wiki and add the no-index, no-follow meta tag to all wiki pages.
So, allow it in so it can see for itself that everything is explicitly blocked per page? Then, after some period of time, reinstate the `robots.txt` that disallows everything. 
Yes, let is crawl and discover the no-index meta-tags, but no, don't add robots.txt again. There will be ever more links from the read-only page to the wiki, and there is nothing to be gained from blocking crawling in robots.txt. We initially did that, because of performance concerns on MDN, but that's not really a thing for the wiki.
So, the AC are:
- [ ] all pages in the wiki have the no-index, no-follow meta tag
- [ ] robots.txt is set to allow all.
![Screen Shot 2020-02-14 at 15 31 55](https://user-images.githubusercontent.com/4051932/74539902-27f31f00-4f3f-11ea-872c-cea00c8dae62.png)
Trying to wrap my head around this now, but I discovered that the spike has passed. Just in case this is unexpected, I thought it'd be worthwhile to highlight it here.
It doesn't look as dramatic, but it's still 30k warnings. That was the original spike. 
Sent in a PR for noindex-ing the whole wiki. ~~The robots.txt stuff happens through a setting over in mdn-infra. Ryan gave me an intro to it this week, so I'm gonna give that a stab as well.~~ Nevermind, I think Peter would tell me to change the defaults instead, so I'm gonna make that part of the other PR.
We want
```
User-agent: *
Disallow:
```
for the wiki domain

Resolving patch:
diff --git a/kuma/landing/views.py b/kuma/landing/views.py
--- a/kuma/landing/views.py
+++ b/kuma/landing/views.py
@@ -50,6 +50,13 @@ def promote_buttons(request):
     return render(request, "landing/promote_buttons.html")
 
 
+ROBOTS_ALL_ALLOWED_TXT = """\
+User-agent: *
+Sitemap: https://wiki.developer.mozilla.org/sitemap.xml
+
+Disallow:
+"""
+
 ROBOTS_ALLOWED_TXT = """\
 User-agent: *
 Sitemap: https://developer.mozilla.org/sitemap.xml
@@ -119,7 +126,10 @@ def robots_txt(request):
     if host in settings.ALLOW_ROBOTS_DOMAINS:
         robots = ""
     elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:
-        robots = ROBOTS_ALLOWED_TXT
+        if host == settings.WIKI_HOST:
+            robots = ROBOTS_ALL_ALLOWED_TXT
+        else:
+            robots = ROBOTS_ALLOWED_TXT
     else:
         robots = ROBOTS_GO_AWAY_TXT
     return HttpResponse(robots, content_type="text/plain")


Test patch:
diff --git a/kuma/landing/tests/test_views.py b/kuma/landing/tests/test_views.py
--- a/kuma/landing/tests/test_views.py
+++ b/kuma/landing/tests/test_views.py
@@ -101,6 +101,23 @@ def test_robots_allowed_main_website(client, settings):
     assert b"Disallow: /admin/\n" in content
 
 
+def test_robots_all_allowed_wiki_website(client, settings):
+    """On the wiki website, allow robots with NO restrictions."""
+    host = "main.mdn.moz.works"
+    wiki_host = "wiki." + host
+    settings.WIKI_HOST = wiki_host
+    settings.ALLOWED_HOSTS = [host, wiki_host]
+    settings.ALLOW_ROBOTS_WEB_DOMAINS = [host, wiki_host]
+    response = client.get(reverse("robots_txt"), HTTP_HOST=wiki_host)
+    assert response.status_code == 200
+    assert_shared_cache_header(response)
+    assert response["Content-Type"] == "text/plain"
+    content = response.content
+    assert b"Sitemap: " in content
+    assert b"Disallow:\n" in content
+    assert b"Disallow: /" not in content
+
+
 def test_robots_allowed_main_attachment_host(client, settings):
     """On the main attachment host, allow robots without restrictions."""
     host = "samples.mdn.moz.works"
diff --git a/tests/headless/__init__.py b/tests/headless/__init__.py
--- a/tests/headless/__init__.py
+++ b/tests/headless/__init__.py
@@ -17,7 +17,8 @@
 }
 
 # Kuma web domains that are indexed
-INDEXED_WEB_DOMAINS = {"developer.mozilla.org", "wiki.developer.mozilla.org"}
+WIKI_HOST = "wiki.developer.mozilla.org"
+INDEXED_WEB_DOMAINS = {"developer.mozilla.org", WIKI_HOST}
 
 
 def request(method, url, **kwargs):
diff --git a/tests/headless/test_robots.py b/tests/headless/test_robots.py
--- a/tests/headless/test_robots.py
+++ b/tests/headless/test_robots.py
@@ -3,13 +3,13 @@
 import pytest
 import requests
 
-from . import INDEXED_ATTACHMENT_DOMAINS, INDEXED_WEB_DOMAINS
+from . import INDEXED_ATTACHMENT_DOMAINS, INDEXED_WEB_DOMAINS, WIKI_HOST
 
 
 @pytest.mark.smoke
 @pytest.mark.headless
 @pytest.mark.nondestructive
-def test_robots(any_host_url):
+def test_robots(any_host_url, wiki_host):
     url = any_host_url + "/robots.txt"
     response = requests.get(url)
     assert response.status_code == 200
@@ -20,6 +20,10 @@ def test_robots(any_host_url):
         assert response.text.strip() == ""
     elif hostname in INDEXED_WEB_DOMAINS:
         assert "Sitemap: " in response.text
-        assert "Disallow: /admin/\n" in response.text
+        if hostname == WIKI_HOST:
+            assert "Disallow:\n" in response.text
+            assert "Disallow: /" not in response.text
+        else:
+            assert "Disallow: /admin/\n" in response.text
     else:
         assert "Disallow: /\n" in response.text