Repository: mdn/kuma

Issue:
T - Update robots.tx (Warnings have spiked on Search Console)
**Summary**
We are seeing a large spike in warnings on Google Search Console. The spike started on December 22, 2019.

<img width="952" alt="Coverage" src="https://user-images.githubusercontent.com/557852/72343521-5fb44000-36cf-11ea-8f1f-12955b98e34d.png">

**Steps To Reproduce (STR)**
1. Go to Search Console
2. Open Coverage Report
3. open "Indexed, though blocked by robots.txt"


**Actual behavior**
It looks like a bunch of URLs are blocked from indexing by Google that should not be blocked. Examples are:
* https://developer.mozilla.org/it/docs/Web/API/GlobalEventHandlers/onresize$translate
* https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/end%24edit
* https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidateInit/sdpMid$translate?tolocale=de

When I test these URLs in Search Console's robots.txt tester it tells me that those URLs are allowed, and they should be. $translate, or $edit are not in robots.txt


**Expected behavior**
Those URLs should not be blocked, so that Google can crawl the pages, discover the no-index meta tag and remove them from its index.


**Additional context**
I assume that something is broken when it comes to handling the $ chracter in the URL. Maybe we just remove everything with the $url from our robots.txt. As far as I can tell, almost all of those URLs now live under wiki.developer.mozilla.org anyway.


Discussion:
Granted that the Wiki editing pages should perhaps technically not be *blocked* but surely they shouldn't be indexed, no?

The `<meta name="robots" content="noindex, nofollow">` comes from the fact that `wiki.developer.mozilla.org` is *not* in `settings.ALLOW_ROBOTS_WEB_DOMAINS`.


The whole wiki domain is blocked from crawling.

Re indexing: If you block a page in robots.txt, you make it impossible for Google to see its content. That means it will index the naked URL that it found somewhere. If you allowed it to crawl the page (allowed in robots.txt) Google could see that you don't want the page to be indexed and remove it from the index.
Some more context from Raphael: The weird behavior we are seeing is because those URLs are forwarded to wiki.developer.mozilla.org which has blocked all crawling via robots. txt

So, the solution to this issue would be to remove that block from robots.txt on the wiki and add the no-index, no-follow meta tag to all wiki pages.
So, allow it in so it can see for itself that everything is explicitly blocked per page? Then, after some period of time, reinstate the `robots.txt` that disallows everything. 
Yes, let is crawl and discover the no-index meta-tags, but no, don't add robots.txt again. There will be ever more links from the read-only page to the wiki, and there is nothing to be gained from blocking crawling in robots.txt. We initially did that, because of performance concerns on MDN, but that's not really a thing for the wiki.
So, the AC are:
- [ ] all pages in the wiki have the no-index, no-follow meta tag
- [ ] robots.txt is set to allow all.
![Screen Shot 2020-02-14 at 15 31 55](https://user-images.githubusercontent.com/4051932/74539902-27f31f00-4f3f-11ea-872c-cea00c8dae62.png)
Trying to wrap my head around this now, but I discovered that the spike has passed. Just in case this is unexpected, I thought it'd be worthwhile to highlight it here.
It doesn't look as dramatic, but it's still 30k warnings. That was the original spike. 
Sent in a PR for noindex-ing the whole wiki. ~~The robots.txt stuff happens through a setting over in mdn-infra. Ryan gave me an intro to it this week, so I'm gonna give that a stab as well.~~ Nevermind, I think Peter would tell me to change the defaults instead, so I'm gonna make that part of the other PR.

Resolving patch:
diff --git a/kuma/settings/common.py b/kuma/settings/common.py
--- a/kuma/settings/common.py
+++ b/kuma/settings/common.py
@@ -1140,7 +1140,11 @@ def pipeline_one_scss(slug, **kwargs):
 # Allow robots, but restrict some paths
 # If the domain is a CDN, the CDN origin should be included.
 ALLOW_ROBOTS_WEB_DOMAINS = set(
-    config("ALLOW_ROBOTS_WEB_DOMAINS", default="developer.mozilla.org", cast=Csv())
+    config(
+        "ALLOW_ROBOTS_WEB_DOMAINS",
+        default="developer.mozilla.org,wiki.developer.mozilla.org",
+        cast=Csv(),
+    )
 )
 
 # Allow robots, no path restrictions


Test patch: