Repository: mdn/kuma

Issue:
T - Update robots.txt (remove sites)
**Summary**
There are several page types that we have in our robots.txt, which means that Google can't crawl it to see that we don't want those pages to be indexed.


**Steps To Reproduce (STR)**
1. Go to Search Console
2. go to Coverage > Valid with Warnings > Indexed, though blocked by robots.txt
3. Alternatively: https://developer.mozilla.org/robots.txt


**Actual behavior**
Google has a link to https://developer.mozilla.org/en-US/docs/tag/SpeechRecognitionResult (for example), but that URL is blocked in robots.txt, so it can't follow the link to see that it redirects to https://wiki.developer.mozilla.org/en-US/docs/tag/SpeechRecognitionResult which has a `no-index` directive.


**Expected behavior**
All page types that redirect to the wiki should be removed from the robots.txt of the read-only site, since the wiki has a `no-index` directive on all content already.


**Additional context**
The reason to do this is so we can see actually problematic content show up in our search console reports, instead of this noise. Search console only shows up to 1000 pages as problematic, but there are currently more than 10k warnings, so we might be missing large issues.



Discussion:
URLs that should be removed from robots.txt on the read-only site:

Disallow: /admin/
Disallow: /api/
Disallow: /*/dashboards/*
Disallow: /*docs/feeds
Disallow: /*docs/templates
Disallow: /*docs*Template:
Disallow: /*docs/all
Disallow: /*docs/tag*
Disallow: /*docs/needs-review*
Disallow: /*docs/localization-tag*
Disallow: /*docs/with-errors
Disallow: /*docs/without-parent
Disallow: /*docs/top-level
Disallow: /*docs/new
Disallow: /*docs/get-documents
Disallow: /*docs/submit_akismet_spam
Disallow: /*docs/load*
Disallow: /*docs/Experiment:*
Disallow: /*$api
Disallow: /*$compare
Disallow: /*$revision
Disallow: /*$history
Disallow: /*$children
Disallow: /*$flag
Disallow: /*$locales
Disallow: /*$toc
Disallow: /*$move
Disallow: /*$quick-review
Disallow: /*$samples
Disallow: /*$revert
Disallow: /*$repair_breadcrumbs
Disallow: /*$delete
Disallow: /*$restore
Disallow: /*$purge
Disallow: /*$subscribe
Disallow: /*$subscribe_to_tree
Disallow: /*$vote
Disallow: /*docs.json
Disallow: /*docs/ckeditor_config.js
Disallow: /*/files/
Disallow: /media
Disallow: /*move-requested
Disallow: /*preview-wiki-content
Disallow: /*profiles*/edit
Disallow: /skins
Disallow: /*type=feed
Disallow: /*users/
@atopal Now that I see your comments, I realize that I should have caught this earlier. You're right on, thanks for digging into this Kadir!
No worries Ryan, we had intentionally limited the previous issue.
@atopal I've been so consumed with #6818, that I haven't done this yet! Sorry for the delay! I'm going to work on this today and get it pushed to stage/prod either today or tomorrow.

Resolving patch:
diff --git a/kuma/landing/views.py b/kuma/landing/views.py
--- a/kuma/landing/views.py
+++ b/kuma/landing/views.py
@@ -61,52 +61,14 @@ def promote_buttons(request):
 User-agent: *
 Sitemap: https://developer.mozilla.org/sitemap.xml
 
-Disallow: /admin/
 Disallow: /api/
-Disallow: /*/dashboards/*
-Disallow: /*docs/feeds
-Disallow: /*docs/templates
-Disallow: /*docs*Template:
-Disallow: /*docs/all
-Disallow: /*docs/tag*
-Disallow: /*docs/needs-review*
-Disallow: /*docs/localization-tag*
-Disallow: /*docs/with-errors
-Disallow: /*docs/without-parent
-Disallow: /*docs/top-level
-Disallow: /*docs/new
 Disallow: /*docs/get-documents
-Disallow: /*docs/submit_akismet_spam
-Disallow: /*docs/load*
 Disallow: /*docs/Experiment:*
-Disallow: /*$api
-Disallow: /*$compare
-Disallow: /*$revision
-Disallow: /*$history
 Disallow: /*$children
-Disallow: /*$flag
-Disallow: /*$locales
-Disallow: /*$toc
-Disallow: /*$move
-Disallow: /*$quick-review
-Disallow: /*$samples
-Disallow: /*$revert
-Disallow: /*$repair_breadcrumbs
-Disallow: /*$delete
-Disallow: /*$restore
-Disallow: /*$purge
-Disallow: /*$subscribe
-Disallow: /*$subscribe_to_tree
-Disallow: /*$vote
 Disallow: /*docs.json
-Disallow: /*docs/ckeditor_config.js
 Disallow: /*/files/
 Disallow: /media
-Disallow: /*move-requested
-Disallow: /*preview-wiki-content
 Disallow: /*profiles*/edit
-Disallow: /skins
-Disallow: /*type=feed
 Disallow: /*users/
 """ + "\n".join(
     "Disallow: /{locale}/search".format(locale=locale)


Test patch:
diff --git a/kuma/landing/tests/test_views.py b/kuma/landing/tests/test_views.py
--- a/kuma/landing/tests/test_views.py
+++ b/kuma/landing/tests/test_views.py
@@ -83,7 +83,7 @@ def test_robots_not_allowed(client):
     content = response.content
     assert b"Sitemap: " not in content
     assert b"Disallow: /\n" in content
-    assert b"Disallow: /admin/\n" not in content
+    assert b"Disallow: /api/\n" not in content
 
 
 def test_robots_allowed_main_website(client, settings):
@@ -98,7 +98,7 @@ def test_robots_allowed_main_website(client, settings):
     content = response.content
     assert b"Sitemap: " in content
     assert b"Disallow: /\n" not in content
-    assert b"Disallow: /admin/\n" in content
+    assert b"Disallow: /api/\n" in content
 
 
 def test_robots_all_allowed_wiki_website(client, settings):