Repository: zulip/zulip

Issue:
test-documentation: Fix output spam from external links
Currently, test-documentation run in a development environment (i.e. without `--skip-external-links`) prints like 2 screenfuls of errors like this:
```
2021-04-01 10:20:38 [documentation_crawler] ERROR: Please check link: https://github.com/zulip/zulip/commit/49dbd85a8985b12666087f9ea36acb6f7da0aa4f
2021-04-01 10:20:38 [documentation_crawler] ERROR: Please check link: https://github.com/zulip/zulip-desktop
2021-04-01 10:20:38 [documentation_crawler] ERROR: Please check link: https://github.com/zulip/zulip/issues/10976
```

I imagine this is really confusing for anyone new to our ReadTheDocs documentation.

Most of these are 429 errors because GitHub doesn't want automation hitting their servers all the time; we could probably just suppress most of them that fit a pattern that we expect to be statically correct (E.g. anything in github.com/zulip/ that is not a tree path).


Discussion:
Hello @zulip/server-tooling, @zulip/server-user-docs members, this issue was labeled with the "area: documentation (user)", "area: tooling" labels, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot claim

Resolving patch:
diff --git a/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py b/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py
--- a/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py
+++ b/tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py
@@ -2,6 +2,7 @@
 import os
 import re
 from typing import Callable, Iterator, List, Optional, Union
+from urllib.parse import urlparse
 
 import scrapy
 from scrapy.http import Request, Response
@@ -11,6 +12,15 @@
 from scrapy.utils.url import url_has_any_extension
 from twisted.python.failure import Failure
 
+EXCLUDED_DOMAINS = [
+    # Returns 429 Rate-Limited Errors
+    "github.com",
+    "gist.github.com",
+    # Returns 503 Errors
+    "www.amazon.com",
+    "gitlab.com",
+]
+
 EXCLUDED_URLS = [
     # Google calendar returns 404s on HEAD requests unconditionally
     "https://calendar.google.com/calendar/embed?src=ktiduof4eoh47lmgcl2qunnc0o@group.calendar.google.com",
@@ -19,6 +29,8 @@
     # Returns 404 to HEAD requests unconditionally
     "https://www.git-tower.com/blog/command-line-cheat-sheet/",
     "https://marketplace.visualstudio.com/items?itemName=rafaelmaiolla.remote-vscode",
+    "https://www.transifex.com/zulip/zulip/announcements/",
+    "https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh",
     # Requires authentication
     "https://circleci.com/gh/zulip/zulip/tree/master",
     "https://circleci.com/gh/zulip/zulip/16617",
@@ -164,6 +176,10 @@ def _make_requests(self, url: str) -> Iterator[Request]:
             callback = self.check_fragment
         if getattr(self, "skip_external", False) and self._is_external_link(url):
             return
+        if urlparse(url).netloc in EXCLUDED_DOMAINS:
+            return
+        if url in EXCLUDED_URLS:
+            return
         yield Request(
             url,
             method=method,
@@ -204,13 +220,12 @@ def retry_request_with_get(self, request: Request) -> Iterator[Request]:
         request.dont_filter = True
         yield request
 
-    def exclude_error(self, url: str) -> bool:
-        return url in EXCLUDED_URLS
-
     def error_callback(self, failure: Failure) -> Optional[Union[Failure, Iterator[Request]]]:
         if isinstance(failure.value, HttpError):
             response = failure.value.response
-            if self.exclude_error(response.url):
+            # Hack: The filtering above does not catch this URL,
+            # likely due to a redirect.
+            if urlparse(response.url).netloc == "idmsa.apple.com":
                 return None
             if response.status == 405 and response.request.method == "HEAD":
                 # Method 'HEAD' not allowed, repeat request with 'GET'


Test patch: