Repository: python-discord/site

Issue:
Opening tag page forces request to GitHub
Local testing with `httrack` for broken links results in these errors being printed to the console:

```
Traceback (most recent call last):
  File "/home/jc/.cache/pypoetry/virtualenvs/site-_f36zgtP-py3.11/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jc/.cache/pypoetry/virtualenvs/site-_f36zgtP-py3.11/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jc/.cache/pypoetry/virtualenvs/site-_f36zgtP-py3.11/lib/python3.11/site-packages/sentry_sdk/integrations/django/views.py", line 84, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jc/.cache/pypoetry/virtualenvs/site-_f36zgtP-py3.11/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in view
    self.setup(request, *args, **kwargs)
  File "/home/jc/workspace/pydis-site/pydis_site/apps/content/views/tags.py", line 32, in setup
    self.tag = utils.get_tag(kwargs.get("location"))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jc/workspace/pydis-site/pydis_site/apps/content/utils.py", line 245, in get_tag
    set_tag_commit(tag)
  File "/home/jc/workspace/pydis-site/pydis_site/apps/content/utils.py", line 148, in set_tag_commit
    data.raise_for_status()
  File "/home/jc/.cache/pypoetry/virtualenvs/site-_f36zgtP-py3.11/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 rate limit exceeded' for url 'https://api.github.com/repos/python-discord/bot/commits?path=/bot/resources/tags/strip-gotcha.md'
For more information check: https://httpstatuses.com/403
"GET /pages/tags/strip-gotcha/ HTTP/1.1" 500 95740
```

One for each accessed tag. Suboptimal.

I think it would be better if this would soft-fail, or better, if it wouldn't run a separate request for each tag. Alternatively as a quick fix we could  add the "set GitHub token" as a precondition.


Discussion:

Resolving patch:
diff --git a/pydis_site/apps/content/utils.py b/pydis_site/apps/content/utils.py
--- a/pydis_site/apps/content/utils.py
+++ b/pydis_site/apps/content/utils.py
@@ -1,8 +1,10 @@
 import datetime
 import functools
 import json
+import logging
 import tarfile
 import tempfile
+from http import HTTPStatus
 from io import BytesIO
 from pathlib import Path
 
@@ -18,6 +20,7 @@
 from .models import Commit, Tag
 
 TAG_CACHE_TTL = datetime.timedelta(hours=1)
+log = logging.getLogger(__name__)
 
 
 def github_client(**kwargs) -> httpx.Client:
@@ -144,9 +147,26 @@ def set_tag_commit(tag: Tag) -> None:
 
     # Fetch and set the commit
     with github_client() as client:
-        data = client.get("/repos/python-discord/bot/commits", params={"path": path})
-        data.raise_for_status()
-        data = data.json()[0]
+        response = client.get("/repos/python-discord/bot/commits", params={"path": path})
+        if (
+            # We want to hop out early in three cases:
+            # - We got a forbidden response. (GitHub wrongfully uses this for rate limits.)
+            # - We got ratelimited.
+            response.status_code in (HTTPStatus.FORBIDDEN, HTTPStatus.TOO_MANY_REQUESTS)
+            # - GitHub has unicorn time again and is returning 5xx codes.
+            or int(response.status_code / 100) == 5
+        ):  # pragma: no cover
+            log.warning(
+                "Received code %d from GitHub for commit history for bot file %r",
+                response.status_code, path,
+            )
+            # We hop out early because otherwise, these failures may result in the
+            # overall request to the tag page breaking.
+            return
+
+        # This should only be permanent issues from here, such as bad requests.
+        response.raise_for_status()
+        data = response.json()[0]
 
     commit = data["commit"]
     author, committer = commit["author"], commit["committer"]


Test patch: