Repository: readthedocs/readthedocs.org

Issue:
Addons: make the API endpoint as cacheable as possible
Once we are fully migrated to the new Addons, we will be hitting the `/_/addons/` endpoint _a lot_. This endpoint will be hit by every single request on a documentation page view. We would like to make this endpoint as fast as possible, but also try to cache it to avoid hitting our servers. It can be compared with the current Footer API.

The `/_/addons/` endpoint needs to be cached by:
- `url` query string parameter
- `X-RTD-Hosting-Integrations-Version` HTTP header
- `X-RTD-Hosting-Integrations-API-Version`  HTTP header (once https://github.com/readthedocs/readthedocs.org/pull/10276 gets implemented)

Note the endpoint is behind authentication, so that it's not an issue here.

To build the response, the endpoint depends on:

- Project instance
- Version instance
- Build instance

We need to use the "backend resolver" to build the `base_url` required by the DocDiff addons. That URL depends on the exact filename of the page, so we need cache by `url` parameter. We could talk about different ideas here to be able to not depend on it if we want to. Some ideas the mentioned:

- create a new endpoint to use the backend resolver, and call it for docdiff
- copy / re-implement the "backend resolver" logic in the frontend
- make the current endpoint "expandable" and only return docdiff data when users enable the docdiff in their docs
- add a new query string parameter `docdiff=[true|false]` that's only `true` on pageviews from `*.readthedocs.build`

The last item seems to be the easiest one and will give us most of the benefits, since we 95% of our traffic will call the endpoint with `docdiff=false` 😄 


Discussion:
> `X-RTD-Hosting-Integrations-Version` HTTP header
> `X-RTD-Hosting-Integrations-API-Version`  HTTP header

We also talked about moving these to query string paramenters because it's easier to cache them than HTTP headers: https://developers.cloudflare.com/cache/how-to/cache-keys/#create-custom-cache-keys

I took a look at the Algolia integration we have setup on https://rtd-with-docsearch.readthedocs.io/en/latest/ and it seems they send all the versions in the query string. Something to consider 😉 

![Screenshot_2023-07-17_16-59-01](https://github.com/readthedocs/readthedocs.org/assets/244656/e158c738-1f99-4cc5-9b7d-43d1097d1374)

> We also talked about moving these to query string paramenters because it's easier to cache them than HTTP headers

I implemented this part in #10753 and https://github.com/readthedocs/addons/pull/138


Currently, we are using `unresolver.unresolve_url(url)` to get the `Project` and `Version` objects. I don't think we can easily get rid of the `url=` parameter since we need to, somehow, communicate to the endpoint the project and version the user is reading.

However, it would be interesting to figure it out how to do this because in that case, it will allow us to cache the request only "per project & version & build [^1]" instead of "per project & version & build **&** _exact URL_" as we are currently doing now. There are some ideas here:

1. use the `Referer` HTTP header injected automatically by the browser
2. inject some meta HTML tags in the CF worker: `<meta name="project_slug" content="test-builds">` and `<meta name="version_slug" content="latest">`. Then our JS client could send them in the request `/_/addons/?project_slug=test-builds&version_slug="latest"`

I think 1) is probably the easiest and simplest option since 2) seems pretty hacky 😅 . Following that pattern, I think this could be as follows:

- requests made from **regular** documentation pages:
  - get the `Referer` HTTP header and call `unresolver.unresolve_url(referer)`
  - cache these requests normally (using the `project:version` CDN tags we are already using)
  - do not return any data that depends on the _exact URL_
- requests made on documentation pages that **depend on their exact URL** (e.g. pull request previews)
  - send the `url=` attribute with the `window.location.href` value
  - use the `url=` attribute to call the unresolver
  - populate response data that depends on the exact URL (e.g. `doc_diff.base_url`)
  - cache these requests normally. They won't interfere with the previous cached requests because they have an extra `url=` attribute. They also won't interfere between themselves because they all have different `url=` values and CF will treat them differently 💪🏼 

This sounds pretty simple to understand and follow in my head, but I've been thinking on this pattern for some time already. I'd like to get some feedback on this and decide how we want to move forward.

Note that this will have an impact in our web instances and also in our database. Besides, it will make our addons rendering a lot faster since most of the requests to the endpoint will be cached and returned directly from CF. Currently, the numbers for this endpoint in the last 7 days the response times are:

* AVG: 129 ms
* Median: 76.5 ms
* 95th percentile: 160 ms

[^1]: the build is automatically fetched (no need to pass it to the endpoint) by querying the latest successful build for that particular version.
> get the `Referer` HTTP header and call `unresolver.unresolve_url(referer)`

Hrm, thinking a little more about this, isn't this the same as passing the `url=` attribute? I mean, instead of vary the cache on the `url=` we will need to vary it on `Referer` header, which will have exactly the same behavior, right?

Maybe option 2), even being hacky 🕵️, is the only option that works how we want 😄 
Oh, wait! We are already returning `X-RTD-project` and `X-RTD-Version` headers to the browser 🚀  . This solves this problem in a much better way 💯 

```
$ curl -sIL https://docs.readthedocs.io/en/stable/ | grep x-rtd-
x-rtd-domain: docs.readthedocs.io
x-rtd-path: /proxito/html/docs/stable/index.html
x-rtd-project: docs
x-rtd-project-method: public_domain
x-rtd-version: stable
x-rtd-version-method: path
```

With that data we can call our addons API as follows: `/_/addons/?project__slug=<X-RTD-Project>&version__slug=<X-RTD-Version>`

One thing that I'm not sure how to do is how to get _those headers_ from a request that was done _before_ our `readthedocs-addons.js` file was loaded. If there is no way to do that, we would need to do an extra `HEAD` request to get them:

```javascript
fetch(window.location.href, {method: "HEAD"})
  .then((response) => { 
    const projectSlug = response.headers.get("x-rtd-project"))
    const versionSlug = response.headers.get("x-rtd-version"))
    fetch("/_/addons/?" + new URLSearchParams({ "project__slug": projectSlug, "version__slug": versionSlug})
      .then((response) => {
         // ...
      });
  }
); 
```
Even doing an extra `HEAD` request it will be much better. We can re-search if that possible and manage it as an improvement.

With this approach, we will have two different requests:

1. **Including** `url=` parameter: will be made if there is any addons enable on the page that depends on the URL itself. DocDiff addon is a good example of this. Then, on pages that have DocDiff enabled (pull request previews), our JS will hit `/_/addons/?project__slug=...&version_slug=...` + **`&url=...`**
2. **Omitting** the `url=` parameter: if there aren't any addons enabled on the current page that _requires_ sending the exact URL to the API, the request won't include it: `/_/addons/?project__slug=...&version__slug=...`

With this, our CDN will handle the cache correctly since it will treat the URLs containing only the `project__slug` and `version__slug` as _the same_, no matter what page under the documentation the user is reading. Also, it will treat all the URLs containing `url=` different for all the different pages of the documentation.
Now, with this, we will need to know _before performing the request_ what addons "could be enabled" on the current page to decide whether or not include the `url=`. We will need to ask each addon if they could be enabled, and in that case, if they require sending the `url=` parameter to the backend. Then, **if none of the addons that could be enabled on the page needs to send the `url=` parameter, we don't send it** 👍🏼

I imagine something like:

```javascript
let addons = [ ... ];
let params = {"client-version": "...", "api-version": "..."};
let sendUrlParam = false;
for (const addon of addons) {
  if (addon.requiresURL() && addon.couldBeEnabledOnPage()) {
    sendUrlParam = true;
    break
  }
}

if (sendUrlParam) {
  params.url = window.location.href;
}

...

fetch(url, ....);
```

Then, an addons like DocDiff would implement:

```javascript
export class DocDiffAddon extends AddonBase {

  static requiresUrl() {
    return true;
  }

  static couldBeEnabledOnPage() {
    return window.location.host.endsWith(".readthedocs.build");
  }

}
```

Links to addons code:

* https://github.com/readthedocs/readthedocs-client/blob/ef72cba9a6bb22d125bce73c8a17bbcb15fd7b3d/src/readthedocs-config.js#L7-L30
* https://github.com/readthedocs/readthedocs-client/blob/ef72cba9a6bb22d125bce73c8a17bbcb15fd7b3d/src/index.js#L11-L57
@humitos I don't love the idea of doing an extra HEAD request, because that will increase latency on the rendering of the addons (2 complete HTTP round trips after page load). If we're already injecting data into the page to add the addons JS, then injecting the value of the project & version slug doesn't seem like additional hackiness, and if it saves us that round trip, likely worth doing. 

I don't believe there's a way to capture the headers from the request in our JS, which would be a really nice solution to the problem.

> 1. Including url= parameter: will be made if there is any addons enable on the page that depends on the URL itself. DocDiff addon is a good example of this. Then, on pages that have DocDiff enabled (pull request previews), our JS will hit /_/addons/?project__slug=...&version_slug=... + &url=...
> 2. Omitting the url= parameter: if there aren't any addons enabled on the current page that requires sending the exact URL to the API, the request won't include it: /_/addons/?project__slug=...&version__slug=...

Is this the plan we're going with, assuming that we get the project & version slug *somehow*..., either via a HEAD request, or injected into the page? This logic looks 💯 to me. I think the above logic is great, and then we can try injecting the project/version values in the CF worker, and if that doesn't work, always fall back to a HEAD request.

(Or some other idea we haven't come up with yet 🤔... I think perhaps we could also just parse the URL in JS and only send the hostname and matching version path to the API, but that's harder with the custom URLconf work we've been doing, which would be additional data coming from the server that we need to have in the client, which is the primary problem we're trying to solve here..) 
Thanks @ericholscher for your input here.

Summarizing, this is the current plan that I will implement then:

* inject project/version slug on the HTML using `<meta>` tags via CF worker with the same script we are using to inject the addons js
* grab project/version slug via our addons js and send them to the API
* implement the `Addon.couldBeEnabledOnPage` js function on each addon to decide whether or not send `url=` attribute
* send `url=` attribute if there is at least one addon that has to be enabled on the page

Let me know if you agree with the current plan.

> I think perhaps we could also just parse the URL in JS and only send the hostname and matching version path to the API, but that's harder with the custom URLconf work we've been doing

I thought about this already and I didn't see it clearly since we would have to re-implement the Python resolver logic into JavaScript and always keep it updated with all the changes/feature we introduce on it. I thought that would be fragile and we will end up with bugs on both making them behave differently and hard to debug in the end 🤷🏼 
@humitos that plan sounds 💯. 
> inject project/version slug on the HTML using `<meta>` tags via CF worker with the same script we are using to inject the addons js

This is done and deployed already. See it in action at https://test-builds.readthedocs.io/en/latest/

Resolving patch:
diff --git a/readthedocs/api/v2/permissions.py b/readthedocs/api/v2/permissions.py
--- a/readthedocs/api/v2/permissions.py
+++ b/readthedocs/api/v2/permissions.py
@@ -40,6 +40,17 @@ class IsAuthorizedToViewVersion(permissions.BasePermission):
     def has_permission(self, request, view):
         project = view._get_project()
         version = view._get_version()
+
+        # I had to add this condition here because I want to return a 400 when
+        # the `project-slug` or `version-slug` are not sent to the API
+        # endpoint. In those cases, we don't have a Project/Version and this
+        # function was failing.
+        #
+        # I think it's a valid use case when Project/Version is invalid to be
+        # able to return a good response from the API view.
+        if project is None or version is None:
+            return True
+
         has_access = (
             Version.objects.public(
                 user=request.user,
diff --git a/readthedocs/proxito/views/hosting.py b/readthedocs/proxito/views/hosting.py
--- a/readthedocs/proxito/views/hosting.py
+++ b/readthedocs/proxito/views/hosting.py
@@ -22,7 +22,7 @@
 from readthedocs.core.resolver import resolver
 from readthedocs.core.unresolver import UnresolverError, unresolver
 from readthedocs.core.utils.extend import SettingsOverrideObject
-from readthedocs.projects.models import Feature
+from readthedocs.projects.models import Feature, Project
 
 log = structlog.get_logger(__name__)  # noqa
 
@@ -47,10 +47,19 @@ class BaseReadTheDocsConfigJson(CDNCacheTagsMixin, APIView):
 
     Attributes:
 
-      url (required): absolute URL from where the request is performed
-        (e.g. ``window.location.href``)
-
       api-version (required): API JSON structure version (e.g. ``0``, ``1``, ``2``).
+
+      project-slug (required): slug of the project.
+        Optional if "url" is sent.
+
+      version-slug (required): slug of the version.
+        Optional if "url" is sent.
+
+      url (optional): absolute URL from where the request is performed.
+        When sending "url" attribute, "project-slug" and "version-slug" are ignored.
+        (e.g. ``window.location.href``).
+
+      client-version (optional): JavaScript client version (e.g. ``0.6.0``).
     """
 
     http_method_names = ["get"]
@@ -61,61 +70,73 @@ class BaseReadTheDocsConfigJson(CDNCacheTagsMixin, APIView):
     @lru_cache(maxsize=1)
     def _resolve_resources(self):
         url = self.request.GET.get("url")
-        if not url:
-            # TODO: not sure what to return here when it fails on the `has_permission`
-            return None, None, None, None
+        project_slug = self.request.GET.get("project-slug")
+        version_slug = self.request.GET.get("version-slug")
+
+        project = None
+        version = None
+        build = None
+        filename = None
 
         unresolved_domain = self.request.unresolved_domain
 
         # Main project from the domain.
         project = unresolved_domain.project
 
-        try:
-            unresolved_url = unresolver.unresolve_url(url)
-            # Project from the URL: if it's a subproject it will differ from
-            # the main project got from the domain.
-            project = unresolved_url.project
-            version = unresolved_url.version
-            filename = unresolved_url.filename
-            # This query should use a particular index:
-            # ``builds_build_version_id_state_date_success_12dfb214_idx``.
-            # Otherwise, if the index is not used, the query gets too slow.
-            build = version.builds.filter(
-                success=True,
-                state=BUILD_STATE_FINISHED,
-            ).last()
-
-        except UnresolverError as exc:
-            # If an exception is raised and there is a ``project`` in the
-            # exception, it's a partial match. This could be because of an
-            # invalid URL path, but on a valid project domain. In this case, we
-            # continue with the ``project``, but without a ``version``.
-            # Otherwise, we return 404 NOT FOUND.
-            project = getattr(exc, "project", None)
-            if not project:
-                raise Http404() from exc
-
-            version = None
-            filename = None
-            build = None
+        if url:
+            try:
+                unresolved_url = unresolver.unresolve_url(url)
+                # Project from the URL: if it's a subproject it will differ from
+                # the main project got from the domain.
+                project = unresolved_url.project
+                version = unresolved_url.version
+                filename = unresolved_url.filename
+                # This query should use a particular index:
+                # ``builds_build_version_id_state_date_success_12dfb214_idx``.
+                # Otherwise, if the index is not used, the query gets too slow.
+                build = version.builds.filter(
+                    success=True,
+                    state=BUILD_STATE_FINISHED,
+                ).last()
+
+            except UnresolverError as exc:
+                # If an exception is raised and there is a ``project`` in the
+                # exception, it's a partial match. This could be because of an
+                # invalid URL path, but on a valid project domain. In this case, we
+                # continue with the ``project``, but without a ``version``.
+                # Otherwise, we return 404 NOT FOUND.
+                project = getattr(exc, "project", None)
+                if not project:
+                    raise Http404() from exc
+
+        else:
+            project = Project.objects.filter(slug=project_slug).first()
+            version = Version.objects.filter(slug=version_slug, project=project).first()
+            if version:
+                build = version.builds.last()
 
         return project, version, build, filename
 
     def _get_project(self):
-        project, version, build, filename = self._resolve_resources()
+        project, _, _, _ = self._resolve_resources()
         return project
 
     def _get_version(self):
-        project, version, build, filename = self._resolve_resources()
+        _, version, _, _ = self._resolve_resources()
         return version
 
     def get(self, request, format=None):
         url = request.GET.get("url")
+        project_slug = request.GET.get("project-slug")
+        version_slug = request.GET.get("version-slug")
         if not url:
-            return JsonResponse(
-                {"error": "'url' GET attribute is required"},
-                status=400,
-            )
+            if not project_slug or not version_slug:
+                return JsonResponse(
+                    {
+                        "error": "'project-slug' and 'version-slug' GET attributes are required when not sending 'url'"
+                    },
+                    status=400,
+                )
 
         addons_version = request.GET.get("api-version")
         if not addons_version:
@@ -148,6 +169,7 @@ def get(self, request, format=None):
             version,
             build,
             filename,
+            url,
             user=request.user,
         )
         return JsonResponse(data, json_dumps_params={"indent": 4, "sort_keys": True})
@@ -199,6 +221,7 @@ def get(
         version=None,
         build=None,
         filename=None,
+        url=None,
         user=None,
     ):
         """
@@ -208,12 +231,12 @@ def get(
         best JSON structure for that particular version.
         """
         if addons_version.major == 0:
-            return self._v0(project, version, build, filename, user)
+            return self._v0(project, version, build, filename, url, user)
 
         if addons_version.major == 1:
-            return self._v1(project, version, build, filename, user)
+            return self._v1(project, version, build, filename, url, user)
 
-    def _v0(self, project, version, build, filename, user):
+    def _v0(self, project, version, build, filename, url, user):
         """
         Initial JSON data structure consumed by the JavaScript client.
 
@@ -308,27 +331,6 @@ def _v0(self, project, version, build, filename, user):
                         versions_active_built_not_hidden.values_list("slug", flat=True)
                     ),
                 },
-                "doc_diff": {
-                    "enabled": Feature.ADDONS_DOC_DIFF_DISABLED not in project_features,
-                    # "http://test-builds-local.devthedocs.org/en/latest/index.html"
-                    "base_url": resolver.resolve(
-                        project=project,
-                        # NOTE: we are using LATEST version to compare against to for now.
-                        # Ideally, this should be configurable by the user.
-                        version_slug=LATEST,
-                        language=project.language,
-                        filename=filename,
-                    )
-                    if filename
-                    else None,
-                    "root_selector": "[role=main]",
-                    "inject_styles": True,
-                    # NOTE: `base_host` and `base_page` are not required, since
-                    # we are constructing the `base_url` in the backend instead
-                    # of the frontend, as the doc-diff extension does.
-                    "base_host": "",
-                    "base_page": "",
-                },
                 "flyout": {
                     "enabled": Feature.ADDONS_FLYOUT_DISABLED not in project_features,
                     "translations": [
@@ -347,22 +349,22 @@ def _v0(self, project, version, build, filename, user):
                     "versions": [
                         {
                             # TODO: name this field "display_name"
-                            "slug": version.slug,
+                            "slug": version_.slug,
                             "url": resolver.resolve(
                                 project=project,
-                                version_slug=version.slug,
-                                external=version.type == EXTERNAL,
+                                version_slug=version_.slug,
+                                external=version_.type == EXTERNAL,
                             ),
                         }
-                        for version in versions_active_built_not_hidden
+                        for version_ in versions_active_built_not_hidden
                     ],
                     "downloads": [
                         {
                             # TODO: name this field "display_name"
                             "name": name,
-                            "url": url,
+                            "url": url_,
                         }
-                        for name, url in version_downloads
+                        for name, url_ in version_downloads
                     ],
                     # TODO: find a way to get this data in a reliably way.
                     # We don't have a simple way to map a URL to a file in the repository.
@@ -417,6 +419,38 @@ def _v0(self, project, version, build, filename, user):
             },
         }
 
+        # DocDiff depends on `url=` GET attribute.
+        # This attribute allows us to know the exact filename where the request was made.
+        # If we don't know the filename, we cannot return the data required by DocDiff to work.
+        # In that case, we just don't include the `doc_diff` object in the response.
+        if url:
+            data["addons"].update(
+                {
+                    "doc_diff": {
+                        "enabled": Feature.ADDONS_DOC_DIFF_DISABLED
+                        not in project_features,
+                        # "http://test-builds-local.devthedocs.org/en/latest/index.html"
+                        "base_url": resolver.resolve(
+                            project=project,
+                            # NOTE: we are using LATEST version to compare against to for now.
+                            # Ideally, this should be configurable by the user.
+                            version_slug=LATEST,
+                            language=project.language,
+                            filename=filename,
+                        )
+                        if filename
+                        else None,
+                        "root_selector": "[role=main]",
+                        "inject_styles": True,
+                        # NOTE: `base_host` and `base_page` are not required, since
+                        # we are constructing the `base_url` in the backend instead
+                        # of the frontend, as the doc-diff extension does.
+                        "base_host": "",
+                        "base_page": "",
+                    },
+                }
+            )
+
         # Update this data with the one generated at build time by the doctool
         if version and version.build_data:
             data.update(version.build_data)
@@ -450,7 +484,7 @@ def _v0(self, project, version, build, filename, user):
 
         return data
 
-    def _v1(self, project, version, build, filename, user):
+    def _v1(self, project, version, build, filename, url, user):
         return {
             "api_version": "1",
             "comment": "Undefined yet. Use v0 for now",


Test patch:
diff --git a/readthedocs/proxito/tests/test_hosting.py b/readthedocs/proxito/tests/test_hosting.py
--- a/readthedocs/proxito/tests/test_hosting.py
+++ b/readthedocs/proxito/tests/test_hosting.py
@@ -498,3 +498,81 @@ def test_flyout_subproject_urls(self):
             },
         ]
         assert r.json()["addons"]["flyout"]["translations"] == expected_translations
+
+    def test_send_project_not_version_slugs(self):
+        r = self.client.get(
+            reverse("proxito_readthedocs_docs_addons"),
+            {
+                "api-version": "0.1.0",
+                "client-version": "0.6.0",
+                "project-slug": self.project.slug,
+            },
+            secure=True,
+            headers={
+                "host": "project.dev.readthedocs.io",
+            },
+        )
+        assert r.status_code == 400
+        assert r.json() == {
+            "error": "'project-slug' and 'version-slug' GET attributes are required when not sending 'url'"
+        }
+
+    def test_send_version_not_project_slugs(self):
+        r = self.client.get(
+            reverse("proxito_readthedocs_docs_addons"),
+            {
+                "api-version": "0.1.0",
+                "client-version": "0.6.0",
+                "version-slug": self.version.slug,
+            },
+            secure=True,
+            headers={
+                "host": "project.dev.readthedocs.io",
+            },
+        )
+        assert r.status_code == 400
+        assert r.json() == {
+            "error": "'project-slug' and 'version-slug' GET attributes are required when not sending 'url'"
+        }
+
+    def test_send_project_version_slugs(self):
+        r = self.client.get(
+            reverse("proxito_readthedocs_docs_addons"),
+            {
+                "api-version": "0.1.0",
+                "client-version": "0.6.0",
+                "project-slug": self.project.slug,
+                "version-slug": self.version.slug,
+            },
+            secure=True,
+            headers={
+                "host": "project.dev.readthedocs.io",
+            },
+        )
+        assert r.status_code == 200
+        expected_response = self._get_response_dict("v0")
+        # Remove `addons.doc_diff` from the response because it's not present when `url=` is not sent
+        expected_response["addons"].pop("doc_diff")
+
+        assert self._normalize_datetime_fields(r.json()) == expected_response
+
+    def test_send_project_version_slugs_and_url(self):
+        r = self.client.get(
+            reverse("proxito_readthedocs_docs_addons"),
+            {
+                "api-version": "0.1.0",
+                "client-version": "0.6.0",
+                "url": "https://project.dev.readthedocs.io/en/latest/",
+                # When sending `url=`, slugs are ignored
+                "project-slug": "different-project-slug-than-url",
+                "version-slug": "different-version-slug-than-url",
+            },
+            secure=True,
+            headers={
+                "host": "project.dev.readthedocs.io",
+            },
+        )
+        assert r.status_code == 200
+        assert self._normalize_datetime_fields(r.json()) == self._get_response_dict(
+            "v0"
+        )