Repository: readthedocs/readthedocs.org

Issue:
Remove addons response for last build
We noticed a big spike in time spent in build table queries today and
traced the query back to the addons response. The addons data structure
includes a list of the latest builds for each version.

I've disabled that for now, but it's not yet clear why this became an
issue on Fri, or why the cache doesn't seem to be helping much here.

cc @humitos in case you have some more input here
Addons: review and reenable build listing
See https://github.com/readthedocs/readthedocs.org/pull/10831 for some background on this issue.

We found a large increase in build select query time and traced it back to the usage of `project.version.builds.last()`. This is a rather expensive call, so it was disabled. We will need to return to this to figure out if we need it and how to give this data in a performant way if we do.

A few leftovers from my hotfix:

- [ ] Reenable external version warning notification addon in admin
- [ ] Release a addons library version with defensive measures around the missing data
- [ ] Revert some of the hotfix?


Discussion:
Noted in chat, but I don't exactly know the best way to test this change outside relying on out test suite. I might opt to apply this early or it will take a while to get this out.
Actually, this is not quite the fix :disappointed: 

In the addons library we have this part of the data structure hardcoded in our template:

https://github.com/readthedocs/addons/blob/main/src/notification.js#L54

That means that we can't return `None` here, as the addon library html template string interpolation will throw and exception when it can't find the `config.builds.current.id` attribute.
I'm going to proceed with this hotfix to make sure the database stops complaining. Exceptions from the addon library will be limited to projects using PR builds, as this is the only place this URL is shown. It might be worth disabling the notification addon until we add defensive measures to the configuration access and template rendering to it.
I will release this hotfix just to community for now. Currently, the addons data average response time is still over a few seconds long:

![image](https://github.com/readthedocs/readthedocs.org/assets/1140183/883224ec-75f1-4f17-a73c-533d37ed60ca)

Also note: I disabled the external version warning via the feature flag. We'll need to re-enable that later.

And after hotfix: 
![image](https://github.com/readthedocs/readthedocs.org/assets/1140183/134eecd9-7a31-47e0-b474-5735245f3847)

Seems to have stabilized well, and I haven't noticed any console exceptions yet.
I'm not sure what's the best solution here. Ideally, we should have the `Build` object in the response. We should think about how to make this query to be reliable and fast. I don't really understand why it's slow actually since we have a DB index on `Build.date` and getting the last one should be pretty fast.

```
docs=> \d builds_build

...

Indexes:
    "builds_build_pkey" PRIMARY KEY, btree (id)
    "Build_State" btree (state)
    "builds_buil_project_fea68c_idx" btree (project_id, date)
    "builds_build_date_id_9f6984cf_idx" btree (date, id)
    "builds_build_project_id" btree (project_id)
    "builds_build_version_id_cd1a89966ebc2f3" btree (version_id, state, type)
    "date_index" btree (date)
...

docs=>
```
0.002376s seems to be _pretty_ fast.

```
In [2]: p.builds.last()
SELECT "builds_build"."id",
       "builds_build"."project_id",
       "builds_build"."version_id",
       "builds_build"."type",
       "builds_build"."state",
       "builds_build"."status",
       "builds_build"."date",
       "builds_build"."success",
       "builds_build"."setup",
       "builds_build"."setup_error",
       "builds_build"."output",
       "builds_build"."error",
       "builds_build"."exit_code",
       "builds_build"."commit",
       "builds_build"."version_slug",
       "builds_build"."version_name",
       "builds_build"."version_type",
       "builds_build"."_config",
       "builds_build"."readthedocs_yaml_path",
       "builds_build"."length",
       "builds_build"."builder",
       "builds_build"."cold_storage",
       "builds_build"."task_id"
  FROM "builds_build"
 WHERE "builds_build"."project_id" = 822831
 ORDER BY "builds_build"."date" ASC
 LIMIT 1

Execution time: 0.002376s [Database: default]
Out[2]: SELECT "auth_user"."username"
  FROM "auth_user"
 INNER JOIN "projects_project_users"
    ON ("auth_user"."id" = "projects_project_users"."user_id")
 WHERE "projects_project_users"."project_id" = 822831

Execution time: 0.001484s [Database: default]
<Build: Build langchain for hwchase17 dev2049 WFH eugene-lc efriislangchain (18416334)>

In [3]:
```
I ran this query with `EXPLAIN ANALYZE` and it seems it uses the index `builds_buil_project_fea68c_idx` which is an index for `project_id, date` together.

The query is really fast, `Execution Time: 0.061 ms`. I think the problem should be something different, then. Is it possible this query to get slow when a new `Build` object is created but the index is not yet updated? 🤷🏼 

```
docs=> EXPLAIN ANALYZE SELECT "builds_build"."id",
       "builds_build"."project_id",
       "builds_build"."version_id",
       "builds_build"."type",
       "builds_build"."state",
       "builds_build"."status",
       "builds_build"."date",
       "builds_build"."success",
       "builds_build"."setup",
       "builds_build"."setup_error",
       "builds_build"."output",
       "builds_build"."error",
       "builds_build"."exit_code",
       "builds_build"."commit",
       "builds_build"."version_slug",
       "builds_build"."version_name",
       "builds_build"."version_type",
       "builds_build"."_config",
       "builds_build"."readthedocs_yaml_path",
       "builds_build"."length",
       "builds_build"."builder",
       "builds_build"."cold_storage",
       "builds_build"."task_id"
  FROM "builds_build"
 WHERE "builds_build"."project_id" = 822831
 ORDER BY "builds_build"."date" ASC
 LIMIT 1;
                                                                        QUERY PLAN                                                                        
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.56..4.12 rows=1 width=322) (actual time=0.022..0.023 rows=1 loops=1)
   ->  Index Scan using builds_buil_project_fea68c_idx on builds_build  (cost=0.56..1927.76 rows=542 width=322) (actual time=0.022..0.022 rows=1 loops=1)
         Index Cond: (project_id = 822831)
 Planning Time: 0.586 ms
 Execution Time: 0.061 ms
(5 rows)

docs=>
```
I realized that in my previous queries I was taking the latest build of the project, but not for a particular version. Here is the query updated:

```
docs=> EXPLAIN ANALYZE SELECT "builds_build"."id",
       "builds_build"."project_id",
       "builds_build"."version_id",
       "builds_build"."type",
       "builds_build"."state",
       "builds_build"."status",
       "builds_build"."date",
       "builds_build"."success",
       "builds_build"."setup",
       "builds_build"."setup_error",
       "builds_build"."output",
       "builds_build"."error",
       "builds_build"."exit_code",
       "builds_build"."commit",
       "builds_build"."version_slug",
       "builds_build"."version_name",
       "builds_build"."version_type",
       "builds_build"."_config",
       "builds_build"."readthedocs_yaml_path",
       "builds_build"."length",
       "builds_build"."builder",
       "builds_build"."cold_storage",
       "builds_build"."task_id"
  FROM "builds_build"
 WHERE "builds_build"."version_id" = 4740677
 ORDER BY "builds_build"."date" ASC
 LIMIT 1;
                                                                               QUERY PLAN                                                                                
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1053.54..1053.54 rows=1 width=322) (actual time=0.046..0.046 rows=1 loops=1)
   ->  Sort  (cost=1053.54..1054.33 rows=318 width=322) (actual time=0.045..0.046 rows=1 loops=1)
         Sort Key: date
         Sort Method: quicksort  Memory: 26kB
         ->  Index Scan using builds_build_version_id_cd1a89966ebc2f3 on builds_build  (cost=0.56..1051.95 rows=318 width=322) (actual time=0.017..0.020 rows=2 loops=1)
               Index Cond: (version_id = 4740677)
 Planning Time: 0.087 ms
 Execution Time: 0.075 ms
(8 rows)
```

I uses the index `builds_build_version_id_cd1a89966ebc2f3` which is based on `(version_id, state, type)` but not on `date`.

I'd propose to create an index for this based on `(version_id, state, date, success)` (based on https://github.com/readthedocs/readthedocs.org/pull/10831#discussion_r1363699484)  this will make these queries to be pretty fast, IMO


Resolving patch:
diff --git a/readthedocs/builds/migrations/0054_add_builds_index_for_addons.py b/readthedocs/builds/migrations/0054_add_builds_index_for_addons.py
new file mode 100644
--- /dev/null
+++ b/readthedocs/builds/migrations/0054_add_builds_index_for_addons.py
@@ -0,0 +1,20 @@
+# Generated by Django 4.2.6 on 2023-10-18 11:43
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("builds", "0053_alter_version_build_data"),
+    ]
+
+    operations = [
+        migrations.AlterIndexTogether(
+            name="build",
+            index_together={
+                ("date", "id"),
+                ("version", "state", "type"),
+                ("version", "state", "date", "success"),
+            },
+        ),
+    ]
diff --git a/readthedocs/builds/models.py b/readthedocs/builds/models.py
--- a/readthedocs/builds/models.py
+++ b/readthedocs/builds/models.py
@@ -847,8 +847,11 @@ class Meta:
         ordering = ['-date']
         get_latest_by = 'date'
         index_together = [
-            ['version', 'state', 'type'],
-            ['date', 'id'],
+            # Useful for `/_/addons/` API endpoint.
+            # Query: ``version.builds.filter(success=True, state=BUILD_STATE_FINISHED)``
+            ["version", "state", "date", "success"],
+            ["version", "state", "type"],
+            ["date", "id"],
         ]
         indexes = [
             models.Index(fields=['project', 'date']),
diff --git a/readthedocs/proxito/views/hosting.py b/readthedocs/proxito/views/hosting.py
--- a/readthedocs/proxito/views/hosting.py
+++ b/readthedocs/proxito/views/hosting.py
@@ -17,7 +17,7 @@
     ProjectSerializer,
     VersionSerializer,
 )
-from readthedocs.builds.constants import EXTERNAL, LATEST
+from readthedocs.builds.constants import BUILD_STATE_FINISHED, EXTERNAL, LATEST
 from readthedocs.builds.models import Version
 from readthedocs.core.resolver import resolver
 from readthedocs.core.unresolver import UnresolverError, unresolver
@@ -77,7 +77,13 @@ def _resolve_resources(self):
             project = unresolved_url.project
             version = unresolved_url.version
             filename = unresolved_url.filename
-            build = version.builds.last()
+            # This query should use a particular index:
+            # ``builds_build_version_id_state_date_success_12dfb214_idx``.
+            # Otherwise, if the index is not used, the query gets too slow.
+            build = version.builds.filter(
+                success=True,
+                state=BUILD_STATE_FINISHED,
+            ).last()
 
         except UnresolverError as exc:
             # If an exception is raised and there is a ``project`` in the


Test patch: