Repository: WeblateOrg/weblate

Issue:
Missing index on trans_unit.pending
### Describe the issue

We are experiencing performance issues, bottleneck is at disk level caused by postgres SELECT queries.

For instance the following query takes ~30 seconds:
```
SELECT COUNT(*) AS "__count" FROM "trans_unit"
INNER JOIN "trans_translation" ON ("trans_unit"."translation_id" =
"trans_translation"."id") WHERE ("trans_unit"."pending" AND
"trans_translation"."component_id" = 136)
```

There are ~15M entries in trans_unit.

We have created an index and the query now takes few ms

```
CREATE INDEX CONCURRENTLY trans_unit_pending ON trans_unit(pending)
WHERE pending;
```

### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar filed issues in this repository.

### Steps to reproduce the behavior

Have a lot of entries in trans_unit and run the previous query.

### Expected behavior

_No response_

### Screenshots

_No response_

### Exception traceback

_No response_

### How do you run Weblate?

Docker container

### Weblate versions

_No response_

### Weblate deploy checks

_No response_

### Additional context

_No response_


Discussion:
Index on that field was there, but it generally did not perform that well, so it was changed in aed8cf67cceee210640fbf8ba28520da174c6291 and removed in ab7f2945fc9417a58699ddb6077b7c5ab40062ed. Using condition in WHERE might address the issues we had with the full table index on the field, though.

https://github.com/WeblateOrg/weblate/issues/8770 should also be considered when touching this.

Resolving patch:
diff --git a/weblate/trans/migrations/0010_unit_trans_unit_pending.py b/weblate/trans/migrations/0010_unit_trans_unit_pending.py
new file mode 100644
--- /dev/null
+++ b/weblate/trans/migrations/0010_unit_trans_unit_pending.py
@@ -0,0 +1,25 @@
+# Copyright © Michal Čihař <michal@weblate.org>
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+# Generated by Django 4.2.8 on 2024-01-05 12:04
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("trans", "0009_alter_change_action"),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name="unit",
+            index=models.Index(
+                models.F("translation"),
+                models.F("pending"),
+                condition=models.Q(("pending", True)),
+                name="trans_unit_pending",
+            ),
+        ),
+    ]
diff --git a/weblate/trans/models/unit.py b/weblate/trans/models/unit.py
--- a/weblate/trans/models/unit.py
+++ b/weblate/trans/models/unit.py
@@ -404,6 +404,16 @@ class Meta:
             models.Index(
                 MD5(Lower("context")), "translation", name="trans_unit_context_md5"
             ),
+            # Partial index for pending field to optimize lookup in translation
+            # commit pending method. Full table index performs poorly here because
+            # it becomes huge.
+            # MySQL/MariaDB does not supports condition and uses full index instead.
+            models.Index(
+                "translation",
+                "pending",
+                condition=Q(pending=True),
+                name="trans_unit_pending",
+            ),
         ]
 
     def __str__(self):


Test patch: