Repository: WeblateOrg/weblate

Issue:
Potential O(N^2) problem in spawning celery cleanup_project tasks
### Describe the issue

I have a project with a lot of components, around 5 thousand. After doing some maintenance on the git repo, changing/deleting many translation units from many projects, I noticed that celery is severely backlogged.

It looks like when a component changes sufficiently (e.g. lines are deleted), here https://github.com/WeblateOrg/weblate/blob/b847ad3b9f5c08b6d0e7005d2421ee06ea228109/weblate/trans/models/component.py#L2312 it spawns a cleanup_project() task.

However, cleanup_project here https://github.com/WeblateOrg/weblate/blob/b847ad3b9f5c08b6d0e7005d2421ee06ea228109/weblate/trans/tasks.py#L153 does pretty much what it says on the tin i.e. it goes through every component in the project and cleans it up.

So it seems this is now a O(N^2) problem where I have 5k spawned tasks, and every task goes through 5k components (even tho it only needs to do that once?), and there could probably be severe lock contention as well?

In any case I'm wondering if it's possible to change the task to cleanup a specific component, or somehow make it not spawn duplicate tasks when a lot of components get changed.

### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.

### Steps to reproduce the behavior

_No response_

### Expected behavior

_No response_

### Screenshots

_No response_

### Exception traceback

_No response_

### How do you run Weblate?

Git checkout

### Weblate versions

_No response_

### Weblate deploy checks

_No response_

### Additional context

_No response_


Discussion:
It used to be really project scoped before (4c6e9c80a25e114eb988dead8695f37a2107bf13), but these days it is merely a component scoped. I think it should be actually changed to component scope.

Resolving patch:
diff --git a/weblate/trans/management/commands/cleanuptrans.py b/weblate/trans/management/commands/cleanuptrans.py
--- a/weblate/trans/management/commands/cleanuptrans.py
+++ b/weblate/trans/management/commands/cleanuptrans.py
@@ -6,11 +6,11 @@
 
 from weblate.accounts.tasks import cleanup_social_auth
 from weblate.screenshots.tasks import cleanup_screenshot_files
-from weblate.trans.models import Project
+from weblate.trans.models import Component
 from weblate.trans.tasks import (
+    cleanup_component,
     cleanup_old_comments,
     cleanup_old_suggestions,
-    cleanup_project,
     cleanup_stale_repos,
     cleanup_suggestions,
 )
@@ -25,8 +25,8 @@ def handle(self, *args, **options):
         cleanup_screenshot_files()
         with transaction.atomic():
             cleanup_social_auth()
-        for project in Project.objects.values_list("id", flat=True):
-            cleanup_project(project)
+        for component in Component.objects.filter(template="").iterator():
+            cleanup_component(component.pk)
         cleanup_suggestions()
         cleanup_stale_repos()
         cleanup_old_suggestions()
diff --git a/weblate/trans/models/component.py b/weblate/trans/models/component.py
--- a/weblate/trans/models/component.py
+++ b/weblate/trans/models/component.py
@@ -2306,10 +2306,10 @@ def _create_translations(  # noqa: C901
             self.invalidate_cache()
 
         # Schedule background cleanup if needed
-        if self.needs_cleanup:
-            from weblate.trans.tasks import cleanup_project
+        if self.needs_cleanup and not self.template:
+            from weblate.trans.tasks import cleanup_component
 
-            transaction.on_commit(lambda: cleanup_project.delay(self.project_id))
+            transaction.on_commit(lambda: cleanup_component.delay(self.id))
 
         # Send notifications on new string
         for translation in translations.values():
diff --git a/weblate/trans/tasks.py b/weblate/trans/tasks.py
--- a/weblate/trans/tasks.py
+++ b/weblate/trans/tasks.py
@@ -150,36 +150,40 @@ def commit_pending(hours=None, pks=None, logger=None):
 
 
 @app.task(trail=False)
-def cleanup_project(pk):
+def cleanup_component(pk):
     """
-    Perform cleanup of project models.
+    Perform cleanup of component models.
 
     - Remove stale source Unit objects.
     - Update variants.
     """
     try:
-        project = Project.objects.get(pk=pk)
-    except Project.DoesNotExist:
+        component = Component.objects.get(pk=pk)
+    except Component.DoesNotExist:
         return
 
-    for component in project.component_set.filter(template="").iterator():
-        # Remove stale variants
-        with transaction.atomic():
-            component.update_variants()
+    # Skip monolingual components, these handle cleanups based on the template
+    if component.template:
+        return
 
-        translation = component.source_translation
-        # Skip translations with a filename (eg. when POT file is present)
-        if translation.filename:
-            continue
-        with transaction.atomic():
-            # Remove all units where there is just one referenced unit (self)
-            deleted, details = (
-                translation.unit_set.annotate(Count("unit"))
-                .filter(unit__count__lte=1)
-                .delete()
-            )
-            if deleted:
-                translation.log_info("removed leaf units: %s", details)
+    # Remove stale variants
+    with transaction.atomic():
+        component.update_variants()
+
+    translation = component.source_translation
+    # Skip translations with a filename (eg. when POT file is present)
+    if translation.filename:
+        return
+
+    # Remove all units where there is just one referenced unit (self)
+    with transaction.atomic():
+        deleted, details = (
+            translation.unit_set.annotate(Count("unit"))
+            .filter(unit__count__lte=1)
+            .delete()
+        )
+        if deleted:
+            translation.log_info("removed leaf units: %s", details)
 
 
 @app.task(trail=False)


Test patch: