Repository: WeblateOrg/weblate
Pull request #14531: feat: backup categories
Author: login: amCap1712
Created: 2025-04-11T09:46:35Z | Merged: 2025-04-16T15:12:32Z

Include categories in project backups.

Fixes #13743.

Depends on https://github.com/WeblateOrg/weblate_schemas/pull/426.

Linked issue #13743: Backup categories
### Describe the problem

It turns out that while implementing categories, they were omitted from project backups.

### Describe the solution you would like

Include categories in the project backups.

1. Extend backup schema in https://github.com/WeblateOrg/weblate_schemas/ to cover categories.
2. Add support for backing-up categories.
3. Add support for restoring categories when they are included in the backup.
4. Add test for categories included in the backup.

### Describe alternatives you have considered

_No response_

### Screenshots

_No response_

### Additional context

_No response_
Linked issue comments: user: login: github-actions[bot]
created_at: 2025-04-16T15:12:50Z
body: Thank you for your report; the issue you have reported has just been fixed.

* In case you see a problem with the fix, please comment on this issue.
* In case you see a similar problem, please open a separate issue.
* If you are happy with the outcome, don’t hesitate to support Weblate by making a [donation](https://weblate.org/donate/).
author_association: NONE

Diff:
diff --git a/weblate/trans/backups.py b/weblate/trans/backups.py
index ba022d52ad9b..2db826c18416 100644
--- a/weblate/trans/backups.py
+++ b/weblate/trans/backups.py
@@ -30,6 +30,7 @@
 from weblate.memory.models import Memory
 from weblate.screenshots.models import Screenshot
 from weblate.trans.models import (
+    Category,
     Comment,
     Component,
     Label,
@@ -77,6 +78,13 @@ def __init__(
         self.labels_map: dict[str, Label] = {}
         self.user_cache: dict[str, User] = {}
         self.components_cache: dict[str, Component] = {}
+        self.categories_cache: dict[str, Category] = {}
+
+    @staticmethod
+    def full_slug_without_project(obj: Component | Category) -> str:
+        """Return the full slug for a component or category without the project slug."""
+        parts = obj.get_url_path()
+        return "/".join(parts[1:])
 
     @property
     def supports_restore(self):
@@ -115,6 +123,20 @@ def backup_object(
     ):
         return {field: self.backup_property(obj, field, extras) for field in properties}
 
+    def backup_categories(self, obj: Project | Category) -> list[dict]:
+        if isinstance(obj, Project):
+            categories = obj.category_set.filter(category=None)
+        else:
+            categories = obj.category_set.all()
+        return [
+            self.backup_object(
+                category,
+                self.project_schema["definitions"]["category"]["required"],
+                {"categories": self.backup_categories},
+            )
+            for category in categories
+        ]
+
     def backup_data(self, project) -> None:
         self.project = project
         self.data = {
@@ -131,6 +153,7 @@ def backup_data(self, project) -> None:
                 {"name": label.name, "color": label.color}
                 for label in project.label_set.all()
             ],
+            "categories": self.backup_categories(project),
         }
 
         # Make sure generated backup data is correct
@@ -248,6 +271,11 @@ def backup_component(self, backupzip, component) -> None:
                 ).iterator()
             ],
         }
+        # component category is not a required field
+        if component.category:
+            data["component"]["category"] = self.full_slug_without_project(
+                component.category
+            )
 
         data["screenshots"] = screenshots = []
         for screenshot in Screenshot.objects.filter(
@@ -289,7 +317,7 @@ def backup_component(self, backupzip, component) -> None:
         self.backup_dir(
             backupzip,
             component.full_path,
-            f"{self.VCS_PREFIX}{component.slug}",
+            f"{self.VCS_PREFIX}{self.full_slug_without_project(component)}",
         )
 
     @transaction.atomic
@@ -451,9 +479,12 @@ def restore_component(self, zipfile, data) -> None:  # noqa: C901
             # Update linked_component attribute
             if kwargs["repo"].startswith(new_slug):
                 kwargs["linked_component"] = self.components_cache[
-                    kwargs["repo"].removeprefix("weblate://")
+                    kwargs["repo"].removeprefix(new_slug)
                 ]
 
+        if "category" in kwargs:
+            kwargs["category"] = self.categories_cache[kwargs["category"]]
+
         component = Component(project=self.project, **kwargs)
         # Trigger pre_save to update git export URL
         pre_save.send(
@@ -595,7 +626,7 @@ def restore_component(self, zipfile, data) -> None:  # noqa: C901
         component.schedule_update_checks()
 
         # Update cache
-        self.components_cache[component.full_slug] = component
+        self.components_cache[self.full_slug_without_project(component)] = component
 
     def import_language(self, code: str):
         if not self.languages_cache:
@@ -608,6 +639,23 @@ def import_language(self, code: str):
             )
             return language
 
+    def restore_categories(
+        self, categories: list[dict], parent_category: Category | None = None
+    ):
+        category_objs = [
+            Category(
+                name=category["name"],
+                slug=category["slug"],
+                category=parent_category,
+                project=self.project,
+            )
+            for category in categories
+        ]
+        category_objs = Category.objects.bulk_create(category_objs)
+        for category, obj in zip(categories, category_objs, strict=False):
+            self.categories_cache[self.full_slug_without_project(obj)] = obj
+            self.restore_categories(category["categories"], obj)
+
     @transaction.atomic
     def restore(self, project_name: str, project_slug: str, user: User, billing=None):
         if not isinstance(self.filename, str):
@@ -630,6 +678,8 @@ def restore(self, project_name: str, project_slug: str, user: User, billing=None
                 Label(project=project, **entry) for entry in self.data["labels"]
             )
             self.labels_map = {label.name: label for label in labels}
+            if "categories" in self.data:
+                self.restore_categories(self.data["categories"], None)
 
             # Import translation memory
             memory = self.load_memory(zipfile)
diff --git a/weblate/trans/tests/test_backups.py b/weblate/trans/tests/test_backups.py
index 8f787f4b1f83..933c78b81765 100644
--- a/weblate/trans/tests/test_backups.py
+++ b/weblate/trans/tests/test_backups.py
@@ -19,7 +19,7 @@
 from weblate.checks.models import Check
 from weblate.screenshots.models import Screenshot
 from weblate.trans.backups import ProjectBackup
-from weblate.trans.models import Comment, Project, Suggestion, Unit, Vote
+from weblate.trans.models import Category, Comment, Project, Suggestion, Unit, Vote
 from weblate.trans.tasks import cleanup_project_backup_download, cleanup_project_backups
 from weblate.trans.tests.test_views import ViewTestCase
 from weblate.trans.tests.utils import get_test_file
@@ -36,6 +36,11 @@ def test_create_backup(self) -> None:
         # Create linked component
         self.create_link_existing()
         # Additional content to test on backups
+        category = self.project.category_set.create(
+            name="My Category", slug="my-category"
+        )
+        self.component.category = category
+        self.component.save()
         label = self.project.label_set.create(name="Label", color="navy")
         unit = self.component.source_translation.unit_set.all()[0]
         unit.labels.add(label)
@@ -66,7 +71,7 @@ def test_create_backup(self) -> None:
             self.assertIn("weblate-backup.json", files)
             self.assertIn("components/test.json", files)
             self.assertIn("components/glossary.json", files)
-            self.assertIn("vcs/test/.git/index", files)
+            self.assertIn("vcs/my-category/test/.git/index", files)
             self.assertIn("vcs/glossary/.git/index", files)
 
         restore = ProjectBackup(backup.filename)
@@ -122,6 +127,26 @@ def test_create_backup(self) -> None:
             set(self.project.component_set.values_list("slug", flat=True)),
             set(restored.component_set.values_list("slug", flat=True)),
         )
+        self.assertEqual(
+            Category.objects.filter(project=self.project).count(),
+            Category.objects.filter(project=restored).count(),
+        )
+        self.assertEqual(
+            set(self.project.category_set.values_list("slug", flat=True)),
+            set(restored.category_set.values_list("slug", flat=True)),
+        )
+
+        def component_category_mapping(p):
+            """Return a mapping of component slugs to component category slugs."""
+            return {
+                co.slug: co.category.slug if co.category else None
+                for co in p.component_set.all()
+            }
+
+        self.assertEqual(
+            component_category_mapping(self.project),
+            component_category_mapping(restored),
+        )
         # Verify that Git operations work on restored repos
         restored.do_reset()
 


Reviews:
user: login: amCap1712
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T09:47:17Z
user: login: nijel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T13:30:02Z
user: login: nijel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T13:37:27Z
user: login: amCap1712
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:22:13Z
user: login: amCap1712
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T12:26:54Z
user: login: nijel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T12:52:10Z

Review comments:
diff_hunk: @@ -163,7 +163,7 @@ dependencies = [
   "unidecode>=1.3.8,<1.4",
   "user-agents>=2.0,<2.3",
   "weblate-language-data>=2025.2",
-  "weblate-schemas==2024.2"
+  "weblate_schemas @ git+https://github.com/amCap1712/weblate_schemas.git@categories"
path: pyproject.toml
user: login: amCap1712
body: To allow for testing the PR, until a new release is made including the schema changes.
created_at: 2025-04-11T09:47:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,7 +163,7 @@ dependencies = [
   "unidecode>=1.3.8,<1.4",
   "user-agents>=2.0,<2.3",
   "weblate-language-data>=2025.2",
-  "weblate-schemas==2024.2"
+  "weblate_schemas @ git+https://github.com/amCap1712/weblate_schemas.git@categories"
path: pyproject.toml
user: login: nijel
body: That's fine. I'd like to release it once we have a schema for https://github.com/WeblateOrg/weblate/issues/13744 ready as well.
created_at: 2025-04-11T13:30:02Z
author_association: MEMBER
diff_hunk: @@ -608,6 +639,23 @@ def import_language(self, code: str):
             )
             return language
 
+    def restore_categories(
+        self, categories: list[dict], parent_category: Category = None
+    ):
+        for category in categories:
+            category_obj = Category(
+                name=category["name"],
+                slug=category["slug"],
+                category=parent_category,
+                project=self.project,
+            )
+            category_obj = Category.objects.bulk_create([category_obj])[0]
path: weblate/trans/backups.py
user: login: nijel
body: Why not bulk create all of the categories at once in one `bulk_create`?
created_at: 2025-04-11T13:35:39Z
author_association: MEMBER
diff_hunk: @@ -630,6 +678,8 @@ def restore(self, project_name: str, project_slug: str, user: User, billing=None
                 Label(project=project, **entry) for entry in self.data["labels"]
             )
             self.labels_map = {label.name: label for label in labels}
+            if "categories" in self.data:
+                self.restore_categories(self.data["categories"], None)
path: weblate/trans/backups.py
user: login: nijel
body: The type annotation doesn't allow `None` here.
created_at: 2025-04-11T13:36:07Z
author_association: MEMBER
diff_hunk: @@ -630,6 +678,8 @@ def restore(self, project_name: str, project_slug: str, user: User, billing=None
                 Label(project=project, **entry) for entry in self.data["labels"]
             )
             self.labels_map = {label.name: label for label in labels}
+            if "categories" in self.data:
+                self.restore_categories(self.data["categories"], None)
path: weblate/trans/backups.py
user: login: amCap1712
body: When the parameter of a function has a default value of None, the annotation automatically picks up that is optional.
![image](https://github.com/user-attachments/assets/553e76fb-7e31-4629-a95a-e875d3046f2d)
 If you prefer, I can make it explicit.
created_at: 2025-04-16T10:22:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -608,6 +639,23 @@ def import_language(self, code: str):
             )
             return language
 
+    def restore_categories(
+        self, categories: list[dict], parent_category: Category = None
+    ):
+        for category in categories:
+            category_obj = Category(
+                name=category["name"],
+                slug=category["slug"],
+                category=parent_category,
+                project=self.project,
+            )
+            category_obj = Category.objects.bulk_create([category_obj])[0]
path: weblate/trans/backups.py
user: login: amCap1712
body: fixed.
created_at: 2025-04-16T12:26:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -630,6 +678,8 @@ def restore(self, project_name: str, project_slug: str, user: User, billing=None
                 Label(project=project, **entry) for entry in self.data["labels"]
             )
             self.labels_map = {label.name: label for label in labels}
+            if "categories" in self.data:
+                self.restore_categories(self.data["categories"], None)
path: weblate/trans/backups.py
user: login: nijel
body: Yes, please see feedback from mypy on https://github.com/WeblateOrg/weblate/pull/14531/files. It is not yet enforced because we're not yet there with the type annotations, but I want new code to pass.
created_at: 2025-04-16T12:52:03Z
author_association: MEMBER

PR comments:
user: login: nijel
created_at: 2025-04-16T09:34:46Z
body: Let's make this into 5.11.1. https://github.com/WeblateOrg/weblate/pull/14609 updates the schema and this can be then rebased on top of it.
author_association: MEMBER
user: login: nijel
created_at: 2025-04-16T15:12:44Z
body: Merged, thanks for your contribution!
author_association: MEMBER