Repository: WeblateOrg/weblate

Issue:
IntegrityError during restore from project backup
### Describe the issue

I have an issue where I try to restore a project in Weblate from a project backup (done via the Weblate UI).
The inital upload is fine but after I enter the project name and press "Save" the UI shows a "Server Error".
Looking into the logs I see the a IntegrityError (see below).

However, this makes no sense to me since the project works fine in the Weblate from where I took the backup.
Furthermore, this does **not** happen with all projects, I successfully exported & imported other projects like this into this instance.

### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar filed issues in this repository.

### Steps to reproduce the behavior

1. Export project using backup UI via browser
2. Upload zip file from step one into target instance via web UI (under create project -> import)
3. Enter project name
4. Press "Save"

### Expected behavior

Project should be restored.

### Screenshots

_No response_

### Exception traceback

```pytb
nginx stdout | 10.89.0.1 - - [31/Jan/2024:12:16:55 +0000] "GET /create/project/import/ HTTP/1.0" 200 15442 "https://weblate-ts.hilti.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
celery-celery stderr | [2024-01-31 12:17:14,286: INFO/MainProcess] Task weblate.utils.tasks.heartbeat[eeb39cfa-eba3-40d8-baf6-5c142d64d568] received
gunicorn stderr | [2024-01-31 12:17:29,650: ERROR/438] Internal Server Error: /create/project/import/
gunicorn stderr | Traceback (most recent call last):
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
gunicorn stderr |     return self.cursor.execute(sql, params)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
gunicorn stderr |     raise ex.with_traceback(None)
gunicorn stderr | psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "trans_translation_component_id_language_id_331fa2a7_uniq"
gunicorn stderr | DETAIL:  Key (component_id, language_id)=(4, 201) already exists.
gunicorn stderr |
gunicorn stderr | The above exception was the direct cause of the following exception:
gunicorn stderr |
gunicorn stderr | Traceback (most recent call last):
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
gunicorn stderr |     response = get_response(request)
gunicorn stderr |                ^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
gunicorn stderr |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
gunicorn stderr |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
gunicorn stderr |     return self.dispatch(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
gunicorn stderr |     return bound_method(*args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
gunicorn stderr |     return view_func(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
gunicorn stderr |     return bound_method(*args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/ratelimit.py", line 132, in rate_wrap
gunicorn stderr |     return function(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/views/create.py", line 123, in dispatch
gunicorn stderr |     return super().dispatch(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
gunicorn stderr |     return handler(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/views/create.py", line 179, in post
gunicorn stderr |     return super().post(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/views/create.py", line 98, in post
gunicorn stderr |     return super().post(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 184, in post
gunicorn stderr |     return super().post(request, *args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 153, in post
gunicorn stderr |     return self.form_valid(form)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/views/create.py", line 191, in form_valid
gunicorn stderr |     project = self.projectbackup.restore(
gunicorn stderr |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
gunicorn stderr |     return func(*args, **kwds)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/backups.py", line 578, in restore
gunicorn stderr |     self.load_components(zipfile, self.restore_component)
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/backups.py", line 333, in load_components
gunicorn stderr |     callback(zipfile, data)
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/trans/backups.py", line 425, in restore_component
gunicorn stderr |     translations = Translation.objects.bulk_create(translations)
gunicorn stderr |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
gunicorn stderr |     return getattr(self.get_queryset(), name)(*args, **kwargs)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 803, in bulk_create
gunicorn stderr |     returned_columns = self._batched_insert(
gunicorn stderr |                        ^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1831, in _batched_insert
gunicorn stderr |     self._insert(
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
gunicorn stderr |     return query.get_compiler(using=using).execute_sql(returning_fields)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
gunicorn stderr |     cursor.execute(sql, params)
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
gunicorn stderr |     return self._execute_with_wrappers(
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
gunicorn stderr |     return executor(sql, params, many, context)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
gunicorn stderr |     with self.db.wrap_database_errors:
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
gunicorn stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
gunicorn stderr |     return self.cursor.execute(sql, params)
gunicorn stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gunicorn stderr |   File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
gunicorn stderr |     raise ex.with_traceback(None)
gunicorn stderr | django.db.utils.IntegrityError: duplicate key value violates unique constraint "trans_translation_component_id_language_id_331fa2a7_uniq"
gunicorn stderr | DETAIL:  Key (component_id, language_id)=(4, 201) already exists.
```


### How do you run Weblate?

Docker container

### Weblate versions

The version from which I took the backup is 4.14.1 (quite old I know which is why I am trying to move to a new instance)
I tried importing the project into instances with version 5.3.1, 5.0.2 and 4.18.2.

### Weblate deploy checks

```shell
check stderr | System check identified some issues:
check stderr |
check stderr | INFOS:
check stderr | ?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
check stderr |  HINT: https://docs.weblate.org/en/weblate-5.3.1/admin/install.html#collecting-errors
check stderr | ?: (weblate.I028) Backups are not configured, it is highly recommended for production use
check stderr |  HINT: https://docs.weblate.org/en/weblate-5.3.1/admin/backup.html
check stderr |
check stderr | System check identified 2 issues (1 silenced).
```


### Additional context

_No response_


Discussion:
This can probably happen if the old server had defined languages, which are also handled by Weblate built-in aliases. Upon restore, it will try to create multiple translations with a same language. A workaround would be to create the missing languages manually.

Resolving patch:
diff --git a/weblate/trans/backups.py b/weblate/trans/backups.py
--- a/weblate/trans/backups.py
+++ b/weblate/trans/backups.py
@@ -8,6 +8,7 @@
 
 import json
 import os
+from collections import defaultdict
 from datetime import datetime
 from itertools import chain
 from shutil import copyfileobj
@@ -329,6 +330,18 @@ def load_components(self, zipfile, callback: Callable | None = None):
                     raise ValueError(
                         f'Component {data["component"]["name"]} uses unsupported VCS: {data["component"]["vcs"]}'
                     )
+                # Validate translations have unique languages
+                languages = defaultdict(list)
+                for item in data["translations"]:
+                    language = self.import_language(item["language_code"])
+                    languages[language.code].append(item["language_code"])
+
+                for code, values in languages.items():
+                    if len(values) > 1:
+                        raise ValueError(
+                            f"Several languages from backup map to single language on this server {values} -> {code}"
+                        )
+
                 if callback is not None:
                     callback(zipfile, data)
 


Test patch:
diff --git a/weblate/trans/tests/data/projectbackup-duplicate.zip b/weblate/trans/tests/data/projectbackup-duplicate.zip
new file mode 100644
Binary files /dev/null and b/weblate/trans/tests/data/projectbackup-duplicate.zip differ
diff --git a/weblate/trans/tests/test_backups.py b/weblate/trans/tests/test_backups.py
--- a/weblate/trans/tests/test_backups.py
+++ b/weblate/trans/tests/test_backups.py
@@ -23,6 +23,7 @@
 
 TEST_SCREENSHOT = get_test_file("screenshot.png")
 TEST_BACKUP = get_test_file("projectbackup-4.14.zip")
+TEST_BACKUP_DUPLICATE = get_test_file("projectbackup-duplicate.zip")
 
 
 class BackupsTest(ViewTestCase):
@@ -154,6 +155,13 @@ def test_restore_4_14(self):
             set(restored.label_set.values_list("name", "color")),
         )
 
+    def test_restore_duplicate(self):
+        if not connection.features.can_return_rows_from_bulk_insert:
+            raise SkipTest("Not supported")
+        restore = ProjectBackup(TEST_BACKUP_DUPLICATE)
+        with self.assertRaises(ValueError):
+            restore.validate()
+
     def test_cleanup(self):
         cleanup_project_backups()
         self.assertLessEqual(len(self.project.list_backups()), 3)