Repository: WeblateOrg/weblate

Issue:
Updating any file on remote repo triggers an update for all linked components
### Describe the issue

By design component linking is supposed to speed up several operations and save space since this way Weblate has only one copy of the remote repo. However having all components linked to one repo triggers an issue: if you update any file(s) on the remote repo, Weblate will update all of the components that were linked to said repo. This causes performance and efficiency problems in a project where there are many components, and some of them are very large. In my case such update takes 1-1.5h

### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.

### Steps to reproduce the behavior

1. Add the first component from the remote repo
2. Add the rest of components either manually or via automatic component creation, specifying the repo via `weblate://<first_component_URL>`
3. After this is done, commit anything into the remote repo (e.g. change any file)
4. All of the linked components will start updating and you can watch the progress by clicking the little clock icon near the component name

### Expected behavior

Only affected component is updated.

### Screenshots

_No response_

### Exception traceback

_No response_

### How do you run Weblate?

Docker container

### Weblate versions

- Weblate: 4.11.2
- Django: 4.0.3
- siphashc: 2.1
- translate-toolkit: 3.6.0
- lxml: 4.7.1
- Pillow: 9.0.1
- bleach: 4.1.0
- python-dateutil: 2.8.2
- social-auth-core: 4.2.0
- social-auth-app-django: 5.0.0
- django-crispy-forms: 1.14.0
- oauthlib: 3.2.0
- django-compressor: 3.1
- djangorestframework: 3.13.1
- django-filter: 21.1
- django-appconf: 1.0.5
- user-agents: 2.2.0
- filelock: 3.6.0
- jellyfish: 0.9.0
- openpyxl: 3.0.9
- celery: 5.2.3
- kombu: 5.2.3
- translation-finder: 2.12
- weblate-language-data: 2022.2
- html2text: 2020.1.16
- pycairo: 1.20.1
- pygobject: 3.42.0
- diff-match-patch: 20200713
- requests: 2.27.1
- django-redis: 5.2.0
- hiredis: 2.0.0
- sentry_sdk: 1.5.6
- Cython: 0.29.28
- misaka: 2.1.1
- GitPython: 3.1.27
- borgbackup: 1.2.0
- pyparsing: 3.0.7
- pyahocorasick: 1.4.4
- python-redis-lock: 3.7.0
- charset-normalizer: 2.0.12
- Python: 3.10.2
- Git: 2.30.2
- psycopg2: 2.9.3
- psycopg2-binary: 2.9.3
- phply: 1.2.5
- ruamel.yaml: 0.17.21
- tesserocr: 2.5.2
- akismet: 1.1
- boto3: 1.21.12
- zeep: 4.1.0
- aeidon: 1.10.1
- iniparse: 0.5
- mysqlclient: 2.1.0
- Mercurial: 6.1
- git-svn: 2.30.2
- git-review: 2.2.0
- Redis server: 6.2.6
- PostgreSQL server: 13.6
- Database backends: django.db.backends.postgresql
- Cache backends: default:RedisCache, avatar:FileBasedCache
- Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.gmail.com
- OS encoding: filesystem=utf-8, default=utf-8
- Celery: redis://cache:6379/1, redis://cache:6379/1, regular
- Platform: Linux 5.10.60.1-microsoft-standard-WSL2 (x86_64)

### Weblate deploy checks

```shell
System check identified some issues:

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.11.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.11.2/admin/backup.html

System check identified 2 issues (1 silenced).
```


### Additional context

_No response_


Discussion:
The update should be quite fast in case no matching files have been changed. Can you please check the logs to see what actually takes long?
Just tried it, and it's same as here - #7382 - Postgres spams into its output the same error after I made a commit on remote repo on one component.
```
2022-02-28 08:24:15.730 UTC [13284] ERROR:  duplicate key value violates unique constraint "trans_unit_translation_id_id_hash_ab945589_uniq"
2022-02-28 08:24:15.730 UTC [13284] DETAIL:  Key (translation_id, id_hash)=(22249, -4609853089055109701) already exists.
2022-02-28 08:24:15.730 UTC [13284] STATEMENT:  INSERT INTO "trans_unit" ("translation_id", "id_hash", "location", "context", "note", "flags", "source", "previous_source", "target", "state", "original_state", "details", "position", "num_words", "priority", "pending", "timestamp", "extra_flags", "explanation", "variant_id", "source_unit_id") VALUES (22249,  -4609853089055109701, '', '', '', '', 'white mage', '', '', 0, 0, '{"add_unit": true}', 563, 2, 100, true, '2022-02-28T08:24:15.729933+00:00'::timestamptz, '', '', NULL, 4422961) RETURNING "trans_unit"."id"
2022-02-28 08:24:15.734 UTC [13284] ERROR:  duplicate key value violates unique constraint "trans_unit_translation_id_id_hash_ab945589_uniq"
2022-02-28 08:24:15.734 UTC [13284] DETAIL:  Key (translation_id, id_hash)=(237, -3035173709074540976) already exists.
2022-02-28 08:24:15.734 UTC [13284] STATEMENT:  INSERT INTO "trans_unit" ("translation_id", "id_hash", "location", "context", "note", "flags", "source", "previous_source", "target", "state", "original_state", "details", "position", "num_words", "priority", "pending", "timestamp", "extra_flags", "explanation", "variant_id", "source_unit_id") VALUES (237,  -3035173709074540976, '', '', '', '', 'black mage', '', '', 0, 0, '{"add_unit": true}', 557, 2, 100, true, '2022-02-28T08:24:15.734304+00:00'::timestamptz, '', '', NULL, 4423423) RETURNING "trans_unit"."id"
2022-02-28 08:24:15.736 UTC [13284] ERROR:  duplicate key value violates unique constraint "trans_unit_translation_id_id_hash_ab945589_uniq"
2022-02-28 08:24:15.736 UTC [13284] DETAIL:  Key (translation_id, id_hash)=(239, -3035173709074540976) already exists.
2022-02-28 08:24:15.736 UTC [13284] STATEMENT:  INSERT INTO "trans_unit" ("translation_id", "id_hash", "location", "context", "note", "flags", "source", "previous_source", "target", "state", "original_state", "details", "position", "num_words", "priority", "pending", "timestamp", "extra_flags", "explanation", "variant_id", "source_unit_id") VALUES (239,  -3035173709074540976, '', '', '', '', 'black mage', '', '', 0, 0, '{"add_unit": true}', 557, 2, 100, true, '2022-02-28T08:24:15.736381+00:00'::timestamptz, '', '', NULL, 4423423) RETURNING "trans_unit"."id"
2022-02-28 08:24:15.738 UTC [13284] ERROR:  duplicate key value violates unique constraint "trans_unit_translation_id_id_hash_ab945589_uniq"
2022-02-28 08:24:15.738 UTC [13284] DETAIL:  Key (translation_id, id_hash)=(240, -3035173709074540976) already exists.
2022-02-28 08:24:15.738 UTC [13284] STATEMENT:  INSERT INTO "trans_unit" ("translation_id", "id_hash", "location", "context", "note", "flags", "source", "previous_source", "target", "state", "original_state", "details", "position", "num_words", "priority", "pending", "timestamp", "extra_flags", "explanation", "variant_id", "source_unit_id") VALUES (240,  -3035173709074540976, '', '', '', '', 'black mage', '', '', 0, 0, '{"add_unit": true}', 557, 2, 100, true, '2022-02-28T08:24:15.738461+00:00'::timestamptz, '', '', NULL, 4423423) RETURNING "trans_unit"."id"
```
Is any of the affected component used as a glossary? Does it have any terminology entries?
We have glossaries with terminology entries, but they're not part of remote repo and therefore are not linked with any other components.
I ask because this looks like a glossary sync, maybe the logic to trigger it is wrong. Can you try disabling the glossary flag on the glossary components and see if something changes?
Sorry it took some time.
I've disabled all glossaries and `postgres` now didn't throw any errors. However, the whole repo update still took ~45min in total. I've attached logs that I got from `docker logs --follow <container_ID>` command.

[weblate-test.zip](https://github.com/WeblateOrg/weblate/files/8404350/weblate-test.zip)
The terminology sync glossary is verbose, but executed in the background, so it won't slow down the result (I've created https://github.com/WeblateOrg/weblate/issues/7482 to improve this code though).

What file system do you use to run Weblate? That might have some performance implications as well.

Looking into log, I've noticed that processing some components takes much longer than others. A typical component check takes about 30-50ms, but for example "FFXIV Translation/cut_scene-050-voiceman_05005" (exd/cut_scene/050/voiceman_05005/en.csv) takes 500ms. Thousands of components are there in total, and the time adds up. Is there something special about this component?

If these files are huge, the bottleneck might be in:

https://github.com/WeblateOrg/weblate/blob/f27196a7dc878f1b350ceda1ae4cce71e87ba2f3/weblate/trans/models/component.py#L1961-L1969

That parses the monolingual template for every component. It is there as a sanity check to avoid processing translations in case it cannot be parsed, but it will definitely significantly slow down the no-op update. It was introduced in https://github.com/WeblateOrg/weblate/commit/53370ef058e7cf10092c91db2a9c37b10da4e364


Resolving patch:
diff --git a/weblate/trans/exceptions.py b/weblate/trans/exceptions.py
--- a/weblate/trans/exceptions.py
+++ b/weblate/trans/exceptions.py
@@ -31,3 +31,11 @@ class FileParseError(WeblateError):
 
 class PluralFormsMismatch(WeblateError):
     """Plural forms do not match the language."""
+
+
+class InvalidTemplate(WeblateError):
+    """Template file can not be parsed."""
+
+    def __init__(self, nested, message=None):
+        super().__init__(message or self.__doc__)
+        self.nested = nested
diff --git a/weblate/trans/models/component.py b/weblate/trans/models/component.py
--- a/weblate/trans/models/component.py
+++ b/weblate/trans/models/component.py
@@ -55,7 +55,7 @@
     PROJECT_NAME_LENGTH,
     REPO_LENGTH,
 )
-from weblate.trans.exceptions import FileParseError
+from weblate.trans.exceptions import FileParseError, InvalidTemplate
 from weblate.trans.fields import RegexField
 from weblate.trans.mixins import CacheKeyMixin, PathMixin, URLMixin
 from weblate.trans.models.alert import ALERTS, ALERTS_IMPORT
@@ -770,6 +770,7 @@ def __init__(self, *args, **kwargs):
         self.batched_checks = set()
         self.needs_variants_update = False
         self._invalidate_scheduled = False
+        self._template_check_done = False
 
     def generate_changes(self, old):
         def getvalue(base, attribute):
@@ -1932,6 +1933,17 @@ def create_translations(
             )
             return False
 
+    def check_template_valid(self):
+        if self._template_check_done:
+            return
+        if self.has_template():
+            # Avoid parsing if template is invalid
+            try:
+                self.template_store.check_valid()
+            except (FileParseError, ValueError) as exc:
+                raise InvalidTemplate(exc)
+        self._template_check_done = True
+
     def _create_translations(  # noqa: C901
         self,
         force: bool = False,
@@ -1957,17 +1969,7 @@ def _create_translations(  # noqa: C901
 
         source_file = self.template
 
-        if self.has_template():
-            # Avoid parsing if template is invalid
-            try:
-                self.template_store.check_valid()
-            except (FileParseError, ValueError) as exc:
-                self.log_warning(
-                    "skipping update due to error in parsing template: %s", exc
-                )
-                self.update_import_alerts()
-                raise
-        else:
+        if not self.has_template():
             # This creates the translation when necessary
             translation = self.source_translation
 
@@ -2025,9 +2027,17 @@ def _create_translations(  # noqa: C901
                         "DuplicateLanguage", codes=codes, language_code=lang.code
                     )
                     continue
-                translation = Translation.objects.check_sync(
-                    self, lang, code, path, force, request=request
-                )
+                try:
+                    translation = Translation.objects.check_sync(
+                        self, lang, code, path, force, request=request
+                    )
+                except InvalidTemplate as error:
+                    self.log_warning(
+                        "skipping update due to error in parsing template: %s",
+                        error.nested,
+                    )
+                    self.update_import_alerts()
+                    raise error.nested
                 was_change |= bool(translation.reason)
                 translations[translation.id] = translation
                 languages[lang.code] = code
diff --git a/weblate/trans/models/translation.py b/weblate/trans/models/translation.py
--- a/weblate/trans/models/translation.py
+++ b/weblate/trans/models/translation.py
@@ -386,6 +386,7 @@ def check_sync(self, force=False, request=None, change=None):  # noqa: C901
         details["reason"] = self.reason
 
         self.log_info("processing %s, %s", self.filename, self.reason)
+        self.component.check_template_valid()
 
         # List of updated units (used for cleanup and duplicates detection)
         updated = {}


Test patch: