Repository: digitalfabrik/integreat-cms

Issue:
Fix XLIFF segmentation
### Motivation
<!-- A clear and concise description of what the motivation for the new feature is, and what problem it is solving. -->
In turns out, that the segmentation of XLIFF files is more complicated than initially thought.
The current Wordpress + WPML setup does the following:

1. The content in the database is just plain text with `\n`, `\r` and `\t` characters to represent paragraphs
2. The corresponding `<p>` tags are only inserted in the page API, not in the XLIFF export/import
3. The content is segmented in MemoQ with a specialized WPML filter that presumably converts each sentence into one XLIFF segment

When we add `<p>`-tags during the data migration from WordPress to Django (see https://github.com/digitalfabrik/integreat-cms/issues/1118 and https://github.com/digitalfabrik/cms/pull/1126), segmentation breaks and our translation memories don't work anymore.

### Proposed Solution
<!-- A clear and concise description of the feature you would like to add, and how it solves the motivating problem. -->
I see three options:
1. Just keep the wordpress database structure without `<p>`-tags and add them on runtime in the API - probably bad for performance and it doesn't make too much sense
2. The quick and dirty approach: Just reverse the `wpautop()` function and apply it before each XLIFF export in Django. After that, the result should be identical to the content WPML is exporting and the MemoQ-WPML-filter can also be applied to the Django-XLIFF.
3. The long-term solution: Find out what the MemoQ-WPML-filter is doing and do the segmentation already during the XLIFF-export in Django. If we manage to create the same segments, this would be better because the idea of the XLIFF standard in the first place is to put each segment into one `<trans-unit>` tag and not to put the entire content in one large `<![CDATA[]]>`-section and apply some dark magic to get segments out of this large blob (then why use XLIFF in the first place?)...



### Additional Context
<!-- Add any other information or screenshots about the feature request here. -->




Discussion:
So, a short summary of the found solution: 

1. For our XLIFF-Im/Exports we need the specific WPML-header in our XLIFF-File, so the WPML filter successfully detects an WPML file:
`<header><phase-group><phase  phase-name="shortcodes" process-name="Shortcodes  identification"></phase><phase phase-name="post_type"  process-name="Post  type"></phase></phase-group></header>`
2. The id field of the <trans unit> has to be the same as in the old XLIFF files as segments with different context ids will never be considered as the same context and even though they are the exact content this won't result in an 101% match. Therefore, the id field should be "body" to match our old TMs
3. Consequences for the translation process with MemoQ: To match XLIFF-files created by our django system, we have to import them with the WPML filter and then the matches with our TM should work out :slightly_smiling_face: 

Resolving patch:
diff --git a/integreat_cms/core/settings.py b/integreat_cms/core/settings.py
--- a/integreat_cms/core/settings.py
+++ b/integreat_cms/core/settings.py
@@ -739,7 +739,7 @@
 XLIFF_DEFAULT_FIELDS = ("title", "content")
 
 #: A mapping for changed field names to preserve backward compatibility after a database field was renamed
-XLIFF_LEGACY_FIELDS = {"body": "content", "text": "content"}
+XLIFF_LEGACY_FIELDS = {"body": "content"}
 
 #: The directory where xliff files are stored
 XLIFF_ROOT = os.environ.get("INTEGREAT_CMS_XLIFF_ROOT", os.path.join(BASE_DIR, "xliff"))
diff --git a/integreat_cms/xliff/xliff1_serializer.py b/integreat_cms/xliff/xliff1_serializer.py
--- a/integreat_cms/xliff/xliff1_serializer.py
+++ b/integreat_cms/xliff/xliff1_serializer.py
@@ -1,6 +1,7 @@
 import logging
 from urllib.parse import urlparse
 
+from django.conf import settings
 from django.core.serializers import base
 
 from ..cms.models import Page, PageTranslation, Language
@@ -54,6 +55,23 @@ def start_object(self, obj):
                 "target-language": obj.language.slug,
             },
         )
+        # This header is required to make sure the XLIFF file can be segmented with MemoQ's WPML filter to get the same
+        # translation memory as with the legacy export via WordPress/WPML. See also:
+        # https://docs.memoq.com/current/en/Places/wpml-xliff-filter.html
+        self.xml.startElement("header", {})
+        self.xml.startElement("phase-group", {})
+        self.xml.addQuickElement(
+            "phase",
+            attrs={
+                "phase-name": "shortcodes",
+                "process-name": "Shortcodes identification",
+            },
+        )
+        self.xml.addQuickElement(
+            "phase", attrs={"phase-name": "post_type", "process-name": "Post type"}
+        )
+        self.xml.endElement("phase-group")
+        self.xml.endElement("header")
         self.xml.startElement("body", {})
 
     def handle_field(self, obj, field):
@@ -66,9 +84,14 @@ def handle_field(self, obj, field):
         :param field: The model field
         :type field: ~django.db.models.Field
         """
+        # Use legacy field name if available
+        REVERSE_XLIFF_LEGACY_FIELDS = dict(
+            map(reversed, settings.XLIFF_LEGACY_FIELDS.items())
+        )
+        field_name = REVERSE_XLIFF_LEGACY_FIELDS.get(field.name, field.name)
         attrs = {
-            "id": field.name,
-            "resname": field.name,
+            "id": field_name,
+            "resname": field_name,
             "restype": "string",
             "datatype": "html",
         }


Test patch: