Repository: WeblateOrg/weblate

Issue:
XLF missing <target> translation in latest version of Weblate
**Describe the bug**
We updated our (docker-compose) Weblate installation (by simply doing a `docker-compose pull`) to the latest version (unfortunately I don't remember the exact version we had before). Since that update, when we translate a string, it only ends up as `<source>` but not as `<target>` in the translated XLF file.

**To Reproduce**
Steps to reproduce the behavior:
1. Translate an existing (but not yet translated) string in an existing language.
2. Commit, rebase and push to GIT
3. Translated XLF contains new `<trans-unit>` with `<source>` (translated!) but without `<target>`

**Expected behavior**
The XLF file must contain both `<source>` and `<target>` tags.
It used to work with the previous version of Weblate.

**Examples**
Correct translation with a previous version of Weblate (unfortunately unknown version pulled on January 25, 2019):

```xml
    <trans-unit id="personsHeader" datatype="html">
        <source>Persons</source>
        <target>Personnes</target><context-group purpose="location">
          <context context-type="sourcefile">app/app.component.ts</context>
          <context context-type="linenumber">6</context>
        </context-group>
        <note priority="1" from="description">Header for the persons list</note>
      </trans-unit>
```

Wrong translation (in the same file) with the latest version of Weblate (version see below):
```xml
      <trans-unit id="networkConfigTitle" datatype="html">
        <source>Configuration Réseau</source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/network-config/network-config.component.html</context>
          <context context-type="linenumber">13</context>
        </context-group>
        <note priority="1" from="description">Network configuration title</note>
      </trans-unit>
```

**Server configuration and status**

```
 * Weblate 3.5.1
 * Python 3.6.7
 * Django 2.1.7
 * Celery 4.2.1
 * celery-batches 0.2
 * six 1.12.0
 * social-auth-core 3.1.0
 * social-auth-app-django 3.1.0
 * django-appconf 1.0.3
 * translate-toolkit 2.3.1
 * translation-finder 1.0
 * Whoosh 2.7.4
 * defusedxml 0.5.0
 * Git 2.17.1
 * Pillow 5.1.0
 * python-dateutil 2.8.0
 * lxml 4.2.1
 * django-crispy-forms 1.7.2
 * django_compressor 2.2
 * djangorestframework 3.9.2
 * user-agents 1.1.0
 * jellyfish 0.7.1
 * diff-match-patch 20121119
 * pytz 2018.9
 * pyuca 1.2
 * PyYAML 3.12
 * tesserocr 2.4.0
 * Mercurial 4.5.3
 * git-svn 2.17.1
 * Database backends: django.db.backends.postgresql
 * Cache backends: avatar:FileBasedCache, default:RedisCache
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-46-generic (x86_64)
```

```
System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W018) You should not have DEBUG set to True in deployment.

INFOS:
?: (weblate.I021) Error collection is not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-3.5.1/admin/install.html#collecting-errors

System check identified 5 issues (0 silenced).
```

**Additional context**
If you need additional information (like component configuration in Weblate), please feel free to ask here.



Discussion:
Can you please list your translation component configuration?
Possibly related to https://github.com/WeblateOrg/weblate/issues/2600
**_Part 1:_**
![image](https://user-images.githubusercontent.com/7595126/55326088-3c453f00-5487-11e9-872f-a12afecf25c5.png)
**_Part 2:_**
![image](https://user-images.githubusercontent.com/7595126/55326146-5a12a400-5487-11e9-91f6-a01858b562e2.png)
**_Part 3:_**
![image](https://user-images.githubusercontent.com/7595126/55326188-6e56a100-5487-11e9-91a5-eeb376348fa5.png)

I hope, these are the relevant sections.
BTW, I think the `<source>` should be copied from the base file and only the `<target>` should be updated. With the old version it was just copying the same string in both places.
Just for completeness these are the translation units from the messages.xlf (base file):

```xml
      <trans-unit id="personsHeader" datatype="html">
        <source>Persons</source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/homescreen/homescreen.component.html</context>
          <context context-type="linenumber">4</context>
        </context-group>
        <note priority="1" from="description">Header for the persons list</note>
      </trans-unit>
```

```xml
      <trans-unit id="networkConfigTitle" datatype="html">
        <source>
    Network Configuration
  </source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/network-config/network-config.component.html</context>
          <context context-type="linenumber">13</context>
        </context-group>
        <note priority="1" from="description">Network configuration title</note>
      </trans-unit>
```
Indeed it looks like the same issue as @ShadowXVII is having. As I'm not allowing the editing of the base file, issue #2521 didn't bother me; but the change b21089b is a regression for me as well.
Unset monolingual base file in settings and it will work correctly. It should be set only if you are using monolingual translations, also see https://docs.weblate.org/en/latest/formats.html#bilingual-and-monolingual-formats.
Thanks for your reply. I'm a bit confused; we had the following setup until now:
1. Angular on the build server generates the (English) src/frontend/web/src/i18n/messages.xlf
1. Weblate uses messages.xlf to import new strings to be translated
1. Weblate generates the messages.de.xlf and messages.fr.xlf files with all available translations
1. Weblate commits and pushes those changes back to our Git

This used to work just fine for us until the mentioned recent change.

If we now remove the "Monolingual base language file", how will Weblate know when Angular added new translatable strings in messages.xlf? Angular does *not* generate messages.de.xlf and messages.fr.xlf.
Hmm, this is in conflict what was requested in https://github.com/WeblateOrg/weblate/issues/2521, we probably need some setting to decide between these two.
#2521 should only apply to the base/source file when "allow editing base language" is on -- i.e messages.xlf (which only contains sources, right @UncleSamSwiss ?)

The other files should continue to work like normal (source and target).
@ShadowXVII: right.

I agree with all @ShadowXVII says, but it would be great if we had <source> showing the original string from the base language file. This is not strictly necessary, but it would make more sense when you read a translated file: you would see the source and the translation, not twice the translation.

If what I'm writing here doesn't make sense, please let me know and I can elaborate with an example.
I do have the same issue translating a TYPO3 project. It has the following (common?) file layout for its xliff files:

`locallang.xlf` is a monolingual source file. It contains only source and no target elements. It is mostly hand-written by the developers when they add new keys. 

`*.locallang.xlf` are bilingual translation files. These files should contain the same source strings as `locallang.xlf` (unless stale) and target elements in their respective languages. 

So I do need both: A monolingual translation base and bilingual translation files. This is how I tried to configure this in the component setup:

![Screenshot 2019-04-08 at 23 39 05](https://user-images.githubusercontent.com/243504/55758612-8b770b00-5a57-11e9-8636-c2b777902fa0.png)

Expected behavior:
Editors should be allowed to edit source. 
* If a translator edits a translation source, the _source_ element in locallang.xlf should be updated. The source elements of the translation files shall be left untouched (stale). 
* If an editor edits a translation, he/she shall see the source from the monolingual locallang.xlf source file. Once the editor saves the translation, the _source_ element of the bilingual translation file *.locallang.xlf shall be updated to match the source from the monolingual source file. The _target_ element in the file should be updated to the newly provided translation. 

Actual behavior:
If a translator edits a translation source, the _source_ element in locallang.xlf gets updated. Great!
If an editor edits a translation via GUI, the _source_ element in the respective *.locallang.xlf file gets set to the translated string. Instead, I would expect the source to be updated to the translation source from the loccallang.xlf file and the target to be set to the newly provided translation. 

As far as I can see, we do need a mixed mode that allows to define a monolingual source file as well as bilingual translation files at the same time. I do believe that this is also what @UncleSamSwiss is asking for. 


Resolving patch:
diff --git a/weblate/formats/base.py b/weblate/formats/base.py
--- a/weblate/formats/base.py
+++ b/weblate/formats/base.py
@@ -61,10 +61,11 @@ class TranslationUnit(object):
     It handles ID/template based translations and other API differences.
     """
 
-    def __init__(self, unit, template=None):
+    def __init__(self, parent, unit, template=None):
         """Create wrapper object."""
         self.unit = unit
         self.template = template
+        self.parent = parent
         if template is not None:
             self.mainunit = template
         else:
@@ -186,14 +187,16 @@ def get_identifier(cls):
         return cls.format_id
 
     @classmethod
-    def parse(cls, storefile, template_store=None, language_code=None):
+    def parse(cls, storefile, template_store=None, language_code=None,
+              is_template=False):
         """Parse store and returns TranslationFormat instance.
 
         This wrapper is needed for AutodetectFormat to be able to return
         instance of different class."""
-        return cls(storefile, template_store, language_code)
+        return cls(storefile, template_store, language_code, is_template)
 
-    def __init__(self, storefile, template_store=None, language_code=None):
+    def __init__(self, storefile, template_store=None, language_code=None,
+                 is_template=False):
         """Create file format object, wrapping up translate-toolkit's store."""
         if (not isinstance(storefile, six.string_types) and
                 not hasattr(storefile, 'mode')):
@@ -212,6 +215,7 @@ def __init__(self, storefile, template_store=None, language_code=None):
 
         # Remember template
         self.template_store = template_store
+        self.is_template = is_template
 
     def get_filenames(self):
         if isinstance(self.storefile, six.string_types):
@@ -261,7 +265,7 @@ def _find_unit_template(self, context):
         else:
             add = False
 
-        return (self.unit_class(ttkit_unit, template_ttkit_unit), add)
+        return (self.unit_class(self, ttkit_unit, template_ttkit_unit), add)
 
     @cached_property
     def _source_index(self):
@@ -316,15 +320,17 @@ def save(self):
 
     @cached_property
     def mono_units(self):
-        return [self.unit_class(None, unit) for unit in self.store.units]
+        return [self.unit_class(self, None, unit) for unit in self.store.units]
 
     @cached_property
     def all_units(self):
         """List of all units."""
         if not self.has_template:
-            return [self.unit_class(unit) for unit in self.store.units]
+            return [self.unit_class(self, unit) for unit in self.store.units]
         return [
-            self.unit_class(self.find_unit_mono(unit.context), unit.template)
+            self.unit_class(
+                self, self.find_unit_mono(unit.context), unit.template
+            )
             for unit in self.template_store.mono_units
         ]
 
diff --git a/weblate/formats/ttkit.py b/weblate/formats/ttkit.py
--- a/weblate/formats/ttkit.py
+++ b/weblate/formats/ttkit.py
@@ -202,9 +202,10 @@ class TTKitFormat(TranslationFormat):
     loader = ('', '')
     new_translation = None
 
-    def __init__(self, storefile, template_store=None, language_code=None):
+    def __init__(self, storefile, template_store=None, language_code=None,
+                 is_template=False):
         super(TTKitFormat, self).__init__(
-            storefile, template_store, language_code
+            storefile, template_store, language_code, is_template
         )
         # Set language (needed for some which do not include this)
         if (language_code is not None and
@@ -460,11 +461,15 @@ def set_target(self, target):
             converted = xliff_string_to_rich(target)
         except XMLSyntaxError:
             converted = target
-        # Use source for monolingual files if target is not set
-        if self.template is not None and not self.template.target:
-            self.unit.rich_source = converted
-        else:
-            self.unit.rich_target = converted
+        if self.template is not None:
+            if not self.parent.is_template:
+                # Use source for monolingual files if editing template
+                self.unit.rich_source = converted
+                return
+            if self.unit.source:
+                # Update source if it is already set to keep it in sync
+                self.unit.rich_source = self.template.source
+        self.unit.rich_target = converted
 
     @cached_property
     def xliff_node(self):
@@ -944,9 +949,10 @@ class CSVFormat(TTKitFormat):
     unit_class = CSVUnit
     autoload = ('.csv',)
 
-    def __init__(self, storefile, template_store=None, language_code=None):
+    def __init__(self, storefile, template_store=None, language_code=None,
+                 is_template=False):
         super(CSVFormat, self).__init__(
-            storefile, template_store, language_code
+            storefile, template_store, language_code, is_template
         )
         # Remove template if the file contains source, this is needed
         # for import, but probably usable elsewhere as well
diff --git a/weblate/trans/models/translation.py b/weblate/trans/models/translation.py
--- a/weblate/trans/models/translation.py
+++ b/weblate/trans/models/translation.py
@@ -219,7 +219,8 @@ def load_store(self):
         store = self.component.file_format_cls.parse(
             self.get_filename(),
             self.component.template_store,
-            language_code=self.language_code
+            language_code=self.language_code,
+            is_template=self.is_template,
         )
         store_post_load.send(
             sender=self.__class__,


Test patch: