Repository: WeblateOrg/weblate

Issue:
Auto Translation using Google Translate V3 doesn't preserve New Line characters
### Describe the issue

When a String contains a new line character (`\n`). It's "Google Translate V3" auto-translation, replaces new-line character with a single space. 

But, AWS Translation preserves new-line characters properly.

See Screenshots for the reference.





### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.

### Steps to reproduce the behavior

- Given a string's english value is `Hello\nWorld`
- Then it's auto translation in Hindi Language, using "Google Translate V3" is `हैलो वर्ल्ड` (ISSUE)
- And It's auto translation in Hindi Language, using **AWS** is `नमस्ते\nवर्ल्ड`

### Expected behavior

Auto Translation in Hindi, using "Google Translate V3" should be `हैलो\nवर्ल्ड`

### Screenshots

Google Translate V3 API:
<img width="2048" alt="Google Translate V3 API" src="https://github.com/WeblateOrg/weblate/assets/2399621/8f5b16bb-00de-45e8-adbf-f4c70323dfb2">

AWS API:
<img width="2048" alt="AWS API" src="https://github.com/WeblateOrg/weblate/assets/2399621/18cc766b-8a61-45c0-822a-93e278a39614">

### Exception traceback

_No response_

### How do you run Weblate?

Docker container

### Weblate versions

 * Weblate: 4.17
 * Django: 4.2
 * siphashc: 2.1
 * translate-toolkit: 3.8.6
 * lxml: 4.9.2
 * Pillow: 9.5.0
 * nh3: 0.2.11
 * python-dateutil: 2.8.2
 * social-auth-core: 4.4.2
 * social-auth-app-django: 5.2.0
 * django-crispy-forms: 2.0
 * oauthlib: 3.2.2
 * django-compressor: 4.3.1
 * djangorestframework: 3.14.0
 * django-filter: 23.1
 * django-appconf: 1.0.5
 * user-agents: 2.2.0
 * filelock: 3.11.0
 * rapidfuzz: 3.0.0
 * openpyxl: 3.1.2
 * celery: 5.2.7
 * django-celery-beat: 2.5.0
 * kombu: 5.2.4
 * translation-finder: 2.15
 * weblate-language-data: 2023.4
 * html2text: 2020.1.16
 * pycairo: 1.23.0
 * pygobject: 3.44.1
 * diff-match-patch: 20200713
 * requests: 2.28.2
 * django-redis: 5.2.0
 * hiredis: 2.2.2
 * sentry_sdk: 1.21.1
 * Cython: 0.29.34
 * misaka: 2.1.1
 * GitPython: 3.1.31
 * borgbackup: 1.2.4
 * pyparsing: 3.0.9
 * pyahocorasick: 2.0.0
 * python-redis-lock: 4.0.0
 * charset-normalizer: 3.1.0
 * Python: 3.11.3
 * Git: 2.30.2
 * psycopg2: 2.9.6
 * psycopg2-binary: 2.9.6
 * phply: 1.2.6
 * ruamel.yaml: 0.17.21
 * tesserocr: 2.6.0
 * boto3: 1.26.124
 * zeep: 4.2.1
 * aeidon: 1.12
 * iniparse: 0.5
 * mysqlclient: 2.1.1
 * Mercurial: 6.4.2
 * git-svn: 2.30.2
 * git-review: 2.3.1
 * Redis server: 7.0.11
 * PostgreSQL server: 15.3
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.mailgun.org
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 5.19.0-1024-aws (x86_64)

### Weblate deploy checks

```shell
Output of `docker compose exec --user weblate weblate weblate check --deploy`



System check identified some issues:

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.17/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.17/admin/backup.html
?: (weblate.I031) New Weblate version is available, please upgrade to 5.0.2.
	HINT: https://docs.weblate.org/en/weblate-4.17/admin/upgrade.html

System check identified 3 issues (1 silenced).
```


### Additional context

There seems an issue in REST API invocation in https://github.com/WeblateOrg/weblate/blob/c7915cc0954da39169621ce3bbfa19ba189583fe/weblate/machinery/googlev3.py#L63C16-L63C16

It sends whole String as  an element in the Array.  e.g. `["Hello\nWorld"]`. 

If it sends request as multiple array elements, split by new-line characters. e.g. `["Hello","world"]`.
And then join response elements back by new-line character, then this issue might be resolved.

NOTE: There is an additional cavity, if the request array contains any blank string; then Google treats this as an invalid request. So, such elements should be removed from the request element.




Discussion:
Splitting the text needs to be evaluated as well as it might negatively impact translation if Google treats segments as independent. For example, in Czech this should be translated as `"Ahoj\nsvěte"` while independent words would be translated as `Ahoj`, `svět`.

Meanwhile, I've added highlighting of whitespace in machine translation results in https://github.com/WeblateOrg/weblate/pull/10147, so that it is clearly visible.
This issue has been put aside. It is currently unclear if it will ever be implemented as it seems to cover too narrow of a use case or doesn't seem to fit into Weblate.

Please try to clarify the use case or consider proposing something more generic to make it useful to more users.
So, it looks like this should be reported as a Bug to Google Transtate API service, instead here.
I don't think it's realistic to expect machine translation to always keep newlines at the right place.
I agree it won't always keep. But, here the issue is that, it **always removes**. It's never preserved.

We have added many mark-down texts as a Weblate Strings. And by not preserving new-line characters, their formatting gets screwed with auto-translation.
How we did it in another project https://github.com/arvin-pantas/django-autotranslate/commit/85f0d8d6567070411b8abefa0c32391f2f52fb47

Resolving patch:
diff --git a/weblate/machinery/base.py b/weblate/machinery/base.py
--- a/weblate/machinery/base.py
+++ b/weblate/machinery/base.py
@@ -11,6 +11,7 @@
 import time
 from collections import defaultdict
 from hashlib import md5
+from html import escape, unescape
 from itertools import chain
 from typing import TYPE_CHECKING
 from urllib.parse import quote
@@ -702,3 +703,22 @@ def get_glossary_id(
         # Fetch glossaries again, without using cache
         glossaries = self.get_glossaries(use_cache=False)
         return glossaries[glossary_name]
+
+
+class XMLMachineTranslationMixin:
+    hightlight_syntax = True
+
+    def unescape_text(self, text: str):
+        """Unescaping of the text with replacements."""
+        return unescape(text)
+
+    def escape_text(self, text: str):
+        """Escaping of the text with replacements."""
+        return escape(text)
+
+    def format_replacement(self, h_start: int, h_end: int, h_text: str):
+        """Generates a single replacement."""
+        raise NotImplementedError
+
+    def make_re_placeholder(self, text: str):
+        return re.escape(text)
diff --git a/weblate/machinery/deepl.py b/weblate/machinery/deepl.py
--- a/weblate/machinery/deepl.py
+++ b/weblate/machinery/deepl.py
@@ -4,21 +4,25 @@
 
 from __future__ import annotations
 
-import re
-from html import escape, unescape
 from typing import TYPE_CHECKING
 
 from dateutil.parser import isoparse
 from django.core.cache import cache
 
-from .base import BatchMachineTranslation, GlossaryMachineTranslationMixin
+from .base import (
+    BatchMachineTranslation,
+    GlossaryMachineTranslationMixin,
+    XMLMachineTranslationMixin,
+)
 from .forms import DeepLMachineryForm
 
 if TYPE_CHECKING:
     from weblate.trans.models import Unit
 
 
-class DeepLTranslation(BatchMachineTranslation, GlossaryMachineTranslationMixin):
+class DeepLTranslation(
+    XMLMachineTranslationMixin, GlossaryMachineTranslationMixin, BatchMachineTranslation
+):
     """DeepL (Linguee) machine translation support."""
 
     name = "DeepL"
@@ -119,21 +123,10 @@ def download_multiple_translations(
             ]
         return result
 
-    def unescape_text(self, text: str):
-        """Unescaping of the text with replacements."""
-        return unescape(text)
-
-    def escape_text(self, text: str):
-        """Escaping of the text with replacements."""
-        return escape(text)
-
     def format_replacement(self, h_start: int, h_end: int, h_text: str):
         """Generates a single replacement."""
         return f'<x id="{h_start}"></x>'  # noqa: B028
 
-    def make_re_placeholder(self, text: str):
-        return re.escape(text)
-
     def is_glossary_supported(self, source_language: str, target_language: str) -> bool:
         cache_key = self.get_cache_key("glossary_languages")
         languages = cache.get(cache_key)
diff --git a/weblate/machinery/googlev3.py b/weblate/machinery/googlev3.py
--- a/weblate/machinery/googlev3.py
+++ b/weblate/machinery/googlev3.py
@@ -8,11 +8,12 @@
 from google.cloud.translate import TranslationServiceClient
 from google.oauth2 import service_account
 
+from .base import XMLMachineTranslationMixin
 from .forms import GoogleV3MachineryForm
 from .google import GoogleBaseTranslation
 
 
-class GoogleV3Translation(GoogleBaseTranslation):
+class GoogleV3Translation(XMLMachineTranslationMixin, GoogleBaseTranslation):
     """Google Translate API v3 machine translation support."""
 
     setup = None
@@ -64,7 +65,7 @@ def download_translations(
             "contents": [text],
             "target_language_code": language,
             "source_language_code": source,
-            "mime_type": "text/plain",
+            "mime_type": "text/html",
         }
         response = self.client.translate_text(request)
 
@@ -74,3 +75,16 @@ def download_translations(
             "service": self.name,
             "source": text,
         }
+
+    def format_replacement(self, h_start: int, h_end: int, h_text: str):
+        """Generates a single replacement."""
+        return f'<span translate="no" id="{h_start}">{h_text}</span>'  # noqa: B028
+
+    def cleanup_text(self, text, unit):
+        text, replacements = super().cleanup_text(text, unit)
+
+        # Sanitize newlines
+        replacement = '<br translate="no">'
+        replacements[replacement] = "\n"
+
+        return text.replace("\n", replacement), replacements


Test patch:
diff --git a/weblate/machinery/tests.py b/weblate/machinery/tests.py
--- a/weblate/machinery/tests.py
+++ b/weblate/machinery/tests.py
@@ -655,6 +655,22 @@ def test_mapping(self):
             ["zh-Hant-HK", "zh-TW", "zh"],
         )
 
+    def test_replacements(self):
+        machine_translation = self.get_machine()
+        unit = MockUnit(code="cs", source="Hello,\n%s!", flags="c-format")
+        replaced = 'Hello,<br translate="no"><span translate="no" id="7">%s</span>!'
+        replacements = {
+            '<br translate="no">': "\n",
+            '<span translate="no" id="7">%s</span>': "%s",
+        }
+        self.assertEqual(
+            machine_translation.cleanup_text(unit.source, unit),
+            (replaced, replacements),
+        )
+        self.assertEqual(
+            unit.source, machine_translation.uncleanup_text(replacements, replaced)
+        )
+
 
 class AmagamaTranslationTest(BaseMachineTranslationTest):
     MACHINE_CLS = AmagamaTranslation