Repository: WeblateOrg/weblate

Issue:
Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
Hello!

When I trying to add new component to my project, I got an error:

```
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/384] Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/ForkPoolWorker-1] Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/384] Failure while executing task: UnicodeEncodeError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'kwargs', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_after_save of weblate at 0x7f2c21964908>, 'task_id': 'bc2cbfd9-0fb3-41dd-bc64-6814deb86fb5', 'args': [28, True, False, False, False], 'kwargs': {'skip_push': True}, 'traceback': <traceback object at 0x7f2c1830d908>, 'einfo': <ExceptionInfo: UnicodeEncodeError('utf-8', '\ud83d\udc68\u200d\ud83d\udcbbАгенты', 0, 2, 'surrogates not allowed')>}
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/ForkPoolWorker-1] Failure while executing task: UnicodeEncodeError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'kwargs', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_after_save of weblate at 0x7f2c21964908>, 'task_id': 'bc2cbfd9-0fb3-41dd-bc64-6814deb86fb5', 'args': [28, True, False, False, False], 'kwargs': {'skip_push': True}, 'traceback': <traceback object at 0x7f2c1830d908>, 'einfo': <ExceptionInfo: UnicodeEncodeError('utf-8', '\ud83d\udc68\u200d\ud83d\udcbbАгенты', 0, 2, 'surrogates not allowed')>}
```

(complete log attached to the issue - [creationError.log](https://github.com/WeblateOrg/weblate/files/5690764/creationError.log))
I tried changing file_format atttribute to 'properties-utf8' or just 'properties' but it didn't help a bit. Source language is russian, format - java properties.

Weblate version is 4.3.2.

How I can workaround this error?
Thanks!




Discussion:
Normalizing the properties files so that it doesn't use surrogates should help. But still this should be handled more gracefully.

Can you try it with debug enabled in Weblate, so that the log contains traceback as well?
Hi!

I didn't plan to use surrogates in properties, I think this is an a mistake or some conversion error.
My weblate already running (in docker) with environmental options:

# Weblate setup
WEBLATE_DEBUG=1
WEBLATE_LOGLEVEL=DEBUG

How I can set logging to be more verbose?
That should be good enough .... and it's in the log, sorry for not seeing it there:

```
weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
weblate_1   | celery-main stderr |     R = retval = fun(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
weblate_1   | celery-main stderr |     return self.run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/autoretry.py", line 35, in run
weblate_1   | celery-main stderr |     return task._orig_run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 279, in component_after_save
weblate_1   | celery-main stderr |     changed_git, changed_setup, changed_template, changed_variant, skip_push
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2366, in after_save
weblate_1   | celery-main stderr |     was_change = self.create_translations()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1741, in create_translations
weblate_1   | celery-main stderr |     force, langs, request, changed_template, from_link
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1853, in _create_translations
weblate_1   | celery-main stderr |     self, lang, code, path, force, request=request
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 88, in check_sync
weblate_1   | celery-main stderr |     translation.check_sync(force, request=request)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 410, in check_sync
weblate_1   | celery-main stderr |     self.sync_unit(dbunits, updated, id_hash, unit, pos + 1)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 308, in sync_unit
weblate_1   | celery-main stderr |     newunit.update_from_unit(unit, pos, is_new)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/unit.py", line 589, in update_from_unit
weblate_1   | celery-main stderr |     run_checks=not same_source or not same_target or not same_state,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/unit.py", line 341, in save
weblate_1   | celery-main stderr |     update_fields=update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 754, in save
weblate_1   | celery-main stderr |     force_update=force_update, update_fields=update_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 792, in save_base
weblate_1   | celery-main stderr |     force_update, using, update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 895, in _save_table
weblate_1   | celery-main stderr |     results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 935, in _do_insert
weblate_1   | celery-main stderr |     using=using, raw=raw,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
weblate_1   | celery-main stderr |     return getattr(self.get_queryset(), name)(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 1254, in _insert
weblate_1   | celery-main stderr |     return query.get_compiler(using=using).execute_sql(returning_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
weblate_1   | celery-main stderr |     cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute
weblate_1   | celery-main stderr |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
weblate_1   | celery-main stderr |     return executor(sql, params, many, context)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
```

Not sure if Weblate should try to normalize these or fail on parsing, but it should fail on parsing the file, not when saving it to the database.

See also https://github.com/psycopg/psycopg2/issues/586
As a note: previously, in Weblate version 2.17 this file was imported without any errors. 
This has been changed in Python 3, see above referenced psycopg issue...

Resolving patch:
diff --git a/weblate/trans/util.py b/weblate/trans/util.py
--- a/weblate/trans/util.py
+++ b/weblate/trans/util.py
@@ -31,6 +31,7 @@
 from django.utils.translation import gettext as _
 from django.utils.translation import gettext_lazy
 from lxml import etree
+from translate.misc.multistring import multistring
 from translate.storage.placeables.lisa import parse_xliff, strelem_to_xml
 
 from weblate.utils.data import data_dir
@@ -74,8 +75,14 @@ def get_string(text):
     # Check for null target (happens with XLIFF)
     if text is None:
         return ""
-    if hasattr(text, "strings"):
-        return join_plural(text.strings)
+    if isinstance(text, multistring):
+        return join_plural(get_string(str(item)) for item in text.strings)
+    if isinstance(text, str):
+        # Remove possible surrogates in the string. There doesn't seem to be
+        # a cheap way to detect this, so do the conversion in both cases. In
+        # case of failure, this at least fails when parsing the file instead
+        # being that later when inserting the data to the database.
+        return text.encode("utf-16", "surrogatepass").decode("utf-16")
     # We might get integer or float in some formats
     return str(text)
 


Test patch:
diff --git a/weblate/trans/tests/test_util.py b/weblate/trans/tests/test_util.py
--- a/weblate/trans/tests/test_util.py
+++ b/weblate/trans/tests/test_util.py
@@ -18,8 +18,14 @@
 #
 
 from django.test import SimpleTestCase
+from translate.misc.multistring import multistring
 
-from weblate.trans.util import cleanup_path, cleanup_repo_url, translation_percent
+from weblate.trans.util import (
+    cleanup_path,
+    cleanup_repo_url,
+    get_string,
+    translation_percent,
+)
 
 
 class HideCredentialsTest(SimpleTestCase):
@@ -90,3 +96,19 @@ def test_slash(self):
 
     def test_double_slash(self):
         self.assertEqual(cleanup_path("foo//*.po"), "foo/*.po")
+
+
+class TextConversionTest(SimpleTestCase):
+    def test_multistring(self):
+        self.assertEqual(get_string(multistring(["foo", "bar"])), "foo\x1e\x1ebar")
+
+    def test_surrogates(self):
+        self.assertEqual(
+            get_string("\ud83d\udc68\u200d\ud83d\udcbbАгенты"), "👨‍💻Агенты"
+        )
+
+    def test_none(self):
+        self.assertEqual(get_string(None), "")
+
+    def test_int(self):
+        self.assertEqual(get_string(42), "42")