Repository: doccano/doccano

Issue:
[Bug Report] Dataset page: non-ASCII metadata is displayed as Unicode-escaped sequence
Dataset page: non-ASCII metadata is displayed as Unicode-escaped sequence.

![image](https://user-images.githubusercontent.com/1063219/71439555-26f5fc00-270b-11ea-83ec-d589b284c4c4.png)

(UTF-8 metadata. Displaying at annotation page is correct.)

Doccano 1.0.2
[Bug Report] Dataset page: non-ASCII metadata is displayed as Unicode-escaped sequence
Dataset page: non-ASCII metadata is displayed as Unicode-escaped sequence.

![image](https://user-images.githubusercontent.com/1063219/71439555-26f5fc00-270b-11ea-83ec-d589b284c4c4.png)

(UTF-8 metadata. Displaying at annotation page is correct.)

Doccano 1.0.2


Discussion:
@c-w where to fix this in your mind? I'll try.
@kuraga This looks to be a bug with the v1 UI with which I haven't yet familiarized myself (for my use-cases, I'm sticking with the v0 UI for now). @Hironsan can perhaps provide pointers for the v1 UI.
Hello @kuraga,

I can display non-ascii characters:
![image](https://user-images.githubusercontent.com/6737785/73226900-fc3d0e80-41b4-11ea-878b-5026affea28b.png)

Could you please tell me an example of metadata to analyze the problem for me?
@Hironsan ,
dataset (CSV):
```
text,label,metadata_ключ
"a text","a_label","metadata_значение"
```
result:
![image](https://user-images.githubusercontent.com/1063219/73249542-23e8a280-41c6-11ea-9d58-07f1037e269b.png)
Thank you @kuraga for the information.
It's really helpful.

# Analysis

The issue is caused by `json.dumps` function in `FileParser`:
https://github.com/doccano/doccano/blob/1745821b2e484dc35b79959473a68070c8b5e3ae/app/api/utils.py#L361

When we upload a file, `FileParser` parses it and changes metadata to JSON string by `json.dumps`. By default, it escapes Unicode strings due to `ensure_ascii=True` argument:

```python
>>> import json
>>> json.dumps('metadata_значение')
'"metadata_\\u0437\\u043d\\u0430\\u0447\\u0435\\u043d\\u0438\\u0435"'
```

# Quick-fix solution

Use `ensure_ascii=False`:

```python
>>> json.dumps('metadata_значение', ensure_ascii=False)
'"metadata_значение"'
```

![image](https://user-images.githubusercontent.com/6737785/73256858-0c310e80-4206-11ea-91d5-1896f6102e15.png)

@c-w where to fix this in your mind? I'll try.
@kuraga This looks to be a bug with the v1 UI with which I haven't yet familiarized myself (for my use-cases, I'm sticking with the v0 UI for now). @Hironsan can perhaps provide pointers for the v1 UI.
Hello @kuraga,

I can display non-ascii characters:
![image](https://user-images.githubusercontent.com/6737785/73226900-fc3d0e80-41b4-11ea-878b-5026affea28b.png)

Could you please tell me an example of metadata to analyze the problem for me?
@Hironsan ,
dataset (CSV):
```
text,label,metadata_ключ
"a text","a_label","metadata_значение"
```
result:
![image](https://user-images.githubusercontent.com/1063219/73249542-23e8a280-41c6-11ea-9d58-07f1037e269b.png)
Thank you @kuraga for the information.
It's really helpful.

# Analysis

The issue is caused by `json.dumps` function in `FileParser`:
https://github.com/doccano/doccano/blob/1745821b2e484dc35b79959473a68070c8b5e3ae/app/api/utils.py#L361

When we upload a file, `FileParser` parses it and changes metadata to JSON string by `json.dumps`. By default, it escapes Unicode strings due to `ensure_ascii=True` argument:

```python
>>> import json
>>> json.dumps('metadata_значение')
'"metadata_\\u0437\\u043d\\u0430\\u0447\\u0435\\u043d\\u0438\\u0435"'
```

# Quick-fix solution

Use `ensure_ascii=False`:

```python
>>> json.dumps('metadata_значение', ensure_ascii=False)
'"metadata_значение"'
```

![image](https://user-images.githubusercontent.com/6737785/73256858-0c310e80-4206-11ea-91d5-1896f6102e15.png)


Resolving patch:
diff --git a/app/api/utils.py b/app/api/utils.py
--- a/app/api/utils.py
+++ b/app/api/utils.py
@@ -222,6 +222,10 @@ class FileParser(object):
     def parse(self, file):
         raise NotImplementedError()
 
+    @staticmethod
+    def encode_metadata(data):
+        return json.dumps(data, ensure_ascii=False)
+
 
 class CoNLLParser(FileParser):
     """Uploads CoNLL format file.
@@ -358,7 +362,7 @@ def parse_excel_csv_reader(reader):
             elif len(row) == len(columns) and len(row) >= 2:
                 datum = dict(zip(columns, row))
                 text, label = datum.pop('text'), datum.pop('label')
-                meta = json.dumps(datum)
+                meta = FileParser.encode_metadata(datum)
                 j = {'text': text, 'labels': [label], 'meta': meta}
                 data.append(j)
             else:
@@ -379,7 +383,7 @@ def parse(self, file):
                 data = []
             try:
                 j = json.loads(line)
-                j['meta'] = json.dumps(j.get('meta', {}))
+                j['meta'] = FileParser.encode_metadata(j.get('meta', {}))
                 data.append(j)
             except json.decoder.JSONDecodeError:
                 raise FileParseException(line_num=i, line=line)


Test patch: