Repository: mne-tools/mne-python

Issue:
Annotations do not support Unicode characters during I/O roundtrip
### Description of the problem

Raw data (and possibly epochs, which I haven't tested) with Annotations that contain Unicode descriptions cannot be saved.

This was initially reported at https://mne.discourse.group/t/saving-filtered-data-and-epochs/6848/3

### Steps to reproduce

```Python
# %%
import mne

sample_dir = mne.datasets.sample.data_path()
sample_fname = sample_dir / 'MEG' / 'sample' / 'sample_audvis_raw.fif'

raw = mne.io.read_raw_fif(sample_fname, preload=True)
raw.crop(tmax=60)

annots = mne.Annotations(onset=0, duration=1, description='🙃')
raw.set_annotations(annots)

raw.save('/tmp/foo-raw.fif', overwrite=True)
```


### Link to data

_No response_

### Expected results

File is saved

### Actual results

```python
Writing /tmp/foo-raw.fif
Traceback (most recent call last):
  File "/private/tmp/mwe.py", line 13, in <module>
    raw.save('/tmp/foo-raw.fif', overwrite=True)
  File "<decorator-gen-242>", line 12, in save
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 1691, in save
    _write_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2604, in _write_raw
    cals = _start_writing_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2872, in _start_writing_raw
    _write_annotations(fid, annotations)
  File "/Users/hoechenberger/Development/mne-python/mne/annotations.py", line 1084, in _write_annotations
    write_name_list_sanitized(
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 149, in write_name_list_sanitized
    write_string(fid, kind, _safe_name_list(lst, "write", name))
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 130, in write_string
    str_data = str(data).encode("latin1")
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f643' in position 0: ordinal not in range(256)
```

### Additional information

Since Unicode Annotations can be set, we should be able to write them. Hence I'm reporting this as a bug and not a feature request.

Proposals by @cbrnr, @drammock, and me:

- prevent using non-latin1 characters in Annotations (@hoechenberger) – but this is actually not acceptable in 2023 (@drammock, and I fully agree with that)
- amend FIFF definition to allow Unicode character storage (@cbrnr, @drammock)
- provide HDF5 export for all affected file types (@drammock)
- Maybe we can sneak in the Unicode via an ASCII encoding, think base64, but this sounds rather ugly (@hoechenberger)


Discussion:
This problem is not unique to annotations and is a property of any FIF string at the moment:
```
>>> ave = mne.read_evokeds(mne.datasets.sample.data_path() / "MEG" / "sample" / "sample_audvis-ave.fif")[0]
>>> ave.comment
'Left Auditory'
>>> ave.comment = "Left 🙃"
>>> ave.save("test-ave.fif")
Traceback (most recent call last):
...
  File "/Users/larsoner/python/mne-python/mne/evoked.py", line 1902, in _write_evokeds
    write_string(fid, FIFF.FIFF_COMMENT, e.comment)
  File "/Users/larsoner/python/mne-python/mne/_fiff/write.py", line 131, in write_string
    str_data = str(data).encode("latin1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f643' in position 5: ordinal not in range(256)
```
I don't think it will be trivial to add support to FIF and would also likely break any other software that works with FIF to add such support. Hence I think we should go with option (1) for now and maybe someday implement (2) across the FIF string read/write routines.
... also `latin-1` is not a subset of `utf-8` so backward compat here would be nearly impossible it seems -- writing utf-8 then reading with latin-1 software (MEGIN or old MNE or MNE-MATLAB, etc.) can get garbled stuff, and vice-versa (e.g., https://github.com/mne-tools/fiff-constants/commit/82aaf647c91fd2565f9a868065b1e9003c4279e4 rendering funny on GitHub by latin-1 encoding the superscript 3). So a compatible path forward is not trivial :(
What about this workaround:

> * provide HDF5 export for all affected file types (@drammock)

We've resorted to using HDF5 to overcome FIFF limitations in several other places, too.

When the user sets non-latin-1 strings, we can emit a warning that from now on, only HDF5 is allowed for an I/O round-trip, and raise an exception if they try save as FIFF. Or – raise another warning in that case, and force-convert the strings to latin-1 as good as we can.

To me adding support for writing annot to HDF5 isn't really desirable because you still can't do `raw.set_annotations(annot); raw.save(...)` and have it work. So it's weird to have this partial support. And having `raw.save('...-raw.h5')` doesn't make sense because it's not portable to other software, which I think is important for raw data saving. (HDF5 in principle is readable everywhere but it's not a known/defined format for M/EEG data import in any other package.)

Resolving patch:
diff --git a/mne/_fiff/tag.py b/mne/_fiff/tag.py
--- a/mne/_fiff/tag.py
+++ b/mne/_fiff/tag.py
@@ -4,7 +4,9 @@
 # License: BSD-3-Clause
 
 from functools import partial
+import html
 import struct
+import re
 
 import numpy as np
 from scipy.sparse import csc_matrix, csr_matrix
@@ -265,7 +267,10 @@ def _read_string(fid, tag, shape, rlims):
     """Read a string tag."""
     # Always decode to ISO 8859-1 / latin1 (FIFF standard).
     d = _frombuffer_rows(fid, tag.size, dtype=">c", shape=shape, rlims=rlims)
-    return str(d.tobytes().decode("latin1", "ignore"))
+    string = str(d.tobytes().decode("latin1", "ignore"))
+    if re.search(r"&#[0-9a-fA-F]{6};", string):
+        string = html.unescape(string)
+    return string
 
 
 def _read_complex_float(fid, tag, shape, rlims):
diff --git a/mne/_fiff/write.py b/mne/_fiff/write.py
--- a/mne/_fiff/write.py
+++ b/mne/_fiff/write.py
@@ -128,7 +128,10 @@ def write_julian(fid, kind, data):
 
 def write_string(fid, kind, data):
     """Write a string tag."""
-    str_data = str(data).encode("latin1")
+    try:
+        str_data = str(data).encode("latin1")
+    except UnicodeEncodeError:
+        str_data = str(data).encode("latin1", errors="xmlcharrefreplace")
     data_size = len(str_data)  # therefore compute size here
     my_dtype = ">a"  # py2/3 compatible on writing -- don't ask me why
     if data_size > 0:


Test patch:
diff --git a/mne/tests/test_evoked.py b/mne/tests/test_evoked.py
--- a/mne/tests/test_evoked.py
+++ b/mne/tests/test_evoked.py
@@ -263,6 +263,12 @@ def test_io_evoked(tmp_path):
     ave_complex = read_evokeds(fname_temp)[0]
     assert_allclose(ave.data, ave_complex.data.imag)
 
+    # test non-ascii comments (gh 11684)
+    aves1[0].comment = "🙃"
+    write_evokeds(tmp_path / "evoked-ave.fif", aves1, overwrite=True)
+    aves1_read = read_evokeds(tmp_path / "evoked-ave.fif")[0]
+    assert aves1_read.comment == aves1[0].comment
+
     # test warnings on bad filenames
     fname2 = tmp_path / "test-bad-name.fif"
     with pytest.warns(RuntimeWarning, match="-ave.fif"):