Repository: mne-tools/mne-python

Issue:
Annotations do not support Unicode characters during I/O roundtrip
### Description of the problem

Raw data (and possibly epochs, which I haven't tested) with Annotations that contain Unicode descriptions cannot be saved.

This was initially reported at https://mne.discourse.group/t/saving-filtered-data-and-epochs/6848/3

### Steps to reproduce

```Python
# %%
import mne

sample_dir = mne.datasets.sample.data_path()
sample_fname = sample_dir / 'MEG' / 'sample' / 'sample_audvis_raw.fif'

raw = mne.io.read_raw_fif(sample_fname, preload=True)
raw.crop(tmax=60)

annots = mne.Annotations(onset=0, duration=1, description='🙃')
raw.set_annotations(annots)

raw.save('/tmp/foo-raw.fif', overwrite=True)
```


### Link to data

_No response_

### Expected results

File is saved

### Actual results

```python
Writing /tmp/foo-raw.fif
Traceback (most recent call last):
  File "/private/tmp/mwe.py", line 13, in <module>
    raw.save('/tmp/foo-raw.fif', overwrite=True)
  File "<decorator-gen-242>", line 12, in save
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 1691, in save
    _write_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2604, in _write_raw
    cals = _start_writing_raw(
  File "/Users/hoechenberger/Development/mne-python/mne/io/base.py", line 2872, in _start_writing_raw
    _write_annotations(fid, annotations)
  File "/Users/hoechenberger/Development/mne-python/mne/annotations.py", line 1084, in _write_annotations
    write_name_list_sanitized(
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 149, in write_name_list_sanitized
    write_string(fid, kind, _safe_name_list(lst, "write", name))
  File "/Users/hoechenberger/Development/mne-python/mne/io/write.py", line 130, in write_string
    str_data = str(data).encode("latin1")
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f643' in position 0: ordinal not in range(256)
```

### Additional information

Since Unicode Annotations can be set, we should be able to write them. Hence I'm reporting this as a bug and not a feature request.

Proposals by @cbrnr, @drammock, and me:

- prevent using non-latin1 characters in Annotations (@hoechenberger) – but this is actually not acceptable in 2023 (@drammock, and I fully agree with that)
- amend FIFF definition to allow Unicode character storage (@cbrnr, @drammock)
- provide HDF5 export for all affected file types (@drammock)
- Maybe we can sneak in the Unicode via an ASCII encoding, think base64, but this sounds rather ugly (@hoechenberger)

Discussion:

Patch:
diff --git a/doc/changes/devel.rst b/doc/changes/devel.rst
index ff753de08..8fc12edc6 100644
--- a/doc/changes/devel.rst
+++ b/doc/changes/devel.rst
@@ -65,6 +65,7 @@ Bugs
 - Fix parsing of eye-link :class:`~mne.Annotations` when ``apply_offsets=False`` is provided to :func:`~mne.io.read_raw_eyelink` (:gh:`12003` by `Mathieu Scheltienne`_)
 - Correctly prune channel-specific :class:`~mne.Annotations` when creating :class:`~mne.Epochs` without the channel(s) included in the channel specific annotations (:gh:`12010` by `Mathieu Scheltienne`_)
 - Fix :func:`~mne.viz.plot_volume_source_estimates` with :class:`~mne.VolSourceEstimate` which include a list of vertices (:gh:`12025` by `Mathieu Scheltienne`_)
+- Add support for non-ASCII characters in Annotations, Evoked comments, etc when saving to FIFF format (:gh:`12080` by `Daniel McCloy`_)
 - Correctly handle passing ``"eyegaze"`` or ``"pupil"`` to :meth:`mne.io.Raw.pick` (:gh:`12019` by `Scott Huberty`_)
 
 API changes
diff --git a/doc/changes/names.inc b/doc/changes/names.inc
index be6ee47ec..722f0c2dc 100644
--- a/doc/changes/names.inc
+++ b/doc/changes/names.inc
@@ -212,6 +212,8 @@
 
 .. _Ilias Machairas: https://github.com/JungleHippo
 
+.. _Ivan Zubarev: https://github.com/zubara
+
 .. _Ivana Kojcic: https://github.com/ikojcic
 
 .. _Jaakko Leppakangas: https://github.com/jaeilepp
diff --git a/mne/_fiff/tag.py b/mne/_fiff/tag.py
index 87ea6ca96..f64d16e1c 100644
--- a/mne/_fiff/tag.py
+++ b/mne/_fiff/tag.py
@@ -4,7 +4,9 @@
 # License: BSD-3-Clause
 
 from functools import partial
+import html
 import struct
+import re
 
 import numpy as np
 from scipy.sparse import csc_matrix, csr_matrix
@@ -265,7 +267,10 @@ def _read_string(fid, tag, shape, rlims):
     """Read a string tag."""
     # Always decode to ISO 8859-1 / latin1 (FIFF standard).
     d = _frombuffer_rows(fid, tag.size, dtype=">c", shape=shape, rlims=rlims)
-    return str(d.tobytes().decode("latin1", "ignore"))
+    string = str(d.tobytes().decode("latin1", "ignore"))
+    if re.search(r"&#[0-9a-fA-F]{6};", string):
+        string = html.unescape(string)
+    return string
 
 
 def _read_complex_float(fid, tag, shape, rlims):
diff --git a/mne/_fiff/write.py b/mne/_fiff/write.py
index c66e3245e..b8ed1d2b1 100644
--- a/mne/_fiff/write.py
+++ b/mne/_fiff/write.py
@@ -128,7 +128,10 @@ def write_julian(fid, kind, data):
 
 def write_string(fid, kind, data):
     """Write a string tag."""
-    str_data = str(data).encode("latin1")
+    try:
+        str_data = str(data).encode("latin1")
+    except UnicodeEncodeError:
+        str_data = str(data).encode("latin1", errors="xmlcharrefreplace")
     data_size = len(str_data)  # therefore compute size here
     my_dtype = ">a"  # py2/3 compatible on writing -- don't ask me why
     if data_size > 0:


Test patch:
diff --git a/mne/tests/test_evoked.py b/mne/tests/test_evoked.py
index 7c45bab7a..040978c9f 100644
--- a/mne/tests/test_evoked.py
+++ b/mne/tests/test_evoked.py
@@ -263,6 +263,12 @@ def test_io_evoked(tmp_path):
     ave_complex = read_evokeds(fname_temp)[0]
     assert_allclose(ave.data, ave_complex.data.imag)
 
+    # test non-ascii comments (gh 11684)
+    aves1[0].comment = "🙃"
+    write_evokeds(tmp_path / "evoked-ave.fif", aves1, overwrite=True)
+    aves1_read = read_evokeds(tmp_path / "evoked-ave.fif")[0]
+    assert aves1_read.comment == aves1[0].comment
+
     # test warnings on bad filenames
     fname2 = tmp_path / "test-bad-name.fif"
     with pytest.warns(RuntimeWarning, match="-ave.fif"):