Repository: mne-tools/mne-python

Issue:
UnicodeDecodeError for new version
### Description of the problem

When I read some .edf files by `read_raw_edf()` function, the codes from Version 0.24.1 succeed to read all the files. However, the codes from Version 1.1.1 succeed to read some files, but report an error while reading one file as:
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc8 in position 34915709: invalid continuation byte

### Steps to reproduce

```Python
self.raw_data = read_raw_edf(raw_data_path, preload=False)[:][0]
where `raw_data_path` includes a .edf file.
```


### Link to data

_No response_

### Expected results

Succeed to read all files.

### Actual results

  File "/home/xxx/anaconda3/lib/python3.7/site-packages/mne/io/edf/edf.py", line 1387, in read_raw_edf
    verbose=verbose)
  File "<decorator-gen-229>", line 12, in \_\_init\_\_
  File "/home/xxx/anaconda3/lib/python3.7/site-packages/mne/io/edf/edf.py", line 158, in \_\_init\_\_
    onset, duration, desc = _read_annotations_edf(tal_data[0])
  File "/home/xxx/anaconda3/lib/python3.7/site-packages/mne/io/edf/edf.py", line 1590, in _read_annotations_edf
    triggers = re.findall(pat, tals.decode('utf8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc8 in position 34915709: invalid continuation byte

### Additional information

Platform:         Linux-4.15.0-188-generic-x86_64-with-debian-buster-sid
Python:           3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0]
Executable:       /home/xxx/anaconda3/bin/python
CPU:              x86_64: 256 cores
Memory:           755.8 GB

mne:              1.1.1
numpy:            1.20.3 {blas=mkl_rt, lapack=mkl_rt}
scipy:            1.7.3
matplotlib:       3.5.1 {backend=agg}

sklearn:          0.24.2
numba:            0.54.1
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           1.3.5
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
qtpy:             2.0.1 {PyQt5=5.9.6}
ipympl:           Not found
pyqtgraph:        Not found
pooch:            v1.6.0

mne_bids:         Not found
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found
mne_icalabel:     Not found


Discussion:
or changelog https://mne.tools/stable/whats_new.html says:


   -

   Annotations contained in EDF files are correctly read as UTF-8 according
   to the EDF specification (#10963
   <https://github.com/mne-tools/mne-python/issues/10963> by Clemens Brunner
   <https://github.com/cbrnr>)

So I would conclude that the EDF file you have does not follow the EDF file
standard but I am not an EDF expert

how were these files written?

Message ID: ***@***.***>
>

I have exported these .edf files from original data by the `Neuro Workbench` software (from `NIHON KOHDEN`). The transformation is executed in the software and I did not use explicit codes to do this.
I am not sure that whether the commercial software still follows old encoding type `latin-1`, which causes the error report.
do you have the problem with other EDF readers?

> Message ID: ***@***.***>
>

According to the [EDF specification](https://www.edfplus.info/specs/edfplus.html#annotation), TAL channels are encoded as UTF-8. Previously, we used ISO 8859-1 (aka Latin-1), which worked with files that are apparently very common in practice, but are not compliant with the standard.

I prefer sticking to the standard UTF-8 encoding, but we could think about introducing a new parameter to let users change the annotation encoding if this is really common. Alternatively, we could also `try`/`except` UTF-8 and Latin-1 encodings, which would neither require a new parameter nor any user input.
I would be ok with a new param to specify the encoding. Maybe
`tal_encoding` or just `encoding`?

> Message ID: ***@***.***>
>

Fine with me. I think `encoding="utf8"` would be a good choice if we document it that this refers to annotation channels (other encodings cannot and should not be changed since e.g. header encoding is limited to ASCII).
works for me.

Can you take care of the PR? thx a lot

> Message ID: ***@***.***>
>

> Can you take care of the PR?

Sure, will do!

Resolving patch:
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -84,6 +84,7 @@ class RawEDF(BaseRaw):
         .. versionadded:: 1.1
     %(preload)s
     %(units_edf_bdf_io)s
+    %(encoding_edf)s
     %(verbose)s
 
     See Also
@@ -133,7 +134,7 @@ class RawEDF(BaseRaw):
     @verbose
     def __init__(self, input_fname, eog=None, misc=None, stim_channel='auto',
                  exclude=(), infer_types=False, preload=False, include=None,
-                 units=None, *, verbose=None):
+                 units=None, encoding='utf8', *, verbose=None):
         logger.info('Extracting EDF parameters from {}...'.format(input_fname))
         input_fname = os.path.abspath(input_fname)
         info, edf_info, orig_units = _get_info(input_fname, stim_channel, eog,
@@ -176,7 +177,10 @@ def __init__(self, input_fname, eog=None, misc=None, stim_channel='auto',
             tal_data = self._read_segment_file(
                 np.empty((0, self.n_times)), idx, 0, 0, int(self.n_times),
                 np.ones((len(idx), 1)), None)
-            onset, duration, desc = _read_annotations_edf(tal_data[0])
+            onset, duration, desc = _read_annotations_edf(
+                tal_data[0],
+                encoding=encoding,
+            )
 
         self.set_annotations(Annotations(onset=onset, duration=duration,
                                          description=desc, orig_time=None))
@@ -1303,7 +1307,7 @@ def _find_tal_idx(ch_names):
 @fill_doc
 def read_raw_edf(input_fname, eog=None, misc=None, stim_channel='auto',
                  exclude=(), infer_types=False, include=None, preload=False,
-                 units=None, *, verbose=None):
+                 units=None, encoding='utf8', *, verbose=None):
     """Reader function for EDF or EDF+ files.
 
     Parameters
@@ -1344,6 +1348,7 @@ def read_raw_edf(input_fname, eog=None, misc=None, stim_channel='auto',
         .. versionadded:: 1.1
     %(preload)s
     %(units_edf_bdf_io)s
+    %(encoding_edf)s
     %(verbose)s
 
     Returns
@@ -1406,13 +1411,13 @@ def read_raw_edf(input_fname, eog=None, misc=None, stim_channel='auto',
     return RawEDF(input_fname=input_fname, eog=eog, misc=misc,
                   stim_channel=stim_channel, exclude=exclude,
                   infer_types=infer_types, preload=preload, include=include,
-                  units=units, verbose=verbose)
+                  units=units, encoding=encoding, verbose=verbose)
 
 
 @fill_doc
 def read_raw_bdf(input_fname, eog=None, misc=None, stim_channel='auto',
                  exclude=(), infer_types=False, include=None, preload=False,
-                 units=None, *, verbose=None):
+                 units=None, encoding='utf8', *, verbose=None):
     """Reader function for BDF files.
 
     Parameters
@@ -1453,6 +1458,7 @@ def read_raw_bdf(input_fname, eog=None, misc=None, stim_channel='auto',
         .. versionadded:: 1.1
     %(preload)s
     %(units_edf_bdf_io)s
+    %(encoding_edf)s
     %(verbose)s
 
     Returns
@@ -1508,7 +1514,7 @@ def read_raw_bdf(input_fname, eog=None, misc=None, stim_channel='auto',
     return RawEDF(input_fname=input_fname, eog=eog, misc=misc,
                   stim_channel=stim_channel, exclude=exclude,
                   infer_types=infer_types, preload=preload, include=include,
-                  units=units, verbose=verbose)
+                  units=units, encoding=encoding, verbose=verbose)
 
 
 @fill_doc
@@ -1568,13 +1574,15 @@ def read_raw_gdf(input_fname, eog=None, misc=None, stim_channel='auto',
                   include=include, verbose=verbose)
 
 
-def _read_annotations_edf(annotations):
+@fill_doc
+def _read_annotations_edf(annotations, encoding='utf8'):
     """Annotation File Reader.
 
     Parameters
     ----------
     annotations : ndarray (n_chans, n_samples) | str
         Channel data in EDF+ TAL format or path to annotation file.
+    %(encoding_edf)s
 
     Returns
     -------
@@ -1589,8 +1597,9 @@ def _read_annotations_edf(annotations):
     """
     pat = '([+-]\\d+\\.?\\d*)(\x15(\\d+\\.?\\d*))?(\x14.*?)\x14\x00'
     if isinstance(annotations, str):
-        with open(annotations, encoding='latin-1') as annot_file:
-            triggers = re.findall(pat, annot_file.read())
+        with open(annotations, "rb") as annot_file:
+            triggers = re.findall(pat.encode(), annot_file.read())
+            triggers = [tuple(map(lambda x: x.decode(), t)) for t in triggers]
     else:
         tals = bytearray()
         annotations = np.atleast_2d(annotations)
@@ -1609,8 +1618,13 @@ def _read_annotations_edf(annotations):
                 # Exploit np vectorized processing
                 tals.extend(np.uint8([this_chan % 256, this_chan // 256])
                             .flatten('F'))
-
-        triggers = re.findall(pat, tals.decode('utf8'))
+        try:
+            triggers = re.findall(pat, tals.decode(encoding))
+        except UnicodeDecodeError as e:
+            raise Exception(
+                "Encountered invalid byte in at least one annotations channel."
+                " You might want to try setting \"encoding='latin1'\"."
+            ) from e
 
     events = []
     offset = 0.
@@ -1634,11 +1648,6 @@ def _read_annotations_edf(annotations):
     return zip(*events) if events else (list(), list(), list())
 
 
-def _get_edf_default_event_id(descriptions):
-    mapping = {a: n for n, a in enumerate(sorted(set(descriptions)), start=1)}
-    return mapping
-
-
 def _get_annotations_gdf(edf_info, sfreq):
     onset, duration, desc = list(), list(), list()
     events = edf_info.get('events', None)
diff --git a/mne/utils/docs.py b/mne/utils/docs.py
--- a/mne/utils/docs.py
+++ b/mne/utils/docs.py
@@ -951,6 +951,12 @@ def _reflow_param_docstring(docstring, has_first_line=True, width=75):
     Whether to emit warnings when cropping or omitting annotations.
 """
 
+docdict['encoding_edf'] = """
+encoding : str
+    Encoding of annotations channel(s). Default is "utf8" (the only correct
+    encoding according to the EDF+ standard).
+"""
+
 docdict['epochs_preload'] = """
     Load all epochs from disk when creating the object
     or wait before accessing each epoch (more memory


Test patch:
diff --git a/mne/io/edf/tests/test_edf.py b/mne/io/edf/tests/test_edf.py
--- a/mne/io/edf/tests/test_edf.py
+++ b/mne/io/edf/tests/test_edf.py
@@ -26,7 +26,7 @@
 from mne.utils import requires_pandas, _record_warnings
 from mne.io import read_raw_edf, read_raw_bdf, read_raw_fif, edf, read_raw_gdf
 from mne.io.tests.test_raw import _test_raw_reader
-from mne.io.edf.edf import (_get_edf_default_event_id, _read_annotations_edf,
+from mne.io.edf.edf import (_read_annotations_edf,
                             _read_ch, _parse_prefilter_string, _edf_str,
                             _read_edf_header, _read_header)
 from mne.io.pick import channel_indices_by_type, get_channel_type_constants
@@ -240,7 +240,12 @@ def test_find_events_backward_compatibility():
                        [1280, 0, 2]]
     # test an actual file
     raw = read_raw_edf(edf_path, preload=True)
-    event_id = _get_edf_default_event_id(raw.annotations.description)
+    event_id = {
+        a: n
+        for n, a in enumerate(
+            sorted(set(raw.annotations.description)), start=1
+        )
+    }
     event_id.pop('start')
     events_from_EFA, _ = events_from_annotations(raw, event_id=event_id,
                                                  use_rounding=False)
@@ -359,6 +364,46 @@ def test_read_utf8_annotations():
     assert raw.annotations[1]['description'] == '仰卧'
 
 
+def test_read_latin1_annotations(tmp_path):
+    """Test if annotations encoded as Latin-1 can be read.
+
+    Note that the correct encoding according to the EDF+ standard should be
+    UTF8, but many real-world files are saved with the Latin-1 encoding.
+    """
+    annot = (
+        b"+1.1\x14\xe9\x14\x00\x00"  # +1.1 é
+        b"+1.2\x14\xe0\x14\x00\x00"  # +1.2 à
+        b"+1.3\x14\xe8\x14\x00\x00"  # +1.3 è
+        b"+1.4\x14\xf9\x14\x00\x00"  # +1.4 ù
+        b"+1.5\x14\xe2\x14\x00\x00"  # +1.5 â
+        b"+1.6\x14\xea\x14\x00\x00"  # +1.6 ê
+        b"+1.7\x14\xee\x14\x00\x00"  # +1.7 î
+        b"+1.8\x14\xf4\x14\x00\x00"  # +1.8 ô
+        b"+1.9\x14\xfb\x14\x00\x00"  # +1.9 û
+    )
+    annot_file = tmp_path / "annotations.txt"
+    with open(annot_file, "wb") as f:
+        f.write(annot)
+    with open(annot_file, "rb") as f:
+        tal_channel = _read_ch(
+            f,
+            subtype='EDF',
+            dtype='<i2',
+            samp=-1,
+            dtype_byte=None,
+        )
+    onset, duration, description = _read_annotations_edf(
+        tal_channel,
+        encoding="latin1",
+    )
+    assert onset == (1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9)
+    assert not any(duration)  # all durations are 0
+    assert description == ("é", "à", "è", "ù", "â", "ê", "î", "ô", "û")
+
+    with pytest.raises(Exception, match="Encountered invalid byte in"):
+        _read_annotations_edf(tal_channel)  # default encoding="utf8" fails
+
+
 def test_edf_prefilter_parse():
     """Test prefilter strings from header are parsed correctly."""
     prefilter_basic = ["HP: 0Hz LP: 0Hz"]