Repository: mne-tools/mne-python

Issue:
my event id dict is garbage characters
**READ THIS FIRST:** If you are having trouble getting MNE-Python to work with
your own data, you should ask for help on the
[MNE Forum](https://mne.discourse.group).

Our GitHub issue tracker is only used to report bugs and suggest improvements
to MNE-Python. For any other questions, please use the forum.
Usage questions that are posted as GitHub issues are usually closed without
being answered. See
[the FAQ entry on filing bug reports](https://mne.tools/dev/overview/faq.html#i-think-i-found-a-bug-what-do-i-do)
for more guidance. If you're pretty sure your problem is a bug in MNE-Python,
please **delete this section** and fill in the headings below, replacing the
placeholder text with your own responses. Surround any code samples with triple
backticks above and below the code block (see
[the GitHub guide to markdown](https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown)
for help with issue formatting). Alternatively, you can put your MWE in a
[public gist](https://gist.github.com) and link to it in this issue.


#### Describe the bug
*Replace this text with a description of the bug.*


#### Steps to reproduce
*Replace this text with a code snippet or minimal working example [MWE] to
replicate your problem, using one of the [built-in datasets], preferably the
one called [sample]. If you can't replicate on a built-in dataset, provide also
a link to a small, anonymized portion of your data that does yield the error.*

[MWE]: https://en.wikipedia.org/wiki/Minimal_Working_Example
[built-in datasets]: https://mne.tools/dev/overview/datasets_index.html
[sample]: https://mne.tools/dev/overview/datasets_index.html#sample


#### Expected results
*Replace this text with a description of what you expected to happen.*


#### Actual results
*Replace this text with the actual output, traceback, screenshot, or other
description of the results.*


#### Additional information
*Replace this text with the output of `mne.sys_info()`.*



Discussion:
Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

my code show :
```
def get_data(path):

    rawEEG = read_raw_edf(path, preload=False)
	epochs_data, event_dict = mne.events_from_annotations(rawEEG)
    print(event_dict)
```

and I I have a lot of EDF files,When I use the code above to open a file,my event id dict show:

```
Used Annotations descriptions: [' Bad Signal', 'äº§ç\x94\x9fè§\x89é\x86\x92', 'ä»°å\x8d§', 'å\x81\x9cæ\xad¢ PSG', 'å\x8f³', 'å\x8f³ä¸\x8a', 'å\x90¯å\x8a¨ PSG', 'å·¦', 'æ\x9b´æ\x94¹', 'æ°§å\x87\x8fé¥±å\x92\x8c', 'ç\x9d¡ç\x9c\xa0å\x88\x86æ\x9c\x9f, ä½\x93ä½\x8d, æ°§å\x87\x8fé¥±å\x92\x8c, å\x91¼å\x90¸æ\x9a\x82å\x81\x9c', 'é\x98»å¡\x9eå\x9e\x8bå\x91¼å\x90¸ä¸\x8dè¶³', 'é\x98»å¡\x9eå\x9e\x8bå\x91¼å\x90¸æ\x9a\x82å\x81\x9c', 'é\x98»æ\x8a\x97']
```

and then, i use [EDFbrowser](https://www.teuniz.net/edfbrowser/) to open this file:
![image](https://user-images.githubusercontent.com/20573808/180977963-f93b09ff-48be-4091-8d62-bf46809c2972.png)
this characters  is normal.


but when i open other file,show me result:
```
{'Ambulatory Start Recording': 1, 'Ambulatory Started.': 2, 'Auto-Staging Montage Error': 3, 'Breath Hold (10 seconds)': 4, 'Close Eyes (30 seconds)': 5, 'EEG arousal': 6, 'Eye Blinks X5': 7, 'Eyes Left and Right X5': 8, 'Eyes Up and Down X5': 9, 'Flex Left Foot': 10, 'Flex Right Foot': 11, 'Gain/Filter Change': 12, 'Grit Teeth or Chew (5 seconds)': 13, 'Lights Off': 14, 'Lights On': 15, 'Montage:TREX - Sleep, Ref': 16, 'Nasal Breathing (10 seconds)': 17, 'Normal Breathing': 18, 'Open Eyes (30 seconds)': 19, 'Oral Breathing (10 seconds)': 20, 'Recording Analyzer - ECG': 21, 'Recording Analyzer - QEEG': 22, 'Simulate Snore or Hum (5 seconds)': 23, 'Sleep stage N1': 24, 'Sleep stage N2': 25, 'Sleep stage N3': 26, 'Sleep stage R': 27, 'Sleep stage W': 28, 'Start Recording': 29, 'Started Analyzer - Sleep Events': 30, 'Stopped Analyzer - Sleep Events': 31}
```

now,I have about 1000+ EDF files, but a lot of them show garbled characters, is this the error of the original file?

Can you avoid this kind of mistake
I think we are using a character encoding scheme which does not support international characters. This might be a bug, because the [EDF specification](https://www.edfplus.info/specs/edfplus.html#annotation) mentions that annotations can contain any UTF8 character.
Can you try changing `'latin-1'` to `'utf8'` [here](https://github.com/cbrnr/mne-python/blob/bf7fd4755a95ad43aa7c2dd07b1ed12656f85292/mne/io/edf/edf.py#L1593)? Note that there is a comment stating that decoding to UTF leads to errors sometimes.
yes,my changed code looks like this:
```
# use of latin-1 because characters are only encoded for the first 256
# code points and utf-8 can triggers an "invalid continuation byte"
# error
# triggers = re.findall(pat, tals.decode('latin-1'))
triggers = re.findall(pat, tals.decode('utf8'))
```
and result show:
```
{' Bad Signal': 1, '产生觉醒': 2, '仰卧': 3, '停止 PSG': 4, '右': 5, '右上': 6, '启动 PSG': 7, '左': 8, '更改': 9, '氧减饱和': 10, '睡眠分期, 体位, 氧减饱和, 呼吸暂停': 11, '阻塞型呼吸不足': 12, '阻塞型呼吸暂停': 13, '阻抗': 14}
```

Now that the bug has been identified, can we add a parameter to change the coding directly for `read_RAW_EDF` or `events_from_annotations`

Resolving patch:
diff --git a/mne/datasets/config.py b/mne/datasets/config.py
--- a/mne/datasets/config.py
+++ b/mne/datasets/config.py
@@ -87,7 +87,7 @@
 # respective repos, and make a new release of the dataset on GitHub. Then
 # update the checksum in the MNE_DATASETS dict below, and change version
 # here:                  ↓↓↓↓↓         ↓↓↓
-RELEASES = dict(testing='0.137', misc='0.23')
+RELEASES = dict(testing='0.138', misc='0.23')
 TESTING_VERSIONED = f'mne-testing-data-{RELEASES["testing"]}'
 MISC_VERSIONED = f'mne-misc-data-{RELEASES["misc"]}'
 
@@ -111,7 +111,7 @@
 # Testing and misc are at the top as they're updated most often
 MNE_DATASETS['testing'] = dict(
     archive_name=f'{TESTING_VERSIONED}.tar.gz',
-    hash='md5:134ef384e7b305ebc5785c7f6ef7c637',
+    hash='md5:9aaaef3481a92cbdb04a9ada8ddfe1cd',
     url=('https://codeload.github.com/mne-tools/mne-testing-data/'
          f'tar.gz/{RELEASES["testing"]}'),
     # In case we ever have to resort to osf.io again...
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -1587,10 +1587,7 @@ def _read_annotations_edf(annotations):
                 tals.extend(np.uint8([this_chan % 256, this_chan // 256])
                             .flatten('F'))
 
-        # use of latin-1 because characters are only encoded for the first 256
-        # code points and utf-8 can triggers an "invalid continuation byte"
-        # error
-        triggers = re.findall(pat, tals.decode('latin-1'))
+        triggers = re.findall(pat, tals.decode('utf8'))
 
     events = []
     offset = 0.


Test patch:
diff --git a/mne/io/edf/tests/test_edf.py b/mne/io/edf/tests/test_edf.py
--- a/mne/io/edf/tests/test_edf.py
+++ b/mne/io/edf/tests/test_edf.py
@@ -61,6 +61,7 @@
 test_generator_edf = op.join(data_path, 'EDF', 'test_generator_2.edf')
 edf_annot_sub_s_path = op.join(data_path, 'EDF', 'subsecond_starttime.edf')
 edf_chtypes_path = op.join(data_path, 'EDF', 'chtypes_edf.edf')
+edf_utf8_annotations = op.join(data_path, 'EDF', 'test_utf8_annotations.edf')
 
 eog = ['REOG', 'LEOG', 'IEOG']
 misc = ['EXG1', 'EXG5', 'EXG8', 'M1', 'M2']
@@ -342,6 +343,14 @@ def test_read_annotations(fname, recwarn):
     assert len(annot.onset) == 2
 
 
+@testing.requires_testing_data
+def test_read_utf8_annotations():
+    """Test if UTF8 annotations can be read."""
+    raw = read_raw_edf(edf_utf8_annotations)
+    assert raw.annotations[0]['description'] == 'RECORD START'
+    assert raw.annotations[1]['description'] == '仰卧'
+
+
 def test_edf_prefilter_parse():
     """Test prefilter strings from header are parsed correctly."""
     prefilter_basic = ["HP: 0Hz LP: 0Hz"]