Repository: mne-tools/mne-python

Issue:
Can't load raw nirx data, due to encoding issue
When I try to load NirX data with the `mne.io.read_raw_nirx` function, I'm facing encoding issue with the `.hdr` file. For now, I'm solving this manually by converting the `.hdr` to `utf8` with a simple text file editor like `geany`.

#### MWE

I recorded a test dataset (in a `test_github` folder), which is composed of the following files :
```bash
./NIRS-2020-02-13_001.evt  --  inode/x-empty; charset=binary
./NIRS-2020-02-13_001.set  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001.wl1  --  text/plain; charset=us-ascii
./Standard_probeInfo.mat  --  application/octet-stream; charset=binary
./NIRS-2020-02-13_001.dat  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001.tpl  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001_config.txt  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001.wl2  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001.hdr  --  application/x-wine-extension-ini; charset=iso-8859-1
./NIRS-2020-02-13_001.inf  --  text/plain; charset=us-ascii
./NIRS-2020-02-13_001.avg  --  application/octet-stream; charset=binary
```
These informations were obtained with the following command (linux) : 
```bash
for f in `find | egrep -v Eliminate`; do echo "$f" ' -- ' `file -bi "$f"` ; done
```
On python side, here is the code that I intend to use to load the data. I'm using the last version of MNE (0.20.dev0).
```python
#!/usr/bin/env python3
import os
import mne

path = os.sep.join([os.getcwd(), 'test_github'])
raw_intensity = mne.io.read_raw_nirx(path, verbose=True).load_data()
```
As mentioned [here](https://mne.tools/dev/auto_tutorials/preprocessing/plot_70_fnirs_processing.html?highlight=nirs), I should obtain this kind of output :
```python
Loading /home/circleci/mne_data/MNE-fNIRS-motor-data/Participant-1
Reading 0 ... 23238  =      0.000 ...  2974.464 secs...
```
But actually, the `read_raw_nirx` command raises the following Traceback :
```python


Loading /home/bvaler01/Documents/programmes/NBack/test_github
Traceback (most recent call last):
  File "processing_github.py", line 6, in <module>
    raw_intensity = mne.io.read_raw_nirx(path, verbose=True).load_data()
  File "/home/bvaler01/.local/lib/python3.7/site-packages/mne/io/nirx/nirx.py", line 39, in read_raw_nirx
    return RawNIRX(fname, preload, verbose)
  File "</home/bvaler01/.local/lib/python3.7/site-packages/mne/externals/decorator.py:decorator-gen-198>", line 2, in __init__
  File "/home/bvaler01/.local/lib/python3.7/site-packages/mne/utils/_logging.py", line 89, in wrapper
    return function(*args, **kwargs)
  File "/home/bvaler01/.local/lib/python3.7/site-packages/mne/io/nirx/nirx.py", line 113, in __init__
    hdr_str = f.read()
  File "/usr/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 62: invalid continuation byte
```
The test dataset is available [here](http://s000.tinyupload.com/download.php?file_id=15443728717331148895&t=1544372871733114889579038) (`.zip`). It is a ten-seconds recording, from `NirStar 15.2`. It should contain noise only (headless record).


Discussion:
Do the `.hdr` files have a fixed encoding (i.e. are these files always encoded using `iso-8859-1`)? If so, `open` in https://github.com/mne-tools/mne-python/blob/master/mne/io/nirx/nirx.py#L112 should use the `encoding='latin1'` argument.
All I can say is that I could not found any way to change the encoding of the `.hdr` file in the NirStar preferences. But it should be confirmed by other users. The NirsStar documentation does not say anything about it.
Yeah, that's not unexpected. On which platform do you record? Is it a Windows machine? Are other platforms (macOS, Linux) also supported by the recording software? I could imagine that they just use the (default) platform encoding, which is UTF8 on macOS and Linux, and Latin-1 on Windows. If their encoding differs from platform to platform (or even depends on the actual locale), then it won't be easily possible to find a simple solution. One possible option is to make an educated guess with https://pypi.org/project/chardet/ to choose a suitable encoding when opening the file.
I did the recording on a Windows platform. It seems that there is no alternative (MacOS, linux) NirStar distribution. What seems strange to me, is that *only* the `.hdr` is encoded in `latin1`. What about the others: they have been created from a windows platform too (?).
The other text files are ASCII-encoded, which is a subset of Latin-1. This just means that they don't contain any special characters because the first 128 character encodings are identical in ASCII and Latin-1.
So we have to ensure what format encoding is used by NirStar 15.2. Other users inputs would be great. If format encoding is varying from one version to another (15.0 *vs* 15.2, or windows 7 *vs* windows 10 ?), then, only a `chardet` solution would be reasonable. Am I wrong ?
The safest solution would be to use `chardet` to infer the encoding. However, we have been simply assuming `latin-1` in other functions, so we could as well do it here. It will fix your problem, and will likely work for most other cases. If someone encounters more problems, we can always choose the safer solution then.
Agreed let's just go with `latin-1` and if it turns out to be problematic we'll do something smarter

Resolving patch:
diff --git a/mne/io/nirx/nirx.py b/mne/io/nirx/nirx.py
--- a/mne/io/nirx/nirx.py
+++ b/mne/io/nirx/nirx.py
@@ -39,6 +39,10 @@ def read_raw_nirx(fname, preload=False, verbose=None):
     return RawNIRX(fname, preload, verbose)
 
 
+def _open(fname):
+    return open(fname, 'r', encoding='latin-1')
+
+
 @fill_doc
 class RawNIRX(BaseRaw):
     """Raw object from a NIRX fNIRS file.
@@ -74,7 +78,7 @@ def __init__(self, fname, preload=False, verbose=None):
 
         # Read number of rows/samples of wavelength data
         last_sample = -1
-        for line in open(files['wl1']):
+        for line in _open(files['wl1']):
             last_sample += 1
 
         # Read participant information file
@@ -109,7 +113,7 @@ def __init__(self, fname, preload=False, verbose=None):
         # Read header file
         # The header file isn't compliant with the configparser. So all the
         # text between comments must be removed before passing to parser
-        with open(files['hdr']) as f:
+        with _open(files['hdr']) as f:
             hdr_str = f.read()
         hdr_str = re.sub('#.*?#', '', hdr_str, flags=re.DOTALL)
         hdr = RawConfigParser()
@@ -243,7 +247,8 @@ def prepend(list, str):
             raw_extras=[raw_extras], verbose=verbose)
 
         # Read triggers from event file
-        t = [re.findall(r'(\d+)', line) for line in open(files['evt'])]
+        with _open(files['evt']) as fid:
+            t = [re.findall(r'(\d+)', line) for line in fid]
         onset = np.zeros(len(t), float)
         duration = np.zeros(len(t), float)
         description = [''] * len(t)
@@ -290,7 +295,7 @@ def _read_csv_rows_cols(fname, start, stop, cols, n_cols):
     # reading should be done in C (CPython), as should the conversion to float
     # (NumPy).
     x = np.zeros((stop - start, n_cols))
-    with open(fname, 'r') as fid:
+    with _open(fname) as fid:
         for li, line in enumerate(fid):
             if li >= start:
                 if li >= stop:


Test patch:
diff --git a/mne/io/nirx/tests/test_nirx.py b/mne/io/nirx/tests/test_nirx.py
--- a/mne/io/nirx/tests/test_nirx.py
+++ b/mne/io/nirx/tests/test_nirx.py
@@ -4,6 +4,7 @@
 #          simplified BSD-3 license
 
 import os.path as op
+import shutil
 
 import pytest
 from numpy.testing import assert_allclose, assert_array_equal
@@ -113,6 +114,23 @@ def test_nirx_15_2_short():
         mni_locs[21], [0.0388, -0.0477, 0.0932], atol=allowed_dist_error)
 
 
+@requires_testing_data
+def test_encoding(tmpdir):
+    """Test NIRx encoding."""
+    fname = str(tmpdir.join('latin'))
+    shutil.copytree(fname_nirx_15_2, fname)
+    hdr_fname = op.join(fname, 'NIRS-2019-10-02_003.hdr')
+    hdr = list()
+    with open(hdr_fname, 'rb') as fid:
+        hdr.extend(line for line in fid)
+    hdr[2] = b'Date="jeu. 13 f\xe9vr. 2020"\r\n'
+    with open(hdr_fname, 'wb') as fid:
+        for line in hdr:
+            fid.write(line)
+    # smoke test
+    read_raw_nirx(fname)
+
+
 @requires_testing_data
 def test_nirx_15_2():
     """Test reading NIRX files."""