Repository: mne-tools/mne-python

Issue:
add support for NIRStar 14.2 files
#### Describe the new feature or enhancement
*Please provide a clear and concise description of what you want to add or
change.*

I am trying to import my fNIRS data with
mne.io.read_raw_nirx("/Users/name/Downloads/PROTOCOLLO_TDCS_FILE_GREZZI/SHAM/CONTROLLI/2018-11-27_002/")
but I have this error message

Loading /Users/name/Downloads/PROTOCOLLO_TDCS_FILE_GREZZI/SHAM/CONTROLLI/2018-11-27_002/
<ipython-input-8-14238638d873>:1: RuntimeWarning: Only import of data from NIRScout devices have been thoroughly tested. You are using a "NIRSport 8x8" device. 
  mne.io.read_raw_nirx("/Users/name/Downloads/PROTOCOLLO_TDCS_FILE_GREZZI/SHAM/CONTROLLI/2018-11-27_002/")
<ipython-input-8-14238638d873>:1: RuntimeWarning: Extraction of measurement date from NIRX file failed. This can be caused by files saved in certain locales. Please report this as a github issue. The date is being set to January 1st, 2000, instead of "mar 27 nov 2018""18:13:22.559"
  mne.io.read_raw_nirx("/Users/name/Downloads/PROTOCOLLO_TDCS_FILE_GREZZI/SHAM/CONTROLLI/2018-11-27_002/")
Out[8]: <RawNIRX | 2018-11-27_002, 40 x 1934 (247.4 s), ~54 kB, data not loaded>

We used a continuous wave NIRS system (NIRSport 8X8, Nirx Medical Technologies LLC, Berlin, Germany). The fNIRS data acquisition software was the NIRStar 14.2 (Version 14, Revision 2, Release Build, 2016-04-15 NIRx Medizintechnik GmbH, Berlin, Germany; www.nirx.net).

#### Describe your proposed implementation
*Describe how you think the feature or improvement should be implemented (e.g., 
as a new method on an existing class? as new capability added to an existing
method?) If you're not sure, please delete this section and the next section.*

 Please, could you help me to implement a function that can import my data?

#### Describe possible alternatives
*If you've suggested an implementation above, list here any alternative
implementations you can think of, and brief comments explaining why the chosen
implementation is better.*


#### Additional comments
*Add any other context or screenshots about the feature request here.*



Discussion:
Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

Can you share the problematic file? Or even better, any chance you could record a little 1 sec data file from this system and share that so we could add it to our tests?
cc @rob-luke 
Sure,

Please find enclosed an example of data that gives me that issue
[2020-01-24_SHAM_CTRL_0050.zip](https://github.com/mne-tools/mne-python/files/7738037/2020-01-24_SHAM_CTRL_0050.zip)

I am looking forward to hearing from you
.
Hi @mlrocca,

Thanks for this very clear report. MNE currently only supports reading NIRx data from NIRStar version 15.0 and above. Whereas, you are trying to read data acquired with NIRStar version 14.2. I don't think the file format changed very much, so adding support might not be too much effort. Could you open a PR to add support for older NIRx files?
This might also be related to https://github.com/mne-tools/mne-python/pull/7891 where we had issues handling dates with formats other than Australian English.
Thank you, 

I opened a new issue to add support for older NIRx versions.

If there is also the issue of the dates, is it possible to fix that too?

Best,


Ok thank you,

how long do you think you will take to implement that feature?

Are you going to advice me when it will be implemented?


> Ok thank you,
> 
> how long do you think you will take to implement that feature?

I don't know.  The winter holidays mean many of our developers are not available right now.  

> Are you going to advice me when it will be implemented?

The pull request that fixes the issue will be linked here and will automatically close this issue when it is merged in.  Unless you've disabled it in your GitHub notification settings, you will get a notification when that happens.

Hi,

any update on the implementation of the feature in the object?

Resolving patch:
diff --git a/mne/datasets/config.py b/mne/datasets/config.py
--- a/mne/datasets/config.py
+++ b/mne/datasets/config.py
@@ -87,7 +87,7 @@
 # respective repos, and make a new release of the dataset on GitHub. Then
 # update the checksum in the MNE_DATASETS dict below, and change version
 # here:                  ↓↓↓↓↓         ↓↓↓
-RELEASES = dict(testing='0.128', misc='0.23')
+RELEASES = dict(testing='0.129', misc='0.23')
 TESTING_VERSIONED = f'mne-testing-data-{RELEASES["testing"]}'
 MISC_VERSIONED = f'mne-misc-data-{RELEASES["misc"]}'
 
@@ -111,7 +111,7 @@
 # Testing and misc are at the top as they're updated most often
 MNE_DATASETS['testing'] = dict(
     archive_name=f'{TESTING_VERSIONED}.tar.gz',  # 'mne-testing-data',
-    hash='md5:88c04e31fd496f394fa96fe7cdd70217',
+    hash='md5:6847ad93bc025d2f553112baa25ad2a6',
     url=('https://codeload.github.com/mne-tools/mne-testing-data/'
          f'tar.gz/{RELEASES["testing"]}'),
     folder_name='MNE-testing-data',
diff --git a/mne/io/nirx/_localized_abbr.py b/mne/io/nirx/_localized_abbr.py
new file mode 100644
--- /dev/null
+++ b/mne/io/nirx/_localized_abbr.py
@@ -0,0 +1,60 @@
+"""Localizations for meas_date extraction."""
+# Authors: Eric Larson <larson.eric.d@gmail.com>
+#
+# License: BSD-3-Clause
+
+# This file was generated on 2021/01/31 on an Ubuntu system.
+# When getting "unsupported locale setting" on Ubuntu (e.g., with localepurge),
+# use "sudo locale-gen de_DE" etc. then "sudo update-locale".
+
+"""
+import datetime
+import locale
+print('_localized_abbr = {')
+for loc in ('en_US.utf8', 'de_DE', 'fr_FR', 'it_IT'):
+    print(f'    {repr(loc)}: {{')
+    print('        "month": {', end='')
+    month_abbr = set()
+    for month in range(1, 13):  # Month as locale’s abbreviated name
+        locale.setlocale(locale.LC_TIME, "en_US.utf8")
+        dt = datetime.datetime(year=2000, month=month, day=1)
+        val = dt.strftime("%b").lower()
+        locale.setlocale(locale.LC_TIME, loc)
+        key = dt.strftime("%b").lower()
+        month_abbr.add(key)
+        print(f'{repr(key)}: {repr(val)}, ', end='')
+    print('},  # noqa')
+    print('        "weekday": {', end='')
+    weekday_abbr = set()
+    for day in range(1, 8):  # Weekday as locale’s abbreviated name.
+        locale.setlocale(locale.LC_TIME, "en_US.utf8")
+        dt = datetime.datetime(year=2000, month=1, day=day)
+        val = dt.strftime("%a").lower()
+        locale.setlocale(locale.LC_TIME, loc)
+        key = dt.strftime("%a").lower()
+        assert key not in weekday_abbr, key
+        weekday_abbr.add(key)
+        print(f'{repr(key)}: {repr(val)}, ', end='')
+    print('},  # noqa')
+    print('    },')
+print('}\n')
+"""
+
+_localized_abbr = {
+    'en_US.utf8': {
+        "month": {'jan': 'jan', 'feb': 'feb', 'mar': 'mar', 'apr': 'apr', 'may': 'may', 'jun': 'jun', 'jul': 'jul', 'aug': 'aug', 'sep': 'sep', 'oct': 'oct', 'nov': 'nov', 'dec': 'dec', },  # noqa
+        "weekday": {'sat': 'sat', 'sun': 'sun', 'mon': 'mon', 'tue': 'tue', 'wed': 'wed', 'thu': 'thu', 'fri': 'fri', },  # noqa
+    },
+    'de_DE': {
+        "month": {'jan': 'jan', 'feb': 'feb', 'mär': 'mar', 'apr': 'apr', 'mai': 'may', 'jun': 'jun', 'jul': 'jul', 'aug': 'aug', 'sep': 'sep', 'okt': 'oct', 'nov': 'nov', 'dez': 'dec', },  # noqa
+        "weekday": {'sa': 'sat', 'so': 'sun', 'mo': 'mon', 'di': 'tue', 'mi': 'wed', 'do': 'thu', 'fr': 'fri', },  # noqa
+    },
+    'fr_FR': {
+        "month": {'janv.': 'jan', 'févr.': 'feb', 'mars': 'mar', 'avril': 'apr', 'mai': 'may', 'juin': 'jun', 'juil.': 'jul', 'août': 'aug', 'sept.': 'sep', 'oct.': 'oct', 'nov.': 'nov', 'déc.': 'dec', },  # noqa
+        "weekday": {'sam.': 'sat', 'dim.': 'sun', 'lun.': 'mon', 'mar.': 'tue', 'mer.': 'wed', 'jeu.': 'thu', 'ven.': 'fri', },  # noqa
+    },
+    'it_IT': {
+        "month": {'gen': 'jan', 'feb': 'feb', 'mar': 'mar', 'apr': 'apr', 'mag': 'may', 'giu': 'jun', 'lug': 'jul', 'ago': 'aug', 'set': 'sep', 'ott': 'oct', 'nov': 'nov', 'dic': 'dec', },  # noqa
+        "weekday": {'sab': 'sat', 'dom': 'sun', 'lun': 'mon', 'mar': 'tue', 'mer': 'wed', 'gio': 'thu', 'ven': 'fri', },  # noqa
+    },
+}
diff --git a/mne/io/nirx/nirx.py b/mne/io/nirx/nirx.py
--- a/mne/io/nirx/nirx.py
+++ b/mne/io/nirx/nirx.py
@@ -11,6 +11,7 @@
 
 import numpy as np
 
+from ._localized_abbr import _localized_abbr
 from ..base import BaseRaw
 from ..utils import _mult_cal_one
 from ..constants import FIFF
@@ -185,21 +186,43 @@ def __init__(self, fname, saturated, preload=False, verbose=None):
 
         meas_date = None
         # Several formats have been observed so we try each in turn
-        for dt_code in ['"%a, %b %d, %Y""%H:%M:%S.%f"',
-                        '"%a, %d %b %Y""%H:%M:%S.%f"',
-                        '%Y-%m-%d %H:%M:%S.%f']:
-            try:
-                meas_date = dt.datetime.strptime(datetime_str, dt_code)
-                meas_date = meas_date.replace(tzinfo=dt.timezone.utc)
+        for loc, translations in _localized_abbr.items():
+            do_break = False
+            # So far we are lucky in that all the formats below, if they
+            # include %a (weekday abbr), always come first. Thus we can use
+            # a .split(), replace, and rejoin.
+            loc_datetime_str = datetime_str.split(' ')
+            for key, val in translations['weekday'].items():
+                loc_datetime_str[0] = loc_datetime_str[0].replace(key, val)
+            for ii in range(1, len(loc_datetime_str)):
+                for key, val in translations['month'].items():
+                    loc_datetime_str[ii] = \
+                        loc_datetime_str[ii].replace(key, val)
+            loc_datetime_str = ' '.join(loc_datetime_str)
+            logger.debug(f'Trying {loc} datetime: {loc_datetime_str}')
+            for dt_code in ['"%a, %b %d, %Y""%H:%M:%S.%f"',
+                            '"%a %d %b %Y""%H:%M:%S.%f"',
+                            '"%a, %d %b %Y""%H:%M:%S.%f"',
+                            '%Y-%m-%d %H:%M:%S.%f']:
+                try:
+                    meas_date = dt.datetime.strptime(loc_datetime_str, dt_code)
+                except ValueError:
+                    pass
+                else:
+                    meas_date = meas_date.replace(tzinfo=dt.timezone.utc)
+                    do_break = True
+                    logger.debug(
+                        f'Measurement date language {loc} detected: {dt_code}')
+                    break
+            if do_break:
                 break
-            except ValueError:
-                pass
         if meas_date is None:
             warn("Extraction of measurement date from NIRX file failed. "
-                 "This can be caused by files saved in certain locales. "
+                 "This can be caused by files saved in certain locales "
+                 f"(currently only {list(_localized_abbr)} supported). "
                  "Please report this as a github issue. "
                  "The date is being set to January 1st, 2000, "
-                 "instead of {}".format(datetime_str))
+                 f"instead of {repr(datetime_str)}.")
             meas_date = dt.datetime(2000, 1, 1, 0, 0, 0,
                                     tzinfo=dt.timezone.utc)
 


Test patch:
diff --git a/mne/io/nirx/tests/test_nirx.py b/mne/io/nirx/tests/test_nirx.py
--- a/mne/io/nirx/tests/test_nirx.py
+++ b/mne/io/nirx/tests/test_nirx.py
@@ -23,40 +23,45 @@
     short_channels
 from mne.io.constants import FIFF
 
-fname_nirx_15_0 = op.join(data_path(download=False),
-                          'NIRx', 'nirscout', 'nirx_15_0_recording')
-fname_nirx_15_2 = op.join(data_path(download=False),
-                          'NIRx', 'nirscout', 'nirx_15_2_recording')
-fname_nirx_15_2_short = op.join(data_path(download=False),
-                                'NIRx', 'nirscout',
-                                'nirx_15_2_recording_w_short')
-fname_nirx_15_3_short = op.join(data_path(download=False),
-                                'NIRx', 'nirscout', 'nirx_15_3_recording')
+testing_path = data_path(download=False)
+fname_nirx_15_0 = op.join(
+    testing_path, 'NIRx', 'nirscout', 'nirx_15_0_recording')
+fname_nirx_15_2 = op.join(
+    testing_path, 'NIRx', 'nirscout', 'nirx_15_2_recording')
+fname_nirx_15_2_short = op.join(
+    testing_path, 'NIRx', 'nirscout', 'nirx_15_2_recording_w_short')
+fname_nirx_15_3_short = op.join(
+    testing_path, 'NIRx', 'nirscout', 'nirx_15_3_recording')
 
 
 # This file has no saturated sections
-nirsport1_wo_sat = op.join(data_path(download=False), 'NIRx', 'nirsport_v1',
+nirsport1_wo_sat = op.join(testing_path, 'NIRx', 'nirsport_v1',
                            'nirx_15_3_recording_wo_saturation')
 # This file has saturation, but not on the optode pairing in montage
-nirsport1_w_sat = op.join(data_path(download=False), 'NIRx', 'nirsport_v1',
+nirsport1_w_sat = op.join(testing_path, 'NIRx', 'nirsport_v1',
                           'nirx_15_3_recording_w_saturation_'
                           'not_on_montage_channels')
 # This file has saturation in channels of interest
-nirsport1_w_fullsat = op.join(data_path(download=False), 'NIRx', 'nirsport_v1',
-                              'nirx_15_3_recording_w_'
-                              'saturation_on_montage_channels')
+nirsport1_w_fullsat = op.join(
+    testing_path, 'NIRx', 'nirsport_v1', 'nirx_15_3_recording_w_'
+    'saturation_on_montage_channels')
 
 # NIRSport2 device using Aurora software and matching snirf file
-nirsport2 = op.join(data_path(download=False), 'NIRx', 'nirsport_v2',
-                    'aurora_recording _w_short_and_acc')
-nirsport2_snirf = op.join(data_path(download=False), 'SNIRF', 'NIRx',
-                          'NIRSport2', '1.0.3', '2021-05-05_001.snirf')
+nirsport2 = op.join(
+    testing_path, 'NIRx', 'nirsport_v2', 'aurora_recording _w_short_and_acc')
+nirsport2_snirf = op.join(
+    testing_path, 'SNIRF', 'NIRx', 'NIRSport2', '1.0.3',
+    '2021-05-05_001.snirf')
 
-nirsport2_2021_9 = op.join(data_path(download=False), 'NIRx', 'nirsport_v2',
-                           'aurora_2021_9')
-snirf_nirsport2_20219 = op.join(data_path(download=False),
-                                'SNIRF', 'NIRx', 'NIRSport2', '2021.9',
-                                '2021-10-01_002.snirf')
+nirsport2_2021_9 = op.join(
+    testing_path, 'NIRx', 'nirsport_v2', 'aurora_2021_9')
+snirf_nirsport2_20219 = op.join(
+    testing_path, 'SNIRF', 'NIRx', 'NIRSport2', '2021.9',
+    '2021-10-01_002.snirf')
+
+# NIRStar (with Italian locale)
+nirstar_it = op.join(
+    testing_path, 'NIRx', 'nirstar', '2020-01-24_SHAM_CTRL_0050')
 
 
 @requires_h5py
@@ -456,7 +461,7 @@ def test_nirx_15_3_short():
 
 
 @requires_testing_data
-def test_encoding(tmp_path):
+def test_locale_encoding(tmp_path):
     """Test NIRx encoding."""
     fname = tmp_path / 'latin'
     shutil.copytree(fname_nirx_15_2, fname)
@@ -464,13 +469,23 @@ def test_encoding(tmp_path):
     hdr = list()
     with open(hdr_fname, 'rb') as fid:
         hdr.extend(line for line in fid)
+    # French
     hdr[2] = b'Date="jeu. 13 f\xe9vr. 2020"\r\n'
     with open(hdr_fname, 'wb') as fid:
         for line in hdr:
             fid.write(line)
-    # smoke test
-    with pytest.raises(RuntimeWarning, match='Extraction of measurement date'):
-        read_raw_nirx(fname)
+    read_raw_nirx(fname, verbose='debug')
+    # German
+    hdr[2] = b'Date="mi 13 dez 2020"\r\n'
+    with open(hdr_fname, 'wb') as fid:
+        for line in hdr:
+            fid.write(line)
+    read_raw_nirx(fname, verbose='debug')
+    # Italian
+    raw = read_raw_nirx(nirstar_it, verbose='debug')
+    want_dt = dt.datetime(
+        2020, 1, 24, 10, 57, 41, 454000, tzinfo=dt.timezone.utc)
+    assert raw.info['meas_date'] == want_dt
 
 
 @requires_testing_data
@@ -577,7 +592,8 @@ def test_nirx_15_0():
     [fname_nirx_15_2_short, 1],
     [fname_nirx_15_2, 0],
     [fname_nirx_15_2, 0],
-    [nirsport2_2021_9, 0]
+    [nirsport2_2021_9, 0],
+    [nirstar_it, 0],
 ))
 def test_nirx_standard(fname, boundary_decimal):
     """Test standard operations."""