Repository: mne-tools/mne-python

Issue:
read_raw_curry() does not populate info['meas_date']
#### Describe the new feature or enhancement
When loading Curry data files using **read_raw_curry()** , the raw.info[] property 'meas_date' is not set.  This information is available in the Curry "info" file (the .DAP file for Curry 7 format datasets).

#### Describe your proposed implementation
Add code into the **_read_curry_info** method in  mne\io\curry\curry.py

#### Additional comments
I intend to implement this feature myself, and will issue a PR when ready
read_raw_curry() with preload=True  produces error 
#### Describe the bug
**read_raw_curry()** with **preload=True**  produces error (for some DAT files)

I am working with Curry version 7 data files, using **read_raw_curry()** to load data.  Out of 16 datasets that I am processing, 7 of them produce an error similar to this:

> RuntimeError: Incorrect number of samples (5302650 != 5379500), please report this error to MNE-Python developers


I also determined that the error **_only_** occurs using the method option  **preload=True**.  If I use **preload=False**, the data loads fine for all 16 datasets!

Note that the same problem was reported in [this comment](https://github.com/mne-tools/mne-python/issues/7300#issuecomment-584300596) to issue 7300 .  At that time, there was no clear cause for this error.


#### Steps to reproduce
This is a link to Curry data files (.dat, .rs3, .dap)  for reproducing the error:
https://drive.google.com/file/d/1842rPYAf_LkYF9hQxkKjsMz4MY240m3r/view?usp=sharing

test code:
```
import mne
data_path = 'C:\\read_curry example data\\PH005_baseline.dat'

raw = mne.io.read_raw_curry(data_path,preload=False,verbose=True)
print('Data loaded, with preload=False: ' + data_path +  ', ' + str(raw.info["nchan"]) + ' channels X ' + str(raw.n_times) + ' points')

raw = mne.io.read_raw_curry(data_path,preload=True,verbose=True)
print('Data loaded, with preload=True: ' + data_path +  ', ' + str(raw.info["nchan"]) + ' channels X ' + str(raw.n_times) + ' points')

```

#### Expected results
Would expect to load correctly with either "preload=False" or "preload=True", which would produce this output:

```
Leaving device<->head transform as None (no landmarks found)
Event file not found. No Annotations set.
Data loaded, with preload=False: C:\read_curry example data\PH005_baseline.dat, 70 channels X 76850 points
Leaving device<->head transform as None (no landmarks found)
Reading 0 ... 76849  =      0.000 ...   307.396 secs...
Event file not found. No Annotations set.
Data loaded, with preload=True: C:\read_curry example data\PH005_baseline.dat, 70 channels X 76850 points
```


#### Actual results
First load (with  preload=False) works fine; second load (with  preload=True) throws error:
```
Data loaded, with preload=False: C:\read_curry example data\PH005_baseline.dat, 70 channels X 76850 points
Leaving device<->head transform as None (no landmarks found)
Reading 0 ... 76849  =      0.000 ...   307.396 secs...
Traceback (most recent call last):
  File "c:\Users\todfl\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\ptvsd_launcher.py", line 48, in <module>
    main(ptvsdArgs)
  File "c:\Users\todfl\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main    
    run()
  File "c:\Users\todfl\.vscode\extensions\ms-python.python-2020.2.64397\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code
  File "c:\Users\todfl\Documents\Decker\EEG Processing\EEG Processing Code\read_raw_curry error MWE.py", line 7, in <module>
    raw = mne.io.read_raw_curry(data_path,preload=True,verbose=True)
  File "<decorator-gen-218>", line 20, in read_raw_curry
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\curry\curry.py", line 406, in read_raw_curry
    return RawCurry(fname, preload, verbose)
  File "<decorator-gen-219>", line 20, in __init__
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\curry\curry.py", line 439, in __init__
    super(RawCurry, self).__init__(
  File "<decorator-gen-160>", line 20, in __init__
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\base.py", line 247, in __init__
    self._preload_data(preload)
  File "<decorator-gen-164>", line 20, in _preload_data
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\base.py", line 528, in _preload_data
    self._data = self._read_segment(
  File "<decorator-gen-162>", line 20, in _read_segment
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\base.py", line 412, in _read_segment
    _ReadSegmentFileProtector(self)._read_segment_file(
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\base.py", line 1816, in _read_segment_file
    return self.__raw.__class__._read_segment_file(
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\curry\curry.py", line 469, in _read_segment_file
    _read_segments_file(
  File "C:\ProgramData\Anaconda3\lib\site-packages\mne\io\utils.py", line 221, in _read_segments_file
    raise RuntimeError('Incorrect number of samples (%s != %s), '
  RuntimeError: Incorrect number of samples (5302650 != 5379500), please report this error to MNE-Python developers
```


#### Additional information
Platform:      Windows-10-10.0.18362-SP0
Python:        3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]
Executable:    C:\ProgramData\Anaconda3\python.exe
CPU:           AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD: 16 cores
Memory:        16.0 GB

mne:           0.21.0
numpy:         1.18.5 {blas=mkl_rt, lapack=mkl_rt}
scipy:         1.5.0
matplotlib:    3.3.2 {backend=Qt5Agg}

sklearn:       0.23.1
numba:         0.50.1
nibabel:       Not found
cupy:          Not found
pandas:        1.0.5
dipy:          Not found
mayavi:        Not found
pyvista:       Not found
vtk:           Not found

read_raw_curry() does not populate info['meas_date']
#### Describe the new feature or enhancement
When loading Curry data files using **read_raw_curry()** , the raw.info[] property 'meas_date' is not set.  This information is available in the Curry "info" file (the .DAP file for Curry 7 format datasets).

#### Describe your proposed implementation
Add code into the **_read_curry_info** method in  mne\io\curry\curry.py

#### Additional comments
I intend to implement this feature myself, and will issue a PR when ready


Discussion:

Just a followup -- no surprise that the following code also produces the same error:
```
import mne
data_path = 'C:\\read_curry example data\\PH005_baseline.dat'

raw = mne.io.read_raw_curry(data_path,preload=False,verbose=True)
print('Data loaded, with preload=False: ' + data_path +  ', ' + str(raw.info["nchan"]) + ' channels X ' + str(raw.n_times) + ' points')
raw.load_data() 
```

The **raw.load_data()**  line throws the same error as described above:
`RuntimeError: Incorrect number of samples (5302650 != 5379500), please report this error to MNE-Python developers
`
can you look into it yourself? otherwise we'll see who can help

>

Ah, you mean look into the code to find the problem _myself_... I hadn't considered doing that! ;-)
Sure, I will give it a shot!  I'm not an expert in Python dev, but I will have a go at it.
yes ! let us know if you need help

>



Resolving patch:
diff --git a/mne/datasets/utils.py b/mne/datasets/utils.py
--- a/mne/datasets/utils.py
+++ b/mne/datasets/utils.py
@@ -240,7 +240,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
     path = _get_path(path, key, name)
     # To update the testing or misc dataset, push commits, then make a new
     # release on GitHub. Then update the "releases" variable:
-    releases = dict(testing='0.106', misc='0.6')
+    releases = dict(testing='0.108', misc='0.6')
     # And also update the "md5_hashes['testing']" variable below.
     # To update any other dataset, update the data archive itself (upload
     # an updated version) and update the md5 hash.
@@ -326,7 +326,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
         sample='12b75d1cb7df9dfb4ad73ed82f61094f',
         somato='32fd2f6c8c7eb0784a1de6435273c48b',
         spm='9f43f67150e3b694b523a21eb929ea75',
-        testing='d67eff9e1089f15b69f88931dbbf35df',
+        testing='57a21229ebb11e3c470b5d0fdb80642f',
         multimodal='26ec847ae9ab80f58f204d09e2c08367',
         fnirs_motor='c4935d19ddab35422a69f3326a01fef8',
         opm='370ad1dcfd5c47e029e692c85358a374',
diff --git a/mne/io/curry/curry.py b/mne/io/curry/curry.py
--- a/mne/io/curry/curry.py
+++ b/mne/io/curry/curry.py
@@ -9,6 +9,7 @@
 from collections import namedtuple
 import re
 import numpy as np
+from datetime import datetime, timezone
 
 from ..base import BaseRaw
 from ..meas_info import create_info
@@ -49,7 +50,8 @@
 SI_UNIT_SCALE = dict(c=1e-2, m=1e-3, u=1e-6, µ=1e-6, n=1e-9, p=1e-12, f=1e-15)
 
 CurryParameters = namedtuple('CurryParameters',
-                             'n_samples, sfreq, is_ascii, unit_dict')
+                             'n_samples, sfreq, is_ascii, unit_dict, '
+                             'n_chans, dt_start, chanidx_in_file')
 
 
 def _get_curry_version(file_extension):
@@ -133,11 +135,18 @@ def _read_curry_parameters(fname):
 
     var_names = ['NumSamples', 'SampleFreqHz',
                  'DataFormat', 'SampleTimeUsec',
+                 'NumChannels',
+                 'StartYear', 'StartMonth', 'StartDay', 'StartHour',
+                 'StartMin', 'StartSec', 'StartMillisec',
                  'NUM_SAMPLES', 'SAMPLE_FREQ_HZ',
-                 'DATA_FORMAT', 'SAMPLE_TIME_USEC']
+                 'DATA_FORMAT', 'SAMPLE_TIME_USEC',
+                 'NUM_CHANNELS',
+                 'START_YEAR', 'START_MONTH', 'START_DAY', 'START_HOUR',
+                 'START_MIN', 'START_SEC', 'START_MILLISEC']
 
     param_dict = dict()
     unit_dict = dict()
+
     with open(fname) as fid:
         for line in iter(fid):
             if any(var_name in line for var_name in var_names):
@@ -149,10 +158,34 @@ def _read_curry_parameters(fname):
                     unit_dict[type] = data_unit.replace(" ", "") \
                         .replace("\n", "").split("=")[-1]
 
+    # look for CHAN_IN_FILE sections, which may or may not exist; issue #8391
+    types = ["meg", "eeg", "misc"]
+    chanidx_in_file = _read_curry_lines(fname,
+                                        ["CHAN_IN_FILE" +
+                                         CHANTYPES[key] for key in types])
+
     n_samples = int(param_dict["numsamples"])
     sfreq = float(param_dict["samplefreqhz"])
     time_step = float(param_dict["sampletimeusec"]) * 1e-6
     is_ascii = param_dict["dataformat"] == "ASCII"
+    n_channels = int(param_dict["numchannels"])
+    try:
+        dt_start = datetime(int(param_dict["startyear"]),
+                            int(param_dict["startmonth"]),
+                            int(param_dict["startday"]),
+                            int(param_dict["starthour"]),
+                            int(param_dict["startmin"]),
+                            int(param_dict["startsec"]),
+                            int(param_dict["startmillisec"]) * 1000,
+                            timezone.utc)
+        # Note that the time zone information is not stored in the Curry info
+        # file, and it seems the start time info is in the local timezone
+        # of the acquisition system (which is unknown); therefore, just set
+        # the timezone to be UTC.  If the user knows otherwise, they can
+        # change it later.  (Some Curry files might include StartOffsetUTCMin,
+        # but its presence is unpredictable, so we won't rely on it.)
+    except (ValueError, KeyError):
+        dt_start = None  # if missing keywords or illegal values, don't set
 
     if time_step == 0:
         true_sfreq = sfreq
@@ -165,7 +198,8 @@ def _read_curry_parameters(fname):
     if true_sfreq <= 0:
         raise ValueError(_msg_invalid.format(true_sfreq))
 
-    return CurryParameters(n_samples, true_sfreq, is_ascii, unit_dict)
+    return CurryParameters(n_samples, true_sfreq, is_ascii, unit_dict,
+                           n_channels, dt_start, chanidx_in_file)
 
 
 def _read_curry_info(curry_paths):
@@ -191,11 +225,28 @@ def _read_curry_info(curry_paths):
 
     all_chans = list()
     for key in ["meg", "eeg", "misc"]:
+        chanidx_is_explicit = (len(curry_params.chanidx_in_file["CHAN_IN_FILE"
+                                   + CHANTYPES[key]]) > 0)    # channel index
+        # position in the datafile may or may not be explicitly declared,
+        # based on the CHAN_IN_FILE section in info file
         for ind, chan in enumerate(labels["LABELS" + CHANTYPES[key]]):
+            chanidx = len(all_chans) + 1    # by default, just assume the
+            # channel index in the datafile is in order of the channel
+            # names as we found them in the labels file
+            if chanidx_is_explicit:  # but, if explicitly declared, use
+                # that index number
+                chanidx = int(curry_params.chanidx_in_file["CHAN_IN_FILE"
+                              + CHANTYPES[key]][ind])
+            if chanidx <= 0:   # if chanidx was explicitly declared to be ' 0',
+                # it means the channel is not actually saved in the data file
+                # (e.g. the "Ref" channel), so don't add it to our list.
+                # Git issue #8391
+                continue
             ch = {"ch_name": chan,
                   "unit": curry_params.unit_dict[key],
                   "kind": FIFFV_CHANTYPES[key],
                   "coil_type": FIFFV_COILTYPES[key],
+                  "ch_idx": chanidx
                   }
             if key == "eeg":
                 loc = np.array(sensors["SENSORS" + CHANTYPES[key]][ind], float)
@@ -222,11 +273,24 @@ def _read_curry_info(curry_paths):
                 ch['coord_frame'] = FIFF.FIFFV_COORD_DEVICE
             all_chans.append(ch)
 
+    ch_count = len(all_chans)
+    assert (ch_count == curry_params.n_chans)  # ensure that we have assembled
+    # the same number of channels as declared in the info (.DAP) file in the
+    # DATA_PARAMETERS section. Git issue #8391
+
+    # sort the channels to assure they are in the order that matches how
+    # recorded in the datafile.  In general they most likely are already in
+    # the correct order, but if the channel index in the data file was
+    # explicitly declared we might as well use it.
+    all_chans = sorted(all_chans, key=lambda ch: ch['ch_idx'])
+
     ch_names = [chan["ch_name"] for chan in all_chans]
     info = create_info(ch_names, curry_params.sfreq)
+    info['meas_date'] = curry_params.dt_start          # for Git issue #8398
     _make_trans_dig(curry_paths, info, curry_dev_dev_t)
 
     for ind, ch_dict in enumerate(info["chs"]):
+        all_chans[ind].pop('ch_idx')
         ch_dict.update(all_chans[ind])
         assert ch_dict['loc'].shape == (12,)
         ch_dict['unit'] = SI_UNITS[all_chans[ind]['unit'][1]]


Test patch:
diff --git a/mne/io/curry/tests/test_curry.py b/mne/io/curry/tests/test_curry.py
--- a/mne/io/curry/tests/test_curry.py
+++ b/mne/io/curry/tests/test_curry.py
@@ -8,6 +8,7 @@
 import os.path as op
 import numpy as np
 from shutil import copyfile
+from datetime import datetime, timezone
 
 import pytest
 from numpy.testing import assert_allclose, assert_array_equal
@@ -47,6 +48,10 @@
 
 missing_event_file = op.join(curry_dir, "test_sfreq_0.dat")
 
+Ref_chan_omitted_file = op.join(curry_dir, 'Ref_channel_omitted Curry7.dat')
+Ref_chan_omitted_reordered_file = op.join(curry_dir, 'Ref_channel_omitted '
+                                          'reordered Curry7.dat')
+
 
 @pytest.fixture(scope='session')
 def bdf_curry_ref():
@@ -338,7 +343,8 @@ def sfreq_testing_data(tmpdir, request):
 
     _mock_info_file(src=in_base_name + 'dap', dst=out_base_name + 'dap',
                     sfreq=sfreq, time_step=time_step)
-
+    _mock_info_file(src=in_base_name + 'rs3', dst=out_base_name + 'rs3',
+                    sfreq=sfreq, time_step=time_step)
     return out_base_name + 'dat'
 
 
@@ -427,4 +433,49 @@ def test_read_curry_annotations_using_mocked_info(tmpdir, name_part):
     assert_array_equal(annot.description, EXPECTED_DESCRIPTION)
 
 
+@testing.requires_testing_data
+@pytest.mark.parametrize('fname,expected_channel_list', [
+    pytest.param(Ref_chan_omitted_file,
+                 ['FP1', 'FPZ', 'FP2', 'VEO', 'EKG', 'Trigger'],
+                 id='Ref omitted, normal order'),
+    pytest.param(Ref_chan_omitted_reordered_file,
+                 ['FP2', 'FPZ', 'FP1', 'VEO', 'EKG', 'Trigger'],
+                 id='Ref omitted, reordered')
+])
+def test_read_files_missing_channel(fname, expected_channel_list):
+    """Test reading data files that has an omitted channel."""
+    # This for Git issue #8391.  In some cases, the 'labels' (.rs3 file will
+    # list channels that are not actually saved in the datafile (such as the
+    # 'Ref' channel).  These channels are denoted in the 'info' (.dap) file
+    # in the CHAN_IN_FILE section with a '0' as their index.
+    # If the CHAN_IN_FILE section is present, the code also assures that the
+    # channels are sorted in the prescribed order.
+    # This test makes sure the data load correctly, and that we end up with
+    # the proper channel list.
+    raw = read_raw_curry(fname, preload=True)
+    assert raw.ch_names == expected_channel_list
+
+
+@testing.requires_testing_data
+@pytest.mark.parametrize('fname,expected_meas_date', [
+    pytest.param(Ref_chan_omitted_file,
+                 datetime(2018, 11, 21, 12, 53, 48,
+                          525000, tzinfo=timezone.utc),
+                 id='valid start date'),
+    pytest.param(curry7_rfDC_file,
+                 None,
+                 id='start date year is 0'),
+    pytest.param(curry7_bdf_file,
+                 None,
+                 id='start date seconds invalid')
+])
+def test_meas_date(fname, expected_meas_date):
+    """Test reading acquisition start datetime info info['meas_date']."""
+    # This for Git issue #8398.  The 'info' (.dap) file includes acquisition
+    # start date & time.  Test that this goes into raw.info['meas_date'].
+    # If the information is not valid, raw.info['meas_date'] should be None
+    raw = read_raw_curry(fname, preload=False)
+    assert raw.info['meas_date'] == expected_meas_date
+
+
 run_tests_if_main()