Repository: mne-tools/mne-python

Issue:
[Question][fNIRS] The use of .dat file
Dear MNE devs and users,

I'm not quite used to fNIRS nor the MNE package (started my internship last week) and I can't figure out the use the *.dat file.

In your [tutorial](https://mne.tools/dev/auto_tutorials/preprocessing/plot_70_fnirs_processing.html#tut-fnirs-processing) concerning the Preprocessing of fNIRS data, the function `mne.io.read_raw_nirx(fnirs_raw_dir, verbose=True)` (line 11 of the first box) requires a *.dat file to create a **RawNIRX** instance, although my tutor explained to me that *.dat are essentially files used to display data in real-time (i.e. during the experiment) and as such, are based on standard parameters : they're not usable after the experiment because they do not contain useful (to be processed) data, since the probed data is stored in *.wl1 and *.wl1, events in *.evt and the metadata in *.hdr and *_probeInfo.mat.

I searched through the code and even though you do [load](https://github.com/mne-tools/mne-python/blob/f6c6ff2842ff020816bfc203f00e03410c9e0ea3/mne/io/nirx/nirx.py#L74) the file, I can't find where you use it (or if you use it all)? Am I missing something ?

Thanks in advance,

David


Discussion:
are you sure you talk about the same .dat format? .dat is a terrible
extension choice as it is used
for very different things.

> .dat is a terrible extension choice as it is usedfor very different things.

Definitely...

The fNIRS is still pretty new in our lab, but I'm almsot sure of it yeah. We're using both NIRScout and NIRSport, if it may help. The data provided by machines happen to perfectly suit (in most cases) the requirements (*.wl1, *.wl2, *.hdr ...) but some of our directories lack a *.dat file, which is required by the fNIRS module : `RuntimeError: Expect one dat file, got 0`

My tutor explained ro me what they are for the machines, I don't know if the required *.dat file is exactly the same or not. Regardless, I can't find where it is used (apart from being loaded)
do you have a nirx system?

Yes, both NIRScout and NIRSport. I'm not the one using them though, I'm "just" here to process the data
@rob-luke ?
Actually, `read_raw_nirx` requires either the name of the directory with all the data files, or the header file (extension .hdr). Where exactly in the tutorial is a .dat file required?
The `read_raw_nirx` does not require a *.dat file per se : it returns a **RawNIRX** instance. Thing is, the **RawNIRX** constructor [does](https://github.com/mne-tools/mne-python/blob/f6c6ff2842ff020816bfc203f00e03410c9e0ea3/mne/io/nirx/nirx.py#L72) require a *.dat file :

```python
# Check if required files exist and store names for later use
files = dict()
keys = ('dat', 'evt', 'hdr', 'inf', 'set', 'tpl', 'wl1', 'wl2',
        'config.txt', 'probeInfo.mat')
for key in keys:
    files[key] = glob.glob('%s/*%s' % (fname, key))
    if len(files[key]) != 1:
        raise RuntimeError('Expect one %s file, got %d' %
                            (key, len(files[key]),))
    files[key] = files[key][0]
```

The forloop raises an error if the number of <any_of_the_keys> file differs from 1, especially in my case, a 'dat' file.

EDIT : For what it's worth, [here](https://github.com/mne-tools/mne-python/commit/8861432a3711a5355820b34f76ff6f1ce5d4ad43)'s the commit that introduced said loop; it was authored by @rob-luke and committed by @larsoner 
Got it. Indeed it looks like `'dat'`, `'set'`, `'tpl'`, and `'config.txt'` are never used in the code. Therefore, a quick fix would be to issue a warning if one of these files are missing (since I assume that they could be necessary). Even better would be to determine which of these files are required, and which are just optional (in which case it wouldn't even be necessary to warn). Let's wait for @rob-luke.
Thanks for addressing this @Swy7ch,

You are correct that the dat is not used in the current MNE read nirx function. We could exclude this from the file check as you have pointed out. All the NIRScout data I have seen from several labs over the last two years has contained a `dat` file. Did your device not create it (perhaps this is a NIRSport thing?), or did you delete it? 

Regarding the other files that are currently not used. The current nirx reader is in minimum working form, I am still expanding it to read additional information (from last few days see #7891). The data saved by the device has lots of duplication, for example the same trigger/event information is stored in both the `.evt` and `hdr` file, currently I am just reading one of these, but the plan is to eventually read both and test to ensure they are consistent. Other files contain information that I would like to add to the reader but haven't had a chance yet, such as channel noise measurements (perhaps useful for setting bad channels). 

My understanding is that all of these files are always saved by the vendor software (please correct me if I am wrong here, I don't have access to all devices). My preference is that people simply copy the saved directory and load that directly in MNE. If files are missing it means people have played with the data, and that is something I try to avoid. 

So my preference is that we keep the code as is (provided the dat file is always created by vendor software). People shouldn't modify their raw data, and for one hour measurements the dat file is approximately 8 MB which shouldn't break the data bank.

However, if everyone disagrees with me then my second preference is to throw a warning if the `dat` file is missing saying something like "Warning: The NIRX dat file is missing, this means the data has been modified."

Third preference is to simply remove `dat` from the list of files to check. I'm ok with this solution too. However, I think we should keep the check for `set`, `tp`, and `config` as I do plan to read this in future versions.


-------------

On another note. @Swy7ch I do not have access to a NIRSport device and so there are no NIRSport recordings in the test set. Would you be willing to take a short measurement (10 seconds or so, doesn't need a head attached) and send it to me so I can add it to the test set? If so, [here is an example](https://github.com/mne-tools/mne-testing-data/pull/51) of the detail I would require from you.
> Thanks for addressing this @Swy7ch,
> 
> You are correct that the dat is not used in the current MNE read nirx function. We could exclude this from the file check as you have pointed out. All the NIRScout data I have seen from several labs over the last two years has contained a `dat` file. Did your device not create it (perhaps this is a NIRSport thing?), or did you delete it?

As I said earlier in the thread, I'm a newcomer in the lab I work at (I've been there for 2 weeks) and I'm not the one using the machines : I'm part of the "Technical Team" whose purpose is to provide tools for non-developpers to ease data processing. I will ask my coworkers about this.

> My understanding is that all of these files are always saved by the vendor software (please correct me if I am wrong here, I don't have access to all devices). My preference is that people simply copy the saved directory and load that directly in MNE. If files are missing it means people have played with the data, and that is something I try to avoid.

Haven't had the time to fully scrap the NIRx website, but the data they provide [here](https://support.nirx.de/sample-fnirs-experiments) as samples do not contain *.dat nor *.avg files. They also do not provide *_probeInfo.mat files, so I don't know if they deleted stuff from the output or not.

> So my preference is that we keep the code as is (provided the dat file is always created by vendor software). People shouldn't modify their raw data, and for one hour measurements the dat file is approximately 8 MB which shouldn't break the data bank.
>
> However, if everyone disagrees with me then my second preference is to throw a warning if the `dat` file is missing saying something like "Warning: The NIRX dat file is missing, this means the data has been modified."

I tend to prefer the 2nd option : if we don't actually use a *.dat file, I think it's better to throw a warning instead of exiting. Now of course, if the file data that may be useful, it must be kept in the code, just like the files you cite next. Of course, I agree with the fact that "people shouldn't modify their raw data".
 
> Third preference is to simply remove `dat` from the list of files to check. I'm ok with this solution too. However, I think we should keep the check for `set`, `tp`, and `config` as I do plan to read this in future versions.
> 
> On another note. @Swy7ch I do not have access to a NIRSport device and so there are no NIRSport recordings in the test set. Would you be willing to take a short measurement (10 seconds or so, doesn't need a head attached) and send it to me so I can add it to the test set? If so, [here is an example](https://github.com/mne-tools/mne-testing-data/pull/51) of the detail I would require from you.

As I said I'm a newcomer and didn't conduct the experiment myself, but I will definitely see with my tutor and other researchers in the lab, no problem !

\-\-

Alright I had an epiphany. You said you only worked with NIRScout datasets, and I know some of ours was produced through NIRSport (I just don't know which one of them)... Correct me if I'm wrong, but the NIRSport doesn't provide real-time data display, right ? On the NIRx website, no photo of people wearing a NIRSport shows the device being linked to a display... And if the *.dat (and *.avg) files are only created to provide real-time data display and the NIRSport doesn't support it, then *maybe* the files are not even created in the first place ?

I'll talk about it with my coworkers and try to sort our datasets out to see if my hypothesis makes sense, and I'll come back to you next week. Thanks for the answer, and have a good week-end :)
I'd prefer the second option too. It is better to have a loader that works with a minimum of required files. If there's additional information contained in separate files, and these files are missing, a warning would be much better than not loading the file at all. Also, I wouldn't say that if a file is missing automatically means that the data has been modified - it means that a file is missing, which is different from having modified the data.

Also, some files do not seem to be created by other NIRX devices, which means they should really be treated as optional.

Even if you do plan to add support for reading some other files, in the meantime I'd still fall back to warn if these are not present.

Resolving patch:
diff --git a/mne/io/nirx/nirx.py b/mne/io/nirx/nirx.py
--- a/mne/io/nirx/nirx.py
+++ b/mne/io/nirx/nirx.py
@@ -15,6 +15,7 @@
 from ...annotations import Annotations
 from ...transforms import apply_trans, _get_trans
 from ...utils import logger, verbose, fill_doc
+from ...utils import warn
 
 
 @fill_doc
@@ -71,7 +72,7 @@ def __init__(self, fname, preload=False, verbose=None):
 
         # Check if required files exist and store names for later use
         files = dict()
-        keys = ('dat', 'evt', 'hdr', 'inf', 'set', 'tpl', 'wl1', 'wl2',
+        keys = ('evt', 'hdr', 'inf', 'set', 'tpl', 'wl1', 'wl2',
                 'config.txt', 'probeInfo.mat')
         for key in keys:
             files[key] = glob.glob('%s/*%s' % (fname, key))
@@ -79,6 +80,11 @@ def __init__(self, fname, preload=False, verbose=None):
                 raise RuntimeError('Expect one %s file, got %d' %
                                    (key, len(files[key]),))
             files[key] = files[key][0]
+        if len(glob.glob('%s/*%s' % (fname, 'dat'))) != 1:
+            warn("A single dat file was expected in the specified path, but "
+                 "got %d. This may indicate that the file structure has been "
+                 "modified since the measurement was saved." %
+                 (len(glob.glob('%s/*%s' % (fname, 'dat')))))
 
         # Read number of rows/samples of wavelength data
         last_sample = -1


Test patch:
diff --git a/mne/io/nirx/tests/test_nirx.py b/mne/io/nirx/tests/test_nirx.py
--- a/mne/io/nirx/tests/test_nirx.py
+++ b/mne/io/nirx/tests/test_nirx.py
@@ -5,6 +5,7 @@
 
 import os.path as op
 import shutil
+import os
 
 import pytest
 from numpy.testing import assert_allclose, assert_array_equal
@@ -36,6 +37,17 @@ def test_nirx_hdr_load():
     assert raw.info['sfreq'] == 12.5
 
 
+@requires_testing_data
+def test_nirx_dat_warn(tmpdir):
+    """Test reading NIRX files when missing data."""
+    shutil.copytree(fname_nirx_15_2_short, str(tmpdir) + "/data/")
+    os.rename(str(tmpdir) + "/data" + "/NIRS-2019-08-23_001.dat",
+              str(tmpdir) + "/data" + "/NIRS-2019-08-23_001.tmp")
+    fname = str(tmpdir) + "/data" + "/NIRS-2019-08-23_001.hdr"
+    with pytest.raises(RuntimeWarning, match='A single dat'):
+        read_raw_nirx(fname, preload=True)
+
+
 @requires_testing_data
 def test_nirx_15_2_short():
     """Test reading NIRX files."""