Repository: mne-tools/mne-python

Issue:
Writing NIRS files to FIF is surprisingly slow
#### Describe the bug

I am reading NIRX NIRS files and writing them to fif files. This process takes longer than I expected.

It may be that this is how long writing takes. But I want to make sure there is nothing wrong with how we have written the `read_raw_nirx` function that is causing a slowdown.

Or perhaps fif is not the correct format I should be using? I selected fif as I know that it should work as its being tested [here](https://github.com/mne-tools/mne-python/blob/d0dc8f3d51cfd0db8bcec8f1adee98050152d2df/mne/io/tests/test_raw.py#L129) (called [here](https://github.com/mne-tools/mne-python/blob/master/mne/io/nirx/tests/test_nirx.py#L231)).


#### Steps to reproduce
```python
import os
import time
import mne

fnirs_data_folder = mne.datasets.fnirs_motor.data_path()
fnirs_raw_dir = os.path.join(fnirs_data_folder, 'Participant-1')
raw_intensity = mne.io.read_raw_nirx(fnirs_raw_dir, verbose=True, preload=False)

t = time.time()
raw_intensity.save("test_raw.fif")
print(time.time() - t)
```

On my PC this takes almost 2 minutes.

#### Expected results

I don't have an exact time I would expect this to take, but two minutes to process a 5 MB file seemed slow.


#### Actual results

About 2 mins.


#### Additional information
```
Platform:      Darwin-19.3.0-x86_64-i386-64bit
Python:        3.7.3 (default, Dec 13 2019, 19:58:14)  [Clang 11.0.0 (clang-1100.0.33.17)]
Executable:    /Library/Developer/CommandLineTools/usr/bin/python3
CPU:           i386: 16 cores
Memory:        32.0 GB
mne:           0.21.dev0
numpy:         1.18.2 {blas=openblas, lapack=openblas}
scipy:         1.4.1
matplotlib:    3.2.1 {backend=module://backend_interagg}
sklearn:       0.22.2.post1
numba:         0.48.0
nibabel:       3.0.2
cupy:          Not found
pandas:        1.0.3
dipy:          1.1.1
mayavi:        4.7.1
pyvista:       0.24.1
vtk:           8.1.2
PyQt5:         5.14.1
```


Discussion:
is it also slow if you use preload=True?

if not it's your lazy loading implementation that is slow. Certainly reads
in
more data then requested.

>

>is it also slow if you use preload=True?

I was using `preload=false` as that's what's used in MNE-BIDS which I was starting to implement.

> if not it's your lazy loading implementation that is slow. 

Great pointer. I will look here for efficiency gains.
IIRC during initial implementation we expected it to be slow. Looking at the implementation now, it actually reads every line until it finds the correct one(s), reads them, then breaks.

You could see if switching to the `pandas` implementation is faster. But maybe what would be even better (should be faster than pandas even) is to traverse the entire file exactly once during `__init__` to learn where the line breaks (sample numbers) are, then when on demand reads are called do a `fid.seek` to the proper starting position and then read just the required number of lines.
@kuziekj and I just moved the read_raw_boxy PR https://github.com/mne-tools/mne-python/pull/7717 away from pandas
> @kuziekj and I just moved the read_raw_boxy PR #7717 away from pandas

Indeed it's better not to require it, at least to start. If there ends up being a performance bottleneck and we need to make an (optional!) requirement to read it to boost performance we can do that. But having the non-Pandas reader is still helpful.
Thank you all for the feedback and suggestions on where the code can be improved.

It seems the current implementation is correct but slow. I will leave this issue open as a reminder to address this in the future. But it does not seem urgent.

Resolving patch:
diff --git a/mne/io/nirx/nirx.py b/mne/io/nirx/nirx.py
--- a/mne/io/nirx/nirx.py
+++ b/mne/io/nirx/nirx.py
@@ -245,7 +245,28 @@ def prepend(list, str):
                 info['chs'][ch_idx2 * 2 + 1]['loc'][:3] = ch_locs[ch_idx2, :]
             info['chs'][ch_idx2 * 2]['loc'][9] = fnirs_wavelengths[0]
             info['chs'][ch_idx2 * 2 + 1]['loc'][9] = fnirs_wavelengths[1]
-        raw_extras = {"sd_index": req_ind, 'files': files}
+
+        # Extract the start/stop numbers for samples in the CSV. In theory the
+        # sample bounds should just be 10 * the number of channels, but some
+        # files have mixed \n and \n\r endings (!) so we can't rely on it, and
+        # instead make a single pass over the entire file at the beginning so
+        # that we know how to seek and read later.
+        bounds = dict()
+        for key in ('wl1', 'wl2'):
+            offset = 0
+            bounds[key] = [offset]
+            with open(files[key], 'rb') as fid:
+                for line in fid:
+                    offset += len(line)
+                    bounds[key].append(offset)
+                assert offset == fid.tell()
+
+        # Extras required for reading data
+        raw_extras = {
+            'sd_index': req_ind,
+            'files': files,
+            'bounds': bounds,
+        }
 
         super(RawNIRX, self).__init__(
             info, preload, filenames=[fname], last_samps=[last_sample],
@@ -273,12 +294,12 @@ def _read_segment_file(self, data, idx, fi, start, stop, cals, mult):
         The returned data interleaves the wavelengths.
         """
         sdindex = self._raw_extras[fi]['sd_index']
-        nchan = self._raw_extras[fi]['orig_nchan']
 
         wls = [
             _read_csv_rows_cols(
                 self._raw_extras[fi]['files'][key],
-                start, stop, sdindex, nchan // 2).T
+                start, stop, sdindex,
+                self._raw_extras[fi]['bounds'][key]).T
             for key in ('wl1', 'wl2')
         ]
 
@@ -293,18 +314,11 @@ def _read_segment_file(self, data, idx, fi, start, stop, cals, mult):
         return data
 
 
-def _read_csv_rows_cols(fname, start, stop, cols, n_cols):
-    # The following is equivalent to:
-    # x = pandas.read_csv(fname, header=None, usecols=cols, skiprows=start,
-    #                     nrows=stop - start, delimiter=' ')
-    # But does not require Pandas, and is hopefully fast enough, as the
-    # reading should be done in C (CPython), as should the conversion to float
-    # (NumPy).
-    x = np.zeros((stop - start, n_cols))
-    with _open(fname) as fid:
-        for li, line in enumerate(fid):
-            if li >= start:
-                if li >= stop:
-                    break
-                x[li - start] = np.array(line.split(), float)[cols]
+def _read_csv_rows_cols(fname, start, stop, cols, bounds):
+    with open(fname, 'rb') as fid:
+        fid.seek(bounds[start])
+        data = fid.read(bounds[stop] - bounds[start]).decode('latin-1')
+        x = np.fromstring(data, float, sep=' ')
+    x.shape = (stop - start, -1)
+    x = x[:, cols]
     return x


Test patch: