Repository: mne-tools/mne-python

Issue:
preserve (select) header information from .ncs files in `RawNeuralynx`
### Describe the new feature or enhancement

Each neuralynx `.ncs` file has a header with acquisition metadata (example below). Right now, `read_raw_neuralynx()` doesn't propagate any of that metadata downstream (e.g. in `raw.info`), other than sampling frequency (example below).

It seems to me, `read_raw_neuralynx()` could preserve the time of acquisition (`--TimeCreated` key in header) and any information about online filters if those were applied (`--DSP*` keys in header).

I'm happy to do this, but would need some guidance on which information from header would be useful to include and what the best way of doing so would be.

### Describe your proposed implementation

Certain header properties can vary across `.ncs` channels (I think sampling freq, possibly also DSP params). So if any header information is to be included upon reading the dataset, it will first be checked that the property common across selected channels.

### Describe possible alternatives

Not sure what alternatives would be here.

### Additional context

#### Current raw.info
```
from mne.io import read_raw_neuralynx
from mne.datasets.testing import data_path

testing_path = data_path(download=False) / "neuralynx"
fname_patterns = ["*u*.ncs", "*3_gaps.ncs"]
raw = read_raw_neuralynx(fname=testing_path, preload=True, exclude_fname_patterns=fname_patterns)
raw.info

Out[1]: 
<Info | 7 non-empty values
 bads: []
 ch_names: LAHC1, LAHC2, LAHC3, xAIR1, xEKG1
 chs: 5 sEEG
 custom_ref_applied: False
 highpass: 0.0 Hz
 lowpass: 1000.0 Hz
 meas_date: unspecified
 nchan: 5
 projs: []
 sfreq: 2000.0 Hz
>
```

#### Example header (`LAHC1.ncs`)
```bash
######## Neuralynx Data File Header
-FileType NCS
-FileVersion 3.4
-FileUUID 663100aa-dc63-4458-8e21-80667b048cf5
-SessionUUID c6e1f50d-ff9c-40e4-9b9a-b43f627c74c8
-ProbeName
-OriginalFileName "E:\kristijan\2023-11-02_13-39-27\LAHC1.ncs"
-TimeCreated 2023/11/02 13:39:27
-TimeClosed 2023/11/02 13:42:05

-RecordSize 1044
-ApplicationName Pegasus "2.1.3 "
-AcquisitionSystem AcqSystem1 ATLAS
-ReferenceChannel "Source 01 Reference 1"
-SamplingFrequency 2000
-ADMaxValue 32767
-ADBitVolts 0.000000305175781250000006
-AcqEntName LAHC1
-NumADChannels 1
-ADChannel 8
-InputRange 10000
-InputInverted True

-DSPLowCutFilterEnabled True
-DspLowCutFrequency 0.1
-DspLowCutNumTaps 0
-DspLowCutFilterType DCO
-DSPHighCutFilterEnabled True
-DspHighCutFrequency 500
-DspHighCutNumTaps 256
-DspHighCutFilterType FIR
-DspDelayCompensation Enabled
-DspFilterDelay_<B5>s 3984
```


Discussion:
> Certain header properties can vary across .ncs channels (I think sampling freq, possibly also DSP params). So if any header information is to be included upon reading the dataset, it will first be checked that the property common across selected channels.

There is at least one other format where this is the case. IIRC what we did for example for `info['lowpass']` and `info['highpass']` is take the lowest and highest values, respectively, as this is the most conservative choice. (It would be dangerous to take the *lowest highpass value* for example because you could think all your signals were low-passed quite low, do decimation based on that, and get aliasing.)

So yes you could add some of these, take a look at the info fields to see what would match up:

https://mne.tools/stable/generated/mne.Info.html

For other things, I think it's okay just to document how people can use `neo` to get at the metadata separately.
OK, thanks @larsoner! I think I'd preserve the "recording opened" (in "meas_date"), and the online filter frequencies. For example, if I look at the header for one file from the testing dateset:
```
>>> ncs_header["recording_opened"]
datetime.datetime(2023, 11, 2, 13, 39, 27)  # match this to info["meas_date"]
>>> ncs_header["DspLowCutFrequency"]        # set this to info["highpass"] (to the highest across files if non-uniform)
'0.1'
>>> ncs_header["DspHighCutFrequency"]       # set this to info["lowpass"] (to the lowest across files if non-uniform)
'500'
```

For the meas_date, it seems I need to convert to UTC. The only thing I am not sure is that for python > `3.11` you can use `datetime.UTC` vs in python < `3.11` there's `datetime.timezone.utc` to convert a datetime object to UTC. Not sure if there a general preference?
```ptyhon
import datetime
>>> meas_utc = ncs_header["recording_opened"].astimezone(datetime.UTC)
>>> info.set_meas_date(meas_utc)
<Info | 7 non-empty values
 bads: []
 ch_names: LAHC1, LAHC2, LAHC3, xAIR1, xEKG1
 chs: 5 sEEG
 custom_ref_applied: False
 highpass: 0.0 Hz
 lowpass: 1000.0 Hz
 meas_date: 2023-11-02 17:39:27 UTC
 nchan: 5
 projs: []
 sfreq: 2000.0 Hz
>
```

For the filter, I can't seem to be able to find the `mne.Info().filter()` method. Am I missing something?
```
>>>info["highpass"] = ncs_header["DspLowCutFrequency"]
*** RuntimeError: highpass cannot be set directly. Please use method inst.filter() instead.

>>>info.filter
*** AttributeError: 'Info' object has no attribute 'filter'
```

The hint is meant to say that users should do `raw.filter(None, 40)` for example or whatever. But that's really for end users. Devs who write data format readers can set these things with the `info` unlocked. See for example where this is being worked on for `read_raw_edf` https://github.com/mne-tools/mne-python/pull/12441/files
Thanks! I think I see it now; I was missing the `with info._unlock()` context manager. Will follow up on this next week.

Resolving patch:
diff --git a/mne/decoding/base.py b/mne/decoding/base.py
--- a/mne/decoding/base.py
+++ b/mne/decoding/base.py
@@ -11,6 +11,7 @@
 import numbers
 
 import numpy as np
+from scipy.sparse import issparse
 
 from ..fixes import BaseEstimator, _check_fit_params, _get_check_scoring
 from ..parallel import parallel_func
@@ -106,6 +107,12 @@ def fit(self, X, y, **fit_params):
         self : instance of LinearModel
             Returns the modified instance.
         """
+        # Once we require sklearn 1.1+ we should do:
+        # from sklearn.utils import check_array
+        # X = check_array(X, input_name="X")
+        # y = check_array(y, dtype=None, ensure_2d=False, input_name="y")
+        if issparse(X):
+            raise TypeError("X should be a dense array, got sparse instead.")
         X, y = np.asarray(X), np.asarray(y)
         if X.ndim != 2:
             raise ValueError(
diff --git a/mne/decoding/search_light.py b/mne/decoding/search_light.py
--- a/mne/decoding/search_light.py
+++ b/mne/decoding/search_light.py
@@ -5,6 +5,7 @@
 import logging
 
 import numpy as np
+from scipy.sparse import issparse
 
 from ..fixes import _get_check_scoring
 from ..parallel import parallel_func
@@ -254,6 +255,12 @@ def decision_function(self, X):
 
     def _check_Xy(self, X, y=None):
         """Aux. function to check input data."""
+        # Once we require sklearn 1.1+ we should do something like:
+        # from sklearn.utils import check_array
+        # X = check_array(X, ensure_2d=False, input_name="X")
+        # y = check_array(y, dtype=None, ensure_2d=False, input_name="y")
+        if issparse(X):
+            raise TypeError("X should be a dense array, got sparse instead.")
         X = np.asarray(X)
         if y is not None:
             y = np.asarray(y)
diff --git a/mne/io/neuralynx/neuralynx.py b/mne/io/neuralynx/neuralynx.py
--- a/mne/io/neuralynx/neuralynx.py
+++ b/mne/io/neuralynx/neuralynx.py
@@ -1,5 +1,6 @@
 # License: BSD-3-Clause
 # Copyright the MNE-Python contributors.
+import datetime
 import glob
 import os
 
@@ -38,6 +39,32 @@ def read_raw_neuralynx(
     See Also
     --------
     mne.io.Raw : Documentation of attributes and methods of RawNeuralynx.
+
+    Notes
+    -----
+    Neuralynx files are read from disk using the `Neo package
+    <http://neuralensemble.org/neo/>`__.
+    Currently, only reading of the ``.ncs files`` is supported.
+
+    ``raw.info["meas_date"]`` is read from the ``recording_opened`` property
+    of the first ``.ncs`` file (i.e. channel) in the dataset (a warning is issued
+    if files have different dates of acquisition).
+
+    Channel-specific high and lowpass frequencies of online filters are determined
+    based on the ``DspLowCutFrequency`` and ``DspHighCutFrequency`` header fields,
+    respectively. If no filters were used for a channel, the default lowpass is set
+    to the Nyquist frequency and the default highpass is set to 0.
+    If channels have different high/low cutoffs, ``raw.info["highpass"]`` and
+    ``raw.info["lowpass"]`` are then set to the maximum highpass and minimumlowpass
+    values across channels, respectively.
+
+    Other header variables can be inspected using Neo directly. For example::
+
+        from neo.io import NeuralynxIO  # doctest: +SKIP
+        fname = 'path/to/your/data'  # doctest: +SKIP
+        nlx_reader = NeuralynxIO(dirname=fname)  # doctest: +SKIP
+        print(nlx_reader.header)  # doctest: +SKIP
+        print(nlx_reader.file_headers.items())  # doctest: +SKIP
     """
     return RawNeuralynx(
         fname,
@@ -101,6 +128,61 @@ def __init__(
             sfreq=nlx_reader.get_signal_sampling_rate(),
         )
 
+        ncs_fnames = nlx_reader.ncs_filenames.values()
+        ncs_hdrs = [
+            hdr
+            for hdr_key, hdr in nlx_reader.file_headers.items()
+            if hdr_key in ncs_fnames
+        ]
+
+        # if all files have the same recording_opened date, write it to info
+        meas_dates = np.array([hdr["recording_opened"] for hdr in ncs_hdrs])
+        # to be sure, only write if all dates are the same
+        meas_diff = []
+        for md in meas_dates:
+            meas_diff.append((md - meas_dates[0]).total_seconds())
+
+        # tolerate a +/-1 second meas_date difference (arbitrary threshold)
+        # else issue a warning
+        warn_meas = (np.abs(meas_diff) > 1.0).any()
+        if warn_meas:
+            logger.warning(
+                "Not all .ncs files have the same recording_opened date. "
+                + "Writing meas_date based on the first .ncs file."
+            )
+
+        # Neuarlynx allows channel specific low/highpass filters
+        # if not enabled, assume default lowpass = nyquist, highpass = 0
+        default_lowpass = info["sfreq"] / 2  # nyquist
+        default_highpass = 0
+
+        has_hp = [hdr["DSPLowCutFilterEnabled"] for hdr in ncs_hdrs]
+        has_lp = [hdr["DSPHighCutFilterEnabled"] for hdr in ncs_hdrs]
+        if not all(has_hp) or not all(has_lp):
+            logger.warning(
+                "Not all .ncs files have the same high/lowpass filter settings. "
+                + "Assuming default highpass = 0, lowpass = nyquist."
+            )
+
+        highpass_freqs = [
+            float(hdr["DspLowCutFrequency"])
+            if hdr["DSPLowCutFilterEnabled"]
+            else default_highpass
+            for hdr in ncs_hdrs
+        ]
+
+        lowpass_freqs = [
+            float(hdr["DspHighCutFrequency"])
+            if hdr["DSPHighCutFilterEnabled"]
+            else default_lowpass
+            for hdr in ncs_hdrs
+        ]
+
+        with info._unlock():
+            info["meas_date"] = meas_dates[0].astimezone(datetime.timezone.utc)
+            info["highpass"] = np.max(highpass_freqs)
+            info["lowpass"] = np.min(lowpass_freqs)
+
         # Neo reads only valid contiguous .ncs samples grouped as segments
         n_segments = nlx_reader.header["nb_segment"][0]
         block_id = 0  # assumes there's only one block of recording


Test patch:
diff --git a/mne/io/neuralynx/tests/test_neuralynx.py b/mne/io/neuralynx/tests/test_neuralynx.py
--- a/mne/io/neuralynx/tests/test_neuralynx.py
+++ b/mne/io/neuralynx/tests/test_neuralynx.py
@@ -2,6 +2,7 @@
 # Copyright the MNE-Python contributors.
 import os
 from ast import literal_eval
+from datetime import datetime, timezone
 
 import numpy as np
 import pytest
@@ -103,7 +104,12 @@ def _read_nlx_mat_chan_keep_gaps(matfile: str) -> np.ndarray:
     return x
 
 
+# set known values for the Neuralynx data for testing
 expected_chan_names = ["LAHC1", "LAHC2", "LAHC3", "xAIR1", "xEKG1"]
+expected_hp_freq = 0.1
+expected_lp_freq = 500.0
+expected_sfreq = 2000.0
+expected_meas_date = datetime.strptime("2023/11/02 13:39:27", "%Y/%m/%d  %H:%M:%S")
 
 
 @requires_testing_data
@@ -125,6 +131,14 @@ def test_neuralynx():
         exclude_fname_patterns=fname_patterns,
     )
 
+    # test that we picked the right info from headers
+    assert raw.info["highpass"] == expected_hp_freq, "highpass freq not set correctly"
+    assert raw.info["lowpass"] == expected_lp_freq, "lowpass freq not set correctly"
+    assert raw.info["sfreq"] == expected_sfreq, "sampling freq not set correctly"
+
+    meas_date_utc = expected_meas_date.astimezone(timezone.utc)
+    assert raw.info["meas_date"] == meas_date_utc, "meas_date not set correctly"
+
     # test that channel selection worked
     assert (
         raw.ch_names == expected_chan_names