Repository: mne-tools/mne-bids

Issue:
[BUG] cHPI information in meg recording sidecar set incorrectly
#### Describe the bug
Since the last version of mne_bids, the `ContinuousHeadLocalization` field is set to true in my data (recorded with Neuromag Vectorview), even though I have not activated it during the measurement. First I thought I made a mistake and accidentally activated it, but looking at `raw.plot_psd()` there is no above-baseline activity at the specified frequencies. And since this happens to a bunch of files I have recorded (for whom I am certain that I have not activated cHPI every single time), I am rather sure cHPI was indeed inactive.

I had a look at how mne-bids retrieves cHPI information and sets the field. Essentially, it checks (via mne) whether the `raw.info` file includes frequencies at which the HPIs supposedly oscillated. Indeed such frequencies are set in my raw files. However, my hunch is that they are not diagnostic for whether cHPI was active or not, but are set per default, by some general settings or something like that. Therefore, relying on whether frequencies are set or not, might not be a valid criterion here. 

In fact, looking at `mne.chpi.get_chpi_info()`, which is what is used to decide about cHPI, also has a return value that supposedly indicates whether cHPI is on or off:
https://github.com/mne-tools/mne-python/blob/b7e8c7791b000258b2ba158ddd89eecbbb35af30/mne/chpi.py#L310-L318

which might be a better choice to infer whether or not cHPI or not were used. However, in seems that also these information are non-zero in my case. So, I don't know what is going on. Next week I shall try to record some test files with and without cHPI, to see in which way they differ. 

#### Additional information
. I use the latest stable versions of both mne and mne-bids. The relevant part of the code is this: 
https://github.com/mne-tools/mne-bids/blob/0473a3e50deda6db0a0e3fba5feaa62a3d0486ad/mne_bids/write.py#L751



Discussion:
Thanks for the report!

> Next week I shall try to record some test files with and without cHPI, to see in which way they differ.

looking forward to your results!
> In fact, looking at `mne.chpi.get_chpi_info()`, which is what is used to decide about cHPI, also has a return value that supposedly indicates whether cHPI is on or off:

Excellent find! It would be good to make use of this instead, then! 👍 
Hey, 
A small update on that issue. As you can see, there are clear cHPI related artifiacts in the signal(top) when I chose cHPI to be active, and none (only line frequency) when I deactivate it:
![active cHPI](https://user-images.githubusercontent.com/8819465/149813687-07903ea8-6b0c-4777-87a5-a7e01e6cb627.png)
![inactive cHPI](https://user-images.githubusercontent.com/8819465/149813713-95246bb0-5603-497b-bafa-d1b0cf0d6836.png)

However, the information in the `raw.info` contains the same information (with some recording specific variations). Therefore, it seems that mne is not correctly extracting the status of the cHPI from the fif file (provided that they are properly recorded in it). 

Should I open an issue in the mne repo?


> However, the information in the raw.info contains the same information (with some recording specific variations). 

Whether or not the continuous HPI is on can vary as a function of time, so `info` (which is static) will not tell you what you want to know. Instead you need to look at something with time-varying information, which in this case are the system status channels from `hpi_picks`, which eventually are used with `hpi_ons` to determine which channels are on for example here:

- https://github.com/mne-tools/mne-python/blob/e48ec6cb0689f747f2ec53a162fe13496c03a200/mne/chpi.py#L556
- https://github.com/mne-tools/mne-python/blob/main/mne/chpi.py#L682-L684
Thanks for clarifying @larsoner . If I understand correctly, the hpi's periodically switch on, in the specified frequency, and are off all other times? And there isn't a tag in the header or something that essentially codes whether the "record cHPI" checkbox was ticked?

In any case, mne-bids doesn't handle cHPIs correctly. Unless there is an easy way to access whether cHPI was active or not, the only option I can think off is what Eric proposed, that is, going through the recording and check whether there is a status change in the hpis.
> If I understand correctly, the hpi's periodically switch on, in the specified frequency, and are off all other times? And there isn't a tag in the header or something that essentially codes whether the "record cHPI" checkbox was ticked?

The person running the acquisition can toggle it during the acquisition. The most common case is that it's on the whole time (or sometimes the whole time except the first second or two, depending on the clicking order of start-recording and start-cHPI) or it's off the whole time. But people can do recordings where they just turn it on at specified (or random) intervals if they want.

>  there isn't a tag in the header or something that essentially codes whether the "record cHPI" checkbox was ticked?

No, because it's not static, but rather a time-varying property it really makes sense for this to be a "data channel" that encodes time-point-by-time-point which HPI coils were activated (if any).

> Unless there is an easy way to access whether cHPI was active or not

Something like `n_coils = (raw.get_data('chpi') & some_bitmask).sum(axis=0)` gives you `n_coils` of shape `(n_times,)` with the cHPI-active count. This seems easy enough to me...?
Hi, 
this issue fell of my radar, sorry. 
This week I finally remembered to record some data with and without cHPI and try @larsoner suggestion. Indeed it works quite nice with his suggestions. 

I can make a PR with the changes, but I am not sure about the desired behavior. To set the chpi field in the meg sidecar to true, should the chpi's be on for all samples in the recording, or any samples? Any would be a little easier to implement I think. What do you think?

Oh, and I have only access to Neuromag data. No idea, whether it would work the same for KIT or CTF data


> should the chpi's be on for all samples in the recording, or any samples? Any would be a little easier to implement I think. What do you think?

I would say for any, since for most systems / procedures it's never on at all (and needs to be explicitly enabled). I also have a lot of recordings where it's on for 99.9% of the time because our standard procedure was to "record data" *then* "turn on cHPI" (or whatever it is on the UI), and it would be misleading to mark such recordings as cHPI off -- more misleading than saying cHPI was on when it was only on for a small fraction of time, I think.

> Oh, and I have only access to Neuromag data. No idea, whether it would work the same for KIT or CTF data

I'm not sure about KIT. But at least CTF actually encodes the cHPI *coil locations* in the file in at least some cases, see:

https://github.com/mne-tools/mne-python/blob/main/mne/tests/test_chpi.py#L666-L667

Maybe we should first add a public `mne.chpi.chpi_on(raw)` that determines how many coils were on as a function of time for any raw instance. For Neuromag and CTF data we actually do some calculations, for all others we return `np.zeros(len(raw.times), int)`. Then you can just call this little function within MNE-BIDS to know if cHPI was on at all.

Resolving patch:
diff --git a/mne_bids/write.py b/mne_bids/write.py
--- a/mne_bids/write.py
+++ b/mne_bids/write.py
@@ -28,7 +28,8 @@
 from mne.io.constants import FIFF
 from mne.io.pick import channel_type, _picks_to_idx
 from mne.io import BaseRaw, read_fiducials
-from mne.channels.channels import _unit2human
+from mne.channels.channels import (_unit2human, _get_meg_system)
+from mne.chpi import get_chpi_info
 from mne.utils import (check_version, has_nibabel, logger, warn, Bunch,
                        _validate_type, get_subjects_dir, verbose,
                        ProgressBar)
@@ -825,35 +826,41 @@ def _sidecar_json(raw, task, manufacturer, fname, datatype,
     }
 
     # Compile cHPI information, if any.
-    from mne.io.ctf import RawCTF
-    from mne.io.kit.kit import RawKIT
-
-    chpi = False
-    hpi_freqs = np.array([])
+    system, _ = _get_meg_system(raw.info)
+    chpi = None
+    hpi_freqs = []
     if (datatype == 'meg' and
             parse_version(mne.__version__) > parse_version('0.23')):
         # We need to handle different data formats differently
-        if isinstance(raw, RawCTF):
+        if system == 'CTF_275':
             try:
                 mne.chpi.extract_chpi_locs_ctf(raw)
                 chpi = True
             except RuntimeError:
+                chpi = False
                 logger.info('Could not find cHPI information in raw data.')
-        elif isinstance(raw, RawKIT):
+        elif system == 'KIT':
             try:
                 mne.chpi.extract_chpi_locs_kit(raw)
                 chpi = True
             except (RuntimeError, ValueError):
+                chpi = False
                 logger.info('Could not find cHPI information in raw data.')
-        else:
-            hpi_freqs, _, _ = mne.chpi.get_chpi_info(info=raw.info,
-                                                     on_missing='ignore')
-            if hpi_freqs.size > 0:
-                chpi = True
+        elif system in ['122m', '306m']:
+            # XXX: Remove this version check when support for mne <1.2
+            # is dropped
+            if parse_version(mne.__version__) > parse_version('1.1'):
+                n_active_hpi = mne.chpi.get_active_chpi(raw,
+                                                        on_missing='ignore')
+                chpi = bool(n_active_hpi.sum() > 0)
+                if chpi:
+                    hpi_freqs, _, _ = get_chpi_info(info=raw.info,
+                                                    on_missing='ignore')
+                    hpi_freqs = list(hpi_freqs)
+
     elif datatype == 'meg':
         logger.info('Cannot check for & write continuous head localization '
                     'information: requires MNE-Python >= 0.24')
-        chpi = None
 
     # Define datatype-specific JSON dictionaries
     ch_info_json_common = [
@@ -875,7 +882,7 @@ def _sidecar_json(raw, task, manufacturer, fname, datatype,
 
     if chpi is not None:
         ch_info_json_meg.append(('ContinuousHeadLocalization', chpi))
-        ch_info_json_meg.append(('HeadCoilFrequency', list(hpi_freqs)))
+        ch_info_json_meg.append(('HeadCoilFrequency', hpi_freqs))
 
     if emptyroom_fname is not None:
         ch_info_json_meg.append(('AssociatedEmptyRoom', str(emptyroom_fname)))


Test patch:
diff --git a/mne_bids/tests/test_read.py b/mne_bids/tests/test_read.py
--- a/mne_bids/tests/test_read.py
+++ b/mne_bids/tests/test_read.py
@@ -14,6 +14,8 @@
 import numpy as np
 from numpy.testing import assert_almost_equal
 
+from pkg_resources import parse_version
+
 import mne
 from mne.io.constants import FIFF
 from mne.utils import requires_nibabel, object_diff, requires_version
@@ -713,21 +715,26 @@ def test_handle_chpi_reading(tmp_path):
         meg_json_data = json.load(f)
 
     # cHPI frequency mismatch
-    meg_json_data_freq_mismatch = meg_json_data.copy()
-    meg_json_data_freq_mismatch['HeadCoilFrequency'][0] = 123
-    _write_json(meg_json_path, meg_json_data_freq_mismatch, overwrite=True)
+    if parse_version(mne.__version__) <= parse_version('1.1'):
+        assert 'ContinuousHeadLocalization' not in meg_json_data
+        assert 'HeadCoilFrequency' not in meg_json_data
+    else:
+        meg_json_data_freq_mismatch = meg_json_data.copy()
+        meg_json_data_freq_mismatch['HeadCoilFrequency'][0] = 123
+        _write_json(meg_json_path, meg_json_data_freq_mismatch, overwrite=True)
 
-    with pytest.warns(RuntimeWarning, match='Defaulting to .* mne.Raw object'):
-        raw_read = read_raw_bids(bids_path)
+        with pytest.warns(RuntimeWarning,
+                          match='Defaulting to .* mne.Raw object'):
+            raw_read = read_raw_bids(bids_path)
 
-    # cHPI "off" according to sidecar, but present in the data
-    meg_json_data_chpi_mismatch = meg_json_data.copy()
-    meg_json_data_chpi_mismatch['ContinuousHeadLocalization'] = False
-    _write_json(meg_json_path, meg_json_data_chpi_mismatch, overwrite=True)
+        # cHPI "off" according to sidecar, but present in the data
+        meg_json_data_chpi_mismatch = meg_json_data.copy()
+        meg_json_data_chpi_mismatch['ContinuousHeadLocalization'] = False
+        _write_json(meg_json_path, meg_json_data_chpi_mismatch, overwrite=True)
 
-    raw_read = read_raw_bids(bids_path)
-    assert raw_read.info['hpi_subsystem'] is None
-    assert raw_read.info['hpi_meas'] == []
+        raw_read = read_raw_bids(bids_path)
+        assert raw_read.info['hpi_subsystem'] is None
+        assert raw_read.info['hpi_meas'] == []
 
 
 @pytest.mark.filterwarnings(warning_str['nasion_not_found'])
diff --git a/mne_bids/tests/test_write.py b/mne_bids/tests/test_write.py
--- a/mne_bids/tests/test_write.py
+++ b/mne_bids/tests/test_write.py
@@ -731,7 +731,6 @@ def test_fif(_bids_validate, tmp_path):
 def test_chpi(_bids_validate, tmp_path, format):
     """Test writing of cHPI information."""
     data_path = testing.data_path()
-    kit_data_path = op.join(base_path, 'kit', 'tests', 'data')
 
     if format == 'fif_no_chpi':
         fif_raw_fname = op.join(data_path, 'MEG', 'sample',
@@ -744,6 +743,7 @@ def test_chpi(_bids_validate, tmp_path, format):
         ctf_raw_fname = op.join(data_path, 'CTF', 'testdata_ctf.ds')
         raw = _read_raw_ctf(ctf_raw_fname)
     elif format == 'kit':
+        kit_data_path = op.join(base_path, 'kit', 'tests', 'data')
         kit_raw_fname = op.join(kit_data_path, 'test.sqd')
         kit_hpi_fname = op.join(kit_data_path, 'test_mrk.sqd')
         kit_electrode_fname = op.join(kit_data_path, 'test.elp')
@@ -763,18 +763,25 @@ def test_chpi(_bids_validate, tmp_path, format):
     if parse_version(mne.__version__) <= parse_version('0.23'):
         assert 'ContinuousHeadLocalization' not in meg_json_data
         assert 'HeadCoilFrequency' not in meg_json_data
-    elif format in ['fif_no_chpi', 'kit']:
+    elif format in ['fif_no_chpi', 'fif']:
+        if parse_version(mne.__version__) <= parse_version('1.1'):
+            assert 'ContinuousHeadLocalization' not in meg_json_data
+            assert 'HeadCoilFrequency' not in meg_json_data
+        else:
+            if format == 'fif_no_chpi':
+                assert meg_json_data['ContinuousHeadLocalization'] is False
+                assert meg_json_data['HeadCoilFrequency'] == []
+            elif format == 'fif':
+                assert meg_json_data['ContinuousHeadLocalization'] is True
+                assert_array_almost_equal(meg_json_data['HeadCoilFrequency'],
+                                          [83., 143., 203., 263., 323.])
+    elif format == 'kit':
         # no cHPI info is contained in the sample data
         assert meg_json_data['ContinuousHeadLocalization'] is False
         assert meg_json_data['HeadCoilFrequency'] == []
-    elif format == 'fif':
-        assert meg_json_data['ContinuousHeadLocalization'] is True
-        assert_array_almost_equal(meg_json_data['HeadCoilFrequency'],
-                                  [83., 143., 203., 263., 323.])
     elif format == 'ctf':
         assert meg_json_data['ContinuousHeadLocalization'] is True
-        assert_array_equal(meg_json_data['HeadCoilFrequency'],
-                           np.array([]))
+        assert meg_json_data['HeadCoilFrequency'] == []
 
 
 @pytest.mark.filterwarnings(warning_str['channel_unit_changed'])