Repository: mne-tools/mne-python

Issue:
Support STIM channels in EDF export
I found some issues with our recently added EDF export that we might want to fix before the next release:

1. The error if `EDFlib-Python` is not installed is misleading, because it says `RuntimeError: For exporting to EDF to work, the EDFlib module is needed, but it could not be imported`. However, although the import name is `EDFlib`, the PyPI name is actually `EDFlib-Python` and `EDFlib` is a different PyPI package (which requires Cython etc.).
2. `EDFlib-Python` is super slow compared to `pyEDFlib` (which uses the C implementation). I tested exporting with one of our BDF files from our lab (425MB), and it took `EDFlib-Python` 1 minute and 52 seconds to export. For comparison, `pyEDFlib` took only 6 seconds.
3. Analog stim channels are not exported. This is essential in BDF files, because Biosemi amplifiers do not have discrete triggers and instead always write an analog stim channel. There is no technical reason to not export stim channels in addition to EEG (other than carefully setting the scaling, unit, and physical min/max). I've fixed this in my EDF/BDF export implementation in MNELAB recently (today) in https://github.com/cbrnr/mnelab/pull/230.

The first issue is probably easy to fix. The second issue is pretty serious, we can't expect that users wait almost 2 minutes for something that can be done in 6 seconds. I'd consider switching to `pyEDFlib` instead (they have binary wheels and AFAIR the restriction to use pure Python applies only to our project but not external dependencies). The third issue requires some work, but this is important for BDF files.


Discussion:
@cbrnr Would you have time to work on this next week?
Probably not next week, maybe the week after. I have to finish a paper revision and some other stuff. But we should decide if we want to replace `EDFlib-Python` with `pyEDFlib` – what is everyone's opinion?
It might make sense to do a quick profile and look at EDFlib-Python's code to see if it can easily be improved for example by using NumPy data writing functions rather than `struct` or anything like that. But I guess it also doesn't matter too much which lib we use, so I'm also fine with just switching to `pyEDFlib` if it's easy enough.
> But we should decide if we want to replace `EDFlib-Python` with `pyEDFlib` – what is everyone's opinion?

I'd pick whichever seems to work best at the moment … if you believe `pyEDFlib` is a good choice, let's choose this. It's on `conda-forge` too, so installation shouldn't be a problem.
I think the slowness comes from our own loop here:

https://github.com/mne-tools/mne-python/blob/main/mne/export/_edf.py#L240-L268

`pyEDFlib` doesn't require users to loop over blocks, because it accepts a list of NumPy arrays (one element per channel). I'm pretty sure this can even be simplified to passing the whole 2D array in one go.
if we can I would prefer to rely only on pure Python code. Compiled code
requires more maintenance
and binaries on many platforms.

my 2c

>

But it's an external package with wheels and an optional dependency. NumPy has tons of compiled code and we're not thinking about a pure Python alternative.
And since it's an optional dependency, we can simply replace it if / once it turns out to become problematic. It won't affect our core functionality in any way...
Exactly. I think in this case the speedup might be worth the tradeoff.
as long as someone does the maintenance I have no further objection

>

My only objection with pyedflib writing is that the end file never opened for me in EDFBrowser, which is my primary reason for working in EDF format. My export costs only occur once per file occasionally, so I"m okay with slow exports as long as they facilitate a downstream data analysis. 

My export code at the time could've been incorrect tho.

Although maybe with the super fast data browser I've been peeking at, this could be eventually be irrelevant.
Exports with pyEDFlib worked fine when I tried it, all files opened in EDFbrowser with no problems.
Do you have a code snippet I can try?
This takes 30 sec, if I vectorize the loop over channels it takes 3 sec but produces an incorrect result (at least for my first attempt) -- probably need a new method to write multiple channels simultaneously
```
import mne
import numpy as np
info = mne.create_info(64, 1000., 'eeg')
data = np.random.RandomState(0).randn(64, 1000000)
raw = mne.io.RawArray(data, info)
raw.export('test.edf')
```
MNELAB exports using pyEDFlib: https://github.com/cbrnr/mnelab/blob/main/mnelab/io/writers.py#L46
> MNELAB exports using pyEDFlib: https://github.com/cbrnr/mnelab/blob/main/mnelab/io/writers.py#L46

It seems you never explicitly close the file, is that unnecessary or actually forgotten?
Probably forgotten, thanks for the bug report 😄!
I would like to add a second EDF export function that uses `pyEDFlib`, so export works if either that package or `EDFlib-Python` is installed. Is this possible with our utility functions? What is the preferred way to do this inside `_edf.py`? Somehow I'd like to have an explicit option to choose the package instead of trying to automatically decide based on installed packages (e.g. what happens if both packages are installed?).
The discussion above makes it seem like it's fine just to switch to `pyEDFlib`, which seems preferable over having two code paths. We can always add the old code back later if it turns out to be problematic to use `pyEDFlib` for some reason.
OK, then I'll just replace it with my implementation.
And by "just" I thought that it was going to be as easy as using my existing function, but I don't have the time to adapt all the tests and whatnot (I just tried). The export module is so convoluted that I can't "just" use my existing function. So if anyone wants to take this over please knock yourself out.
@cbrnr Do you still have your function? I could try adapting the tests to it.
Yes, it's here: https://github.com/cbrnr/mnelab/blob/main/mnelab/io/writers.py#L57

Since it's been 1.5 years since I last tried to work on that, it might be worth checking how much functionality has been added to EDF export in MNE which is not present in my implementation.
@cbrnr I wrote something that works using pyedflib. One problem is that the final output differs more than the test allows (test is `1e-5`):
```
absolute tolerance: 0.01

Not equal to tolerance rtol=0, atol=0.01

Mismatched elements: 8540 / 14400 (59.3%)
Max absolute difference: 0.02457271
Max relative difference: 0.00102393
 x: array([0.000000e+00, 1.664960e-03, 3.329920e-03, ..., 2.397043e+01,
       2.397210e+01, 2.397376e+01])
 y: array([0.000000e+00, 1.666667e-03, 3.333333e-03, ..., 2.399500e+01,
       2.399667e+01, 2.399833e+01])

absolute tolerance: 0.1
Success!
```
Is a tolerance of .1 enough? I am not sure why this is happening. It might have to do with the fact that the testing data has a non integer sampling rate and the physical maxes and mins are of high precision (pyedflib only accepts up to 4 decimal places). 
Which test is this? It depends on the scale of the data, but even if that's µV, I think a tolerance of 0.1 is not ... tolerable 😄.
It is `test_export_raw_edf`. I saw in your code you used `writeSamples`. In mine, I replace the current write function with [writePhysicalSamples](writePhysicalSamples). I am not sure why there is such a large difference in values...
In my own testing using `pyedflib.highlevel.write_edf`. I also got values within .1 uV. Is it all right if I make a pull request?
Yes, please submit a PR and we can discuss the tolerance issue there!
@cbrnr I made a PR whenever you have a chance to look :)
The first two issues have been fixed by switching to edfio. Now there's only one issue left, namely exporting STIM channels (I think MISC channels are already supported).

Resolving patch:
diff --git a/mne/export/_edf.py b/mne/export/_edf.py
--- a/mne/export/_edf.py
+++ b/mne/export/_edf.py
@@ -17,60 +17,28 @@
 def _export_raw(fname, raw, physical_range, add_ch_type):
     """Export Raw objects to EDF files.
 
-    TODO: if in future the Info object supports transducer or
-    technician information, allow writing those here.
+    TODO: if in future the Info object supports transducer or technician information,
+    allow writing those here.
     """
-    # get EEG-related data in uV
+    # get voltage-based data in uV
     units = dict(
         eeg="uV", ecog="uV", seeg="uV", eog="uV", ecg="uV", emg="uV", bio="uV", dbs="uV"
     )
 
-    digital_min = -32767
-    digital_max = 32767
+    digital_min, digital_max = -32767, 32767
 
     # load data first
     raw.load_data()
 
-    # remove extra STI channels
-    orig_ch_types = raw.get_channel_types()
-    drop_chs = []
-    if "stim" in orig_ch_types:
-        stim_index = np.argwhere(np.array(orig_ch_types) == "stim")
-        stim_index = np.atleast_1d(stim_index.squeeze()).tolist()
-        drop_chs.extend([raw.ch_names[idx] for idx in stim_index])
-        warn(f"Exporting STIM channels is not supported, dropping indices {stim_index}")
-
-    # Add warning if any channel types are not voltage based.
-    # Users are expected to only export data that is voltage based,
-    # such as EEG, ECoG, sEEG, etc.
-    # Non-voltage channels are dropped by the export function.
-    # Note: we can write these other channels, such as 'misc'
-    # but these are simply a "catch all" for unknown or undesired
-    # channels.
-    voltage_types = list(units) + ["stim", "misc"]
-    non_voltage_ch = [ch not in voltage_types for ch in orig_ch_types]
-    if any(non_voltage_ch):
-        warn(
-            f"Non-voltage channels detected: {non_voltage_ch}. MNE-Python's "
-            "EDF exporter only supports voltage-based channels, because the "
-            "EDF format cannot accommodate much of the accompanying data "
-            "necessary for channel types like MEG and fNIRS (channel "
-            "orientations, coordinate frame transforms, etc). You can "
-            "override this restriction by setting those channel types to "
-            '"misc" but no guarantees are made of the fidelity of that '
-            "approach."
-        )
-
-    ch_names = [ch for ch in raw.ch_names if ch not in drop_chs]
-    ch_types = np.array(raw.get_channel_types(picks=ch_names))
+    ch_types = np.array(raw.get_channel_types())
     n_times = raw.n_times
 
     # get the entire dataset in uV
-    data = raw.get_data(units=units, picks=ch_names)
+    data = raw.get_data(units=units)
 
-    # Sampling frequency in EDF only supports integers, so to allow for
-    # float sampling rates from Raw, we adjust the output sampling rate
-    # for all channels and the data record duration.
+    # Sampling frequency in EDF only supports integers, so to allow for float sampling
+    # rates from Raw, we adjust the output sampling rate for all channels and the data
+    # record duration.
     sfreq = raw.info["sfreq"]
     if float(sfreq).is_integer():
         out_sfreq = int(sfreq)
@@ -78,10 +46,9 @@ def _export_raw(fname, raw, physical_range, add_ch_type):
         # make non-integer second durations work
         if (pad_width := int(np.ceil(n_times / sfreq) * sfreq - n_times)) > 0:
             warn(
-                f"EDF format requires equal-length data blocks, "
-                f"so {pad_width / sfreq} seconds of "
-                "zeros were appended to all channels when writing the "
-                "final block."
+                "EDF format requires equal-length data blocks, so "
+                f"{pad_width / sfreq:.3g} seconds of zeros were appended to all "
+                "channels when writing the final block."
             )
             data = np.pad(data, (0, int(pad_width)))
     else:
@@ -90,15 +57,17 @@ def _export_raw(fname, raw, physical_range, add_ch_type):
         )
         out_sfreq = np.floor(sfreq) / data_record_duration
         warn(
-            f"Data has a non-integer sampling rate of {sfreq}; writing to "
-            "EDF format may cause a small change to sample times."
+            f"Data has a non-integer sampling rate of {sfreq}; writing to EDF format "
+            "may cause a small change to sample times."
         )
 
     # get any filter information applied to the data
     lowpass = raw.info["lowpass"]
     highpass = raw.info["highpass"]
     linefreq = raw.info["line_freq"]
-    filter_str_info = f"HP:{highpass}Hz LP:{lowpass}Hz N:{linefreq}Hz"
+    filter_str_info = f"HP:{highpass}Hz LP:{lowpass}Hz"
+    if linefreq is not None:
+        filter_str_info += " N:{linefreq}Hz"
 
     if physical_range == "auto":
         # get max and min for each channel type data
@@ -106,43 +75,41 @@ def _export_raw(fname, raw, physical_range, add_ch_type):
         ch_types_phys_min = dict()
 
         for _type in np.unique(ch_types):
-            _picks = [n for n, t in zip(ch_names, ch_types) if t == _type]
+            _picks = [n for n, t in zip(raw.ch_names, ch_types) if t == _type]
             _data = raw.get_data(units=units, picks=_picks)
             ch_types_phys_max[_type] = _data.max()
             ch_types_phys_min[_type] = _data.min()
     else:
         # get the physical min and max of the data in uV
-        # Physical ranges of the data in uV is usually set by the manufacturer
-        # and properties of the electrode. In general, physical max and min
-        # should be the clipping levels of the ADC input and they should be
-        # the same for all channels. For example, Nihon Kohden uses +3200 uV
-        # and -3200 uV for all EEG channels (which are the actual clipping
-        # levels of their input amplifiers & ADC).
-        # For full discussion, see: https://github.com/sccn/eeglab/issues/246
+        # Physical ranges of the data in uV are usually set by the manufacturer and
+        # electrode properties. In general, physical min and max should be the clipping
+        # levels of the ADC input, and they should be the same for all channels. For
+        # example, Nihon Kohden uses ±3200 uV for all EEG channels (corresponding to the
+        # actual clipping levels of their input amplifiers & ADC). For a discussion,
+        # see https://github.com/sccn/eeglab/issues/246
         pmin, pmax = physical_range[0], physical_range[1]
 
         # check that physical min and max is not exceeded
         if data.max() > pmax:
             warn(
-                f"The maximum μV of the data {data.max()} is "
-                f"more than the physical max passed in {pmax}.",
+                f"The maximum μV of the data {data.max()} is more than the physical max"
+                f" passed in {pmax}."
             )
         if data.min() < pmin:
             warn(
-                f"The minimum μV of the data {data.min()} is "
-                f"less than the physical min passed in {pmin}.",
+                f"The minimum μV of the data {data.min()} is less than the physical min"
+                f" passed in {pmin}."
             )
         data = np.clip(data, pmin, pmax)
     signals = []
-    for idx, ch in enumerate(ch_names):
+    for idx, ch in enumerate(raw.ch_names):
         ch_type = ch_types[idx]
         signal_label = f"{ch_type.upper()} {ch}" if add_ch_type else ch
         if len(signal_label) > 16:
             raise RuntimeError(
-                f"Signal label for {ch} ({ch_type}) is "
-                f"longer than 16 characters, which is not "
-                f"supported in EDF. Please shorten the "
-                f"channel name before exporting to EDF."
+                f"Signal label for {ch} ({ch_type}) is longer than 16 characters, which"
+                " is not supported by the EDF standard. Please shorten the channel name"
+                "before exporting to EDF."
             )
 
         if physical_range == "auto":
@@ -156,7 +123,7 @@ def _export_raw(fname, raw, physical_range, add_ch_type):
                 out_sfreq,
                 label=signal_label,
                 transducer_type="",
-                physical_dimension="uV",
+                physical_dimension="" if ch_type == "stim" else "uV",
                 physical_range=(pmin, pmax),
                 digital_range=(digital_min, digital_max),
                 prefiltering=filter_str_info,
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -40,6 +40,7 @@
     "TEMP": FIFF.FIFFV_TEMPERATURE_CH,
     "MISC": FIFF.FIFFV_MISC_CH,
     "SAO2": FIFF.FIFFV_BIO_CH,
+    "STIM": FIFF.FIFFV_STIM_CH,
 }
 
 
@@ -369,7 +370,7 @@ def _read_segment_file(data, idx, fi, start, stop, raw_extras, filenames, cals,
 
     # We could read this one EDF block at a time, which would be this:
     ch_offsets = np.cumsum(np.concatenate([[0], n_samps]), dtype=np.int64)
-    block_start_idx, r_lims, d_lims = _blk_read_lims(start, stop, buf_len)
+    block_start_idx, r_lims, _ = _blk_read_lims(start, stop, buf_len)
     # But to speed it up, we really need to read multiple blocks at once,
     # Otherwise we can end up with e.g. 18,181 chunks for a 20 MB file!
     # Let's do ~10 MB chunks:
diff --git a/mne/utils/docs.py b/mne/utils/docs.py
--- a/mne/utils/docs.py
+++ b/mne/utils/docs.py
@@ -1217,12 +1217,9 @@ def _reflow_param_docstring(docstring, has_first_line=True, width=75):
 )
 
 docdict["export_edf_note"] = """
-For EDF exports, only channels measured in Volts are allowed; in MNE-Python
-this means channel types 'eeg', 'ecog', 'seeg', 'emg', 'eog', 'ecg', 'dbs',
-'bio', and 'misc'. 'stim' channels are dropped. Although this function
-supports storing channel types in the signal label (e.g. ``EEG Fz`` or
-``MISC E``), other software may not support this (optional) feature of
-the EDF standard.
+Although this function supports storing channel types in the signal label (e.g.
+``EEG Fz`` or ``MISC E``), other software may not support this (optional) feature of the
+EDF standard.
 
 If ``add_ch_type`` is True, then channel types are written based on what
 they are currently set in MNE-Python. One should double check that all


Test patch:
diff --git a/mne/export/tests/test_export.py b/mne/export/tests/test_export.py
--- a/mne/export/tests/test_export.py
+++ b/mne/export/tests/test_export.py
@@ -169,17 +169,13 @@ def test_double_export_edf(tmp_path):
 
     # export once
     temp_fname = tmp_path / "test.edf"
-    with pytest.warns(RuntimeWarning, match="Exporting STIM channels"):
-        raw.export(temp_fname, add_ch_type=True)
+    raw.export(temp_fname, add_ch_type=True)
     raw_read = read_raw_edf(temp_fname, infer_types=True, preload=True)
 
     # export again
     raw_read.export(temp_fname, add_ch_type=True, overwrite=True)
     raw_read = read_raw_edf(temp_fname, infer_types=True, preload=True)
 
-    # stim channel should be dropped
-    raw.drop_channels("2")
-
     assert raw.ch_names == raw_read.ch_names
     assert_array_almost_equal(raw.get_data(), raw_read.get_data(), decimal=10)
     assert_array_equal(raw.times, raw_read.times)
@@ -257,19 +253,19 @@ def test_rawarray_edf(tmp_path):
 
 
 @edfio_mark()
-def test_edf_export_warns_on_non_voltage_channels(tmp_path):
+def test_edf_export_non_voltage_channels(tmp_path):
     """Test saving a Raw array containing a non-voltage channel."""
     temp_fname = tmp_path / "test.edf"
 
     raw = _create_raw_for_edf_tests()
     raw.set_channel_types({"9": "hbr"}, on_unit_change="ignore")
-    with pytest.warns(RuntimeWarning, match="Non-voltage channels"):
-        raw.export(temp_fname, overwrite=True)
+    raw.export(temp_fname, overwrite=True)
 
     # data should match up to the non-accepted channel
     raw_read = read_raw_edf(temp_fname, preload=True)
     assert raw.ch_names == raw_read.ch_names
     assert_array_almost_equal(raw.get_data()[:-1], raw_read.get_data()[:-1], decimal=10)
+    assert_array_almost_equal(raw.get_data()[-1], raw_read.get_data()[-1], decimal=5)
     assert_array_equal(raw.times, raw_read.times)
 
 
@@ -291,6 +287,7 @@ def test_measurement_date_outside_range_valid_for_edf(tmp_path):
         raw.export(tmp_path / "test.edf", overwrite=True)
 
 
+@pytest.mark.filterwarnings("ignore:Data has a non-integer:RuntimeWarning")
 @pytest.mark.parametrize(
     ("physical_range", "exceeded_bound"),
     [