Repository: mne-tools/mne-python

Issue:
Add support for XDF (labstreaminglayer format)
I think this should be added. I am happy to do this. There is already a python loader for the xdf format so it is just a matter of making output structure compliant with mne.


Discussion:
Yes, this would be a really nice addition! I'm happy to help if you get stuck.
would be great

Ok. I'll put this on my todo list and submit a PR when it is done.

On 5/2/2018 12:22 PM, Alexandre Gramfort wrote:
> would be great
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub 
> <https://github.com/mne-tools/mne-python/issues/5180#issuecomment-385932177>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/ADch7pyowDrCPFbYd9Yc4VZzA5YiYQ-Sks5tuYjNgaJpZM4TvMeo>.
>


That's a good idea. In the repo for the python XDF loader, there is an ongoing discussion to make a "pyxdf" package, however the discussion is stagnating a bit. Mainly because there is a lack of tests for the robustness/reliability of the loader. See [here](https://github.com/sccn/xdf/pull/10) for the discussion.

Integrating the xdf loader into MNE would not circumvent the issue of (so far) lacking tests, but maybe the stagnation is over now :-) 
@sappelhoff I think MNE would be a great place for the XDF loader!
@dmedine it might not be just as easy as putting the output into a different data structure. Due to the nature of XDF, streams are not necessarily regularly sampled. However, MNE expects all signals to have a fixed sampling frequency. Furthermore, I think that all streams must actually have the same sampling frequency (because everything is stored in a NumPy array) - but I'm not completely sure about that. In any case, this means that some kind of resampling will be necessary.
I thought that we discussed this somewhere, but a low-order spline interp (or even linear!) would probably be sufficient. So I don't see this as too big a hurdle.
Yes, it shouldn't be too difficult, just wanted to bring this up in case this was not obvious. Can you confirm that all signals must have the same sampling frequency? If so, how do we choose which sampling rate to resample to? Most XDF files have nominal sampling rates for each stream, so we could resample to the highest.
> Can you confirm that all signals must have the same sampling frequency?

Yes

> If so, how do we choose which sampling rate to resample to? 

Resampling to the highest data channel rate (by default, we could make different options with ``resample='auto'`` kwarg) makes sense to me. In practice I imagine these will all usually match anyway.
There are some developments with the python reader for XDF files. Soon, there might be a pip installable package, see: https://github.com/sccn/xdf/pull/21

That would make this issue easier to solve.
this is a great idea, and we would be happy to support this initiative! 
@dmedine @trmullen do you have time to work on this?
@cbrnr check out the discussion in the XDF issue linked above, specifically there is probably the need for some MNE decision on this statement from @dmedine:

> I said a long time ago that I would shove xdf into MNE. The issue there (if I understand correctly) is that if there are multiple streams, they all have to have the same sampling rate, first time-stamp, etc. This makes it labor intensive enough for me to avoid doing it. **However, if MNE is like EEGLab and it is content to grab only EEG and marker streams, then this might not be such a chore after all.**

@larsoner @agramfort 

Personally, I think that being able to extract an EEG stream from XDF is better than nothing and worth an implementation. An even better solution would be to allow the extraction of *any* single stream out of the XDF file.

... the "best solution" of reading the complete XDF file with all its streams seems tricky indeed
I thought that the existing XDF reader already supports synchronizing time stamps from different streams, so this is one problem already solved. Grabbing a single stream is straightforward, because all streams are returned in a list of dicts, of which we'd need the values for the `'time_series'` and `'time_stamps'` keys.
MNE only supports "streams" with the same sampling rates. So yes if we can
read the EEG data and mimic what we have already for edf files +1

I also feel that we need more integrated efforts between pyedf, pybv,
pyxdf, neo and MNE to avoid duplication of efforts. I am sure I have time
though to coordinate these efforts. It would be a great topic for a code
sprint.

Yes, absolutely! I also have the feeling that we could greatly benefit from closer collaboration. I'm already involved in MNE and pyxdf, so I could at least try to coordinate a bit.
Just to clarify, the XDF reader does synchronize the time stamps in that 
it puts all the streams on the same timebase, but it does not resample 
the data. So time 3.14 in stream A is the same as 3.14 in stream B, but 
they might have completely different sampling rates. Is this also an 
issue for markers, by the way?

On 14.11.2018 10:02, Clemens Brunner wrote:
>
> I thought that the existing XDF reader already supports synchronizing 
> time stamps from different streams, so this is one problem already 
> solved. Grabbing a single stream is straightforward, because all 
> streams are returned in a list of dicts, of which we'd need the values 
> for the |'time_series'| and |'time_stamps'| keys.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/mne-tools/mne-python/issues/5180#issuecomment-438587766>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/ADch7kz9NpZprmcTNmqUNFctNJIQlSZaks5uu9wNgaJpZM4TvMeo>.
>


I currently don't have time to work on this, but I can probably devote 
some in the future.

On 14.11.2018 09:42, Clemens Brunner wrote:
>
> @dmedine <https://github.com/dmedine> @trmullen 
> <https://github.com/trmullen> do you have time to work on this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/mne-tools/mne-python/issues/5180#issuecomment-438582178>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/ADch7iSwBJUWohTQ06__rUaTKnY0oMVtks5uu9digaJpZM4TvMeo>.
>


Resampling to a common sampling rate shouldn't be a big issue as long as we start from tmin and include all time points up to tmax.
for my curiosity how do you remap to the same time stamps? do you
interpolate? how do you decide the regular grid of time samples?

Yes, I'd resample all signals to the highest sampling rate, this will create samples at the same time points. In SigViewer, we do that, but the user can change the resampling rate if the one we choose automatically is too high. If the starting points differ, we add zeros to the signals starting later.
The way LSL puts everything on the same time base is fairly complicated. 
I will try to summarize it in broad strokes.

Basically, when data is being streamed by an outlet into an inlet, there 
are periodic handshakes between inlet and outlet that return the time 
difference between the CPU clock times between the two. The algorithm 
for doing this is canonical and based on the NTP method for doing the 
same. In the LSL recording program, these offsets are written into the 
footer of the data file as the streams are being recorded. Meanwhile 
each datum is also timestamped based on its own CPU time. Then, when the 
file is loaded, the loading program (implemented in Matlab and Python) 
uses this information to form a piecewise linear map between the 
timestamps in each incoming stream and the write times in the recording. 
This corrects the slowly drifting time offsets between various sending 
CPU times and the CPU time of the recording.

If the stream is coming in at a regularly sampled sampling rate (e.g. an 
EEG stream, not an arbitrary marker stream) the differences between 
successive timestamps are also smoothed to correct for jitter. The end 
result is a data structure that has the data for each stream and its 
corresponding, corrected, timestamp values. These streams are then said 
to be synchronized. This has been validated to provide sub-millisecond 
accuracy. This doesn't align the samples to a common grid. That would, 
of course, require an interpolation step, which is the part that I don't 
have time to work on right now. This would, now that I think about it, 
perhaps be a nice feature to add to the LSL loading software. So far as 
I know, no one has ever requested this.

What isn't corrected for (it must be measured) is the lag time between 
hardware acquisition and the time the data gets timestamped. There are 
various means of measuring these relative latencies, but it isn't always 
so straightforward. Back in the day this required a fair amount of 
handmade microphones, photo-diodes, and buckets of salt water. Brain 
Products (shameless plug) sells hardware (StimTrak 
<https://www.brainproducts.com/productdetails.php?id=57>) that can 
provide these measurements through sending hardware triggers. NBS also 
sells a product (which is LSL based) called the LabStreamer 
<https://www.neurobs.com/menu_order/purchase_labstreamer> that helps 
measure these hardware latencies.

One last detail is that LabRecorder (the LSL recording program) picks up 
each stream on a separate thread so they always have a different start time.

On 14.11.2018 15:19, Clemens Brunner wrote:
>
> Yes, I'd resample all signals to the highest sampling rate, this will 
> create samples at the same time points. In SigViewer, we do that, but 
> the user can change the resampling rate if the one we choose 
> automatically is too high. If the starting points differ, we add zeros 
> to the signals starting later.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/mne-tools/mne-python/issues/5180#issuecomment-438677313>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/ADch7oVNgMNFuUxdHZMgBNNe9bkAEw9cks5uvCZsgaJpZM4TvMeo>.
>


thanks a lot for clarifying

then for MNE what we need is a way to resample all streams we want to read
to the same sampling freq
like we do for EDF reader.

Hi @cbrnr -- yes happy to assist with this effort.

Tim


On Wed, Nov 14, 2018, 12:42 AM Clemens Brunner <notifications@github.com
wrote:

> @dmedine <https://github.com/dmedine> @trmullen
> <https://github.com/trmullen> do you have time to work on this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/5180#issuecomment-438582178>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADq4m9UHjP0N8fGf0ttrXEC-tvhmlcThks5uu9digaJpZM4TvMeo>
> .
>

Not adding much to the discussion. But cross linking to [related discussion](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/bids-discussion/NeqFO-lJhsc/_atcTFfIBQAJ) on BIDS mailing list
pyxdf is now available with `pip install pyxdf`
As an initial implementation, what about allowing only one regularly sampled stream (plus optionally any number of marker streams)? That way, we don't have to resample and we can directly use time stamps and data samples provided by `load_xdf`. Optional marker streams could be converted to annotations. This would probably cover many use cases already, and would be a lot better than what we currently have (nothing).
>
> As an initial implementation, what about allowing only one regularly
> sampled stream (plus optionally any number of marker streams)? That way, we
> don't have to resample and we can directly use time stamps and data samples
> provided by load_xdf. Optional marker streams could be converted to
> annotations. This would probably cover many use cases already, and would be
> a lot better than what we currently have (nothing)
>
no objection but I would start with a gist

Here's a first proof-of-concept implementation: https://gist.github.com/cbrnr/7164a12868bc7616056587a41f51e92d

Basically, it loads the first EEG stream it finds and adds the first marker stream if present. For many XDF files, https://github.com/sccn/xdf/pull/42 should be fixed before this works. Comments and suggestions welcome!
would this work with preload=False?

>

Because XDF is a format designed for streamed data, `preload=False` would require to parse the whole file. This is certainly doable, but I think it should be implemented upstream. What other requirements does MNE impose on readers? Blockwise random access reading?
> What other requirements does MNE impose on readers? Blockwise random access reading?

In general yes, one should at a minimum (that I can think of):

1. Populate `info` properly,
2. Give the number of samples in the file, and
3. Provide a function that returns an arbitrary chunk of data for a given `start, stop` sample number pair

If these are provided, all methods of `BaseRaw` should work (I think!).

If they are not / cannot be provided, we'd have to think about how to deal with this.
I think all of these points can be implemented, but only with `preload=True`. XDF makes it really hard to randomly access a portion of data, because it does not reserve a constant amount of bytes per data segment. In any case, jumping around the file is still reasonably fast using boundary chunks, but since these are optional the worst case scenario is that the file must be searched from beginning to end. All of this should be provided upstream, but currently `load_xdf` always loads everything.
pyedf or pyxdf don't support random access of chunks at all? I don't have a
strong feeling on this but that would make a precedent to have readers that
don't support preload.

>

`pyedflib` does support random access, that is you can specify the channel, start time, and end time. I think XDF is really an exception here - I don't know any other file format that's structured similarly, so this might be the only reader for now.
I've added a `parse_xdf` function that does a quick pass over the whole file to read just the headers (but ignore all actual content). This is very fast even for large files (fractions of a second) as compared to the complete `load_xdf` call, which can take several seconds. Maybe we can use `parse_xdf` to fill our meta info even in the case of `preload=False`. But once data is actually needed, we'd need to pull in everything, so it's just a simple lazy loading without random access.
not strong feeling here. Do you need this to process your own data? or some
public data?

>

I don't even work with XDF myself, but I have been involved with the format and I also needed it in SigViewer. But I'm not sure if this was your question...
> In general yes, one should at a minimum (that I can think of):
>
> 1.    Populate info properly,
> 2.  Give the number of samples in the file, and
> 3.    Provide a function that returns an arbitrary chunk of data for a given start, stop sample number pair

All of these require at least one full pass over the file because the number of streams and samples isn't known in advance. Also, with postprocessing the number of samples might change, even for evenly sampled streams.

I'm not an expert, but I think the postprocessing in the current implementation needs all samples and can't just take a time slice, select all chunks which fall into that range and then read only those samples. Plus, the default of a few seconds of data per chunk means that most operations will need to read multiple chunks so preloading might make everything a lot simpler and faster, even if only a subset of data is needed.
> All of these require at least one full pass over the file because the number of streams and samples isn't known in advance. Also, with postprocessing the number of samples might change, even for evenly sampled streams.

Yes, but the pass is much quicker with `parse_xdf` (i.e. when you don't process the contents). Maybe we can use this to fill our meta info when `preload=False`.

> I'm not an expert, but I think the postprocessing in the current implementation needs all samples and can't just take a time slice, select all chunks which fall into that range and then read only those samples. Plus, the default of a few seconds of data per chunk means that most operations will need to read multiple chunks so preloading might make everything a lot simpler and faster, even if only a subset of data is needed.

Correct, once you start working with the actual data, everything needs to be read and processed (clock synchronization). So basically random access just won't be possible.
BTW, `read_raw_xdf` in https://gist.github.com/cbrnr/7164a12868bc7616056587a41f51e92d also supports loading arbitrary streams by name (and since names need not be unique I could/should also add loading by stream ID). If no name is provided, the first EEG stream is used.
I'm okay with only allowing preload=True or string for this format then
in `0.17.1` this is already the case that we can only read with `preload=True`. i.e: edf files with annotations.
ok fair enough then. I assume we can use as soft dep
https://pypi.org/project/pyxdf/ ?

We could, but this should be discussed again. It looks like MNE is much more actively maintained than pyxdf, so we could also think about integrating the XDF code directly into MNE. I'd prefer this.
I would prefer to boost the maintenance team of pyxdf and eventually pyedf
to distrbute the maintenance load. It would scale if we always integrate
the community work.

>

Couldn't you get write permissions for pyxdf @cbrnr ? That would at least remove potential "blocks" due to inactivity from the reviewing / merging side of PRs at pyxdf?

I'd hope that I could (@cboulay WDYT?), but I can't just rage-merge everything without review 😛 - but it would certainly help if I was more directly involved.

BTW, is there a way I can list all GitHub repos I have write access to?
I've been explicitly told that I cannot add any new people to SCCN repos. I recently asked for help and guidance, because it's clear that xdf requires more stewardship than it currently receives (and I don't have time to give), and I received a couple "Chad's right, we need more work on this" and a couple no-answers.

One of my ideas, which was not well received by the xdf oversight committee (*real people but not a real committee), is to break out xdf into submodules just how we did it for labstreaminglayer. Then we can have individuals working with more authority on the code while not granting them the ability to break SCCN's repositories. i.e., any changes that happen upstream in these submodules will not be visible to sccn/xdf users until someone with SCCN authority updates the submodule pointers.

Even though there was resistance to this, I went ahead and created the xdf-modules GitHub organization and put in a few placeholders for submodules it will hold. I'm just asking around now about how to move the code without losing the commit history.

I've already invited a few people to the organization, including @cbrnr , but there isn't much to be done yet until we get the code + commit history moved over.
let me share my thoughts on this. If you want xdf to become a community
standard you need to make pyxdf a project that is not affiliated to SCCN.

if you can move the code to a more neutral org it's great. thanks a lot

>

I think creating separate repos is a step in the right direction (thanks @cboulay for adding me) - I always thought that `pyxdf` shouldn't be combined with MATLAB (and potentially other stuff). I agree with @agramfort, but that's exactly what @cboulay just did - if the new upstream will be xdf-modules/pyxdf, then it is essentially detached from SCCN. In any case, since at least the pyxdf code is BSD licensed, we can always fork it and roll our own thing (that's why I was suggesting to do it in MNE, it's really not that much more to maintain).
forking and maintaining our own code is bad community-wise. It make the
community more fragmented while we should consolidate

>

Yes, I meant migrating and not forking.
@cbrnr you now have write access to https://github.com/xdf-modules/xdf-Python
I will ask the other people who have made Python pull requests to move their PRs to that repo instead.
Thanks. BTW, I've never migrated/transferred a repo, but shouldn't it be possible with the standard git commands (i.e. cloning and pushing to the new repo)? https://www.atlassian.com/git/tutorials/git-move-repository
I've implemented a basic XDF reader in MNELAB:
https://github.com/cbrnr/mnelab/pull/22

There are some caveats (see my comment in that PR), but loading basic EEG streams should work. I'm not sure if we can handle loading multiple streams, because everything gets very slow even with just one long stream (with a high number of channels/samples, e.g. 80 channels, fs 1000Hz, 10 minutes).
@cbrnr very cool. it looks like we might be able to incorporate the lazy evaluation of preload and seek on demand. with xdf, can streams be added during recording or we do always know that it would contain certain streams and only those stream throughout?
@teonbrooks New streams can appear anywhere in the file, but per spec the header for the stream has to appear before the data chunks.
I guess then you would need to be able to read all the headers first to and construct a map of pointers (sample_n --> data_block_n) for the eeg if you wanted to avoid loading the whole file
Yes, it is possible to parse the whole file and just consider the data of the stream you want (and skip over the data you don't want), because each stream has a unique stream ID. That's something I'm already doing with `parse_xdf`, but the current version of `pyxdf` (which I use to actually load data) doesn't support this yet.
@agramfort just thought about your comment - when you say pyxdf should not apparently be associated with SCCN, why is this not a problem with MNE Python hosted at Martinos?
>
> y@agramfort <https://github.com/agramfort> just thought about your
> comment - when you say pyxdf should not apparently be associated with SCCN,
> why is this not a problem with MNE Python hosted at Martinos?
>

mne-python is not hosted on an github org of martinos and it's not
administrated by anyone at martinos
we have the website historically hosted at martinos.org. We use
mne-tools.github.io as nightly build of the doc.
It's true that we have martinos in the url of the website but we have all
the logos of all the institutions
that helped over the years.

Hello, 

I would like to know what the status is on this feature in 2020, as it is still open. Are you still planning to include this at some point? I would be very interested in having this feature built into MNE.

Alternatively, it would be nice if you include a link to relevant software in the documentation if you are not planning to include this in MNE. That way people looking for a solution don't have to go to google and read through this issue. 
Currently only [MNELAB](https://github.com/cbrnr/mnelab) supports loading XDF. We don't know if and when this feature will be available in MNE, because there are a couple of open questions (e.g. how do we handle multiple streams with different sampling frequencies). However, in the meantime you can use either [pyxdf](https://github.com/xdf-modules/pyxdf) or even the reader in MNELAB (`from mnelab.io import read_raw`) if you want to work with XDF files in MNE-Python.
FYI I just used `pyxdf` + `mne.io.RawArray` to read an XDF-formatted file (recorded using LSL stream recorder). I think this is what we should probably "support" from the MNE end so that people format data the way they need. @cbrnr would you be up for making a PR to mention using MNELAB and/or `pyxdf` directly to load data and using `mne.io.RawArray`? To me it's the cleanest solution since there are so many ways people's data can show up in XDF format.
Sure. What would you like me to implement? Just an update to our docs (where)? I'm using the `mne.io.RawArray` approach in [read_raw_xdf](https://github.com/cbrnr/mnelab/blob/caa18e02b42ddd44c8737fe58fe3e7287eb674ac/mnelab/io/xdf.py#L9), so this could be some sort of reference implementation which we could include directly in MNE. However, this supports only one particular stream which users have to select.
> Just an update to our docs (where)? 

Probably update https://mne.tools/stable/auto_tutorials/io/plot_20_reading_eeg_data.html

> I'm using the mne.io.RawArray approach in MNELAB, so this could be some sort of reference implementation which we could include directly in MNE. However, this supports only one particular stream which users have to select.

Yeah I'd rather just show people a little code snippet, probably we are using the same things like:
```
data, header = pyxdf.load_xdf(fn_eeg)
data = data[0]['time_series'].T
info = mne.create_info(...)
raw = mne.io.RawArray(data, info)
```

> this could be some sort of reference implementation which we could include directly in MNE

I'd rather not include an actual implementation in MNE because there is too much flexibility in the format, I'd rather just update the docs to say "something like this might work for your use case" and then people can adapt appropriately based on the data they have.
For cases like this (reading in data from a source type that we don't fully support), I think they make more sense as a how-to example rather than integrated as part of the tutorials.  The NEO how-to is an example of this.

**EDIT** On the other hand, that tutorial is a bit unlike the others in that it's a sort of laundry-list. I think I would still lean toward limiting it to the data formats that we fully support (i.e., have readers for) but maybe add a section to the end called "other formats" where we give general comments about `RawArray` and link out to a couple specific how-to examples like ones for NEO and LSL.
If it is a separate tutorial, I think we should at least link to it from the `plot_20_reading_eeg_data.html`, since that's where people will probably naturally end up if they look into file formats in MNE
agree... I was just editing my above comment to say something similar.
@larsoner OK, this tutorial might be a good place to mention it.

I've linked to my reader in my previous comment, but yeah, I'm basically just doing what you are using. However, I'm also reading custom units as well as marker channels, which I convert to annotations. I would mention my function in addition to some general code snippet, because people can just go ahead and use `from mnelab.io.xdf import read_raw_xdf`, which is probably OK for many use cases.

Resolving patch:
diff --git a/examples/io/plot_read_xdf.py b/examples/io/plot_read_xdf.py
new file mode 100644
--- /dev/null
+++ b/examples/io/plot_read_xdf.py
@@ -0,0 +1,42 @@
+"""
+.. _ex-read-xdf:
+
+====================
+Reading XDF EEG data
+====================
+
+Here we read some sample XDF data. Although we do not analyze it here, this
+recording is of a short parallel auditory response (pABR) experiment
+:footcite:`PolonenkoMaddox2019` and was provided by the `Maddox Lab
+<https://www.urmc.rochester.edu/labs/maddox.aspx>`__.
+"""
+# Authors: Clemens Brunner <clemens.brunner@gmail.com>
+#          Eric Larson <larson.eric.d@gmail.com>
+#
+# License: BSD (3-clause)
+
+import os.path as op
+
+import pyxdf
+
+import mne
+from mne.datasets import misc
+
+fname = op.join(
+    misc.data_path(), 'xdf',
+    'sub-P001_ses-S004_task-Default_run-001_eeg_a2.xdf')
+streams, header = pyxdf.load_xdf(fname)
+data = streams[0]["time_series"].T
+assert data.shape[0] == 5  # four raw EEG plus one stim channel
+data[:4:2] -= data[1:4:2]  # subtract (rereference) to get two bipolar EEG
+data = data[::2]  # subselect
+data[:2] *= (1e-6 / 50 / 2)  # uV -> V and preamp gain
+sfreq = float(streams[0]["info"]["nominal_srate"][0])
+info = mne.create_info(3, sfreq, ["eeg", "eeg", "stim"])
+raw = mne.io.RawArray(data, info)
+raw.plot(scalings=dict(eeg=100e-6), duration=1, start=12)
+
+###############################################################################
+# References
+# ----------
+# .. footbibliography::
diff --git a/mne/datasets/utils.py b/mne/datasets/utils.py
--- a/mne/datasets/utils.py
+++ b/mne/datasets/utils.py
@@ -254,7 +254,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
     path = _get_path(path, key, name)
     # To update the testing or misc dataset, push commits, then make a new
     # release on GitHub. Then update the "releases" variable:
-    releases = dict(testing='0.117', misc='0.8')
+    releases = dict(testing='0.117', misc='0.9')
     # And also update the "md5_hashes['testing']" variable below.
     # To update any other dataset, update the data archive itself (upload
     # an updated version) and update the md5 hash.
@@ -345,7 +345,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
             bst_raw='fa2efaaec3f3d462b319bc24898f440c',
             bst_resting='70fc7bf9c3b97c4f2eab6260ee4a0430'),
         fake='3194e9f7b46039bb050a74f3e1ae9908',
-        misc='0f88194266121dd9409be94184231f25',
+        misc='f832ce9c4c27e83396cc977b293b0aa9',
         sample='12b75d1cb7df9dfb4ad73ed82f61094f',
         somato='32fd2f6c8c7eb0784a1de6435273c48b',
         spm='9f43f67150e3b694b523a21eb929ea75',
diff --git a/tutorials/io/plot_20_reading_eeg_data.py b/tutorials/io/plot_20_reading_eeg_data.py
--- a/tutorials/io/plot_20_reading_eeg_data.py
+++ b/tutorials/io/plot_20_reading_eeg_data.py
@@ -6,8 +6,8 @@
 Importing data from EEG devices
 ===============================
 
-MNE includes various functions and utilities for reading EEG
-data and electrode locations.
+MNE includes various functions and utilities for reading EEG data and electrode
+locations.
 
 .. _import-bv:
 
@@ -16,50 +16,51 @@
 
 The BrainVision file format consists of three separate files:
 
-1. A text header file (``.vhdr``) containing meta data
+1. A text header file (``.vhdr``) containing meta data.
 2. A text marker file (``.vmrk``) containing information about events in the
-   data
-3. A binary data file (``.eeg``) containing the voltage values of the EEG
+   data.
+3. A binary data file (``.eeg``) containing the voltage values of the EEG.
 
-Both text files are based on the
-`Microsoft Windows INI format <https://en.wikipedia.org/wiki/INI_file>`_
-consisting of:
+Both text files are based on the `INI format <https://en.wikipedia.org/wiki/INI_file>`_
+consisting of
 
-* sections marked as ``[square brackets]``
-* comments marked as ``; comment``
-* key-value pairs marked as ``key=value``
+* sections marked as ``[square brackets]``,
+* comments marked as ``; comment``,
+* and key-value pairs marked as ``key=value``.
 
-A documentation for core BrainVision file format is provided by Brain Products.
-You can view the specification hosted on the
-`Brain Products website <https://www.brainproducts.com/productdetails.php?id=21&tab=5>`_
+Brain Products provides documentation for their core BrainVision file format.
+The format specification is hosted on the
+`Brain Products website <https://www.brainproducts.com/productdetails.php?id=21&tab=5>`_.
 
-BrainVision EEG files can be read in using :func:`mne.io.read_raw_brainvision`
-with the ``.vhdr`` header file as an input.
+BrainVision EEG files can be read using :func:`mne.io.read_raw_brainvision`,
+passing the ``.vhdr`` header file as the argument.
 
 .. warning:: Renaming BrainVision files can be problematic due to their
-             multifile structure. See this
+             multi-file structure. See this
              `example <https://mne.tools/mne-bids/stable/auto_examples/rename_brainvision_files.html#sphx-glr-auto-examples-rename-brainvision-files-py>`_
-             for an instruction.
+             for instructions.
 
 .. note:: For *writing* BrainVision files, you can use the Python package
           `pybv <https://pypi.org/project/pybv/>`_.
 
+
 .. _import-edf:
 
 European data format (.edf)
 ===========================
 
-EDF and EDF+ files can be read using :func:`mne.io.read_raw_edf`.
-
-`EDF (European Data Format) <http://www.edfplus.info/specs/edf.html>`_ and
-`EDF+ <http://www.edfplus.info/specs/edfplus.html>`_ are 16-bit formats.
+`EDF <http://www.edfplus.info/specs/edf.html>`_ and
+`EDF+ <http://www.edfplus.info/specs/edfplus.html>`_ files can be read using
+:func:`mne.io.read_raw_edf`. Both variants are 16-bit formats.
 
-The EDF+ files may contain an annotation channel which can be used to store
-trigger information. These annotations are available in ``raw.annotations``.
+EDF+ files may contain annotation channels which can be used to store trigger
+and event information. These annotations are available in ``raw.annotations``.
 
-Saving EDF files is not supported natively yet. `This gist
-<https://gist.github.com/skjerns/bc660ef59dca0dbd53f00ed38c42f6be>`__
-can be used to save any mne.io.Raw into EDF/EDF+/BDF/BDF+.
+Writing EDF files is not supported natively yet. `This gist
+<https://gist.github.com/skjerns/bc660ef59dca0dbd53f00ed38c42f6be>`__ or
+`MNELAB <https://github.com/cbrnr/mnelab>`_ (both of which use
+`pyedflib <https://github.com/holgern/pyedflib>`_ under the hood) can be used
+to export any :class:`mne.io.Raw` object to EDF/EDF+/BDF/BDF+.
 
 
 .. _import-biosemi:
@@ -72,18 +73,18 @@
 be imported with :func:`mne.io.read_raw_bdf`.
 
 BioSemi amplifiers do not perform "common mode noise rejection" automatically.
-The signals in the EEG file are the voltages between each electrode and CMS
+The signals in the EEG file are the voltages between each electrode and the CMS
 active electrode, which still contain some CM noise (50 Hz, ADC reference
-noise, etc., see `the BioSemi FAQ <https://www.biosemi.com/faq/cms&drl.htm>`__
-for further detail).
-Thus, it is advisable to choose a reference (e.g., a single channel like Cz,
-average of linked mastoids, average of all electrodes, etc.) on import of
+noise, etc.). The `BioSemi FAQ <https://www.biosemi.com/faq/cms&drl.htm>`__
+provides more details on this topic.
+Therefore, it is advisable to choose a reference (e.g., a single channel like Cz,
+average of linked mastoids, average of all electrodes, etc.) after importing
 BioSemi data to avoid losing signal information. The data can be re-referenced
 later after cleaning if desired.
 
-.. warning:: The data samples in a BDF file are represented in a 3-byte
+.. warning:: Data samples in a BDF file are represented in a 3-byte
              (24-bit) format. Since 3-byte raw data buffers are not presently
-             supported in the fif format these data will be changed to 4-byte
+             supported in the FIF format, these data will be changed to 4-byte
              integers in the conversion.
 
 
@@ -92,7 +93,7 @@
 General data format (.gdf)
 ==========================
 
-GDF files can be read in using :func:`mne.io.read_raw_gdf`.
+GDF files can be read using :func:`mne.io.read_raw_gdf`.
 
 `GDF (General Data Format) <https://arxiv.org/abs/cs/0608052>`_ is a flexible
 format for biomedical signals that overcomes some of the limitations of the
@@ -100,26 +101,26 @@
 and uses an event table. An updated specification (GDF v2) was released in
 2011 and adds fields for additional subject-specific information (gender,
 age, etc.) and allows storing several physical units and other properties.
-Both specifications are supported in MNE.
+Both specifications are supported by MNE.
 
 
 .. _import-cnt:
 
-Neuroscan CNT data format (.cnt)
-================================
+Neuroscan CNT (.cnt)
+====================
 
-CNT files can be read in using :func:`mne.io.read_raw_cnt`.
-The channel locations can be read from a montage or the file header. If read
+CNT files can be read using :func:`mne.io.read_raw_cnt`.
+Channel locations can be read from a montage or the file header. If read
 from the header, the data channels (channels that are not assigned to EOG, ECG,
-EMG or misc) are fit to a sphere and assigned a z-value accordingly. If a
+EMG or MISC) are fit to a sphere and assigned a z-value accordingly. If a
 non-data channel does not fit to the sphere, it is assigned a z-value of 0.
 
 .. warning::
     Reading channel locations from the file header may be dangerous, as the
-    x_coord and y_coord in ELECTLOC section of the header do not necessarily
-    translate to absolute locations. Furthermore, EEG-electrode locations that
+    x_coord and y_coord in the ELECTLOC section of the header do not necessarily
+    translate to absolute locations. Furthermore, EEG electrode locations that
     do not fit to a sphere will distort the layout when computing the z-values.
-    If you are not sure about the channel locations in the header, use of a
+    If you are not sure about the channel locations in the header, using a
     montage is encouraged.
 
 
@@ -128,31 +129,33 @@
 EGI simple binary (.egi)
 ========================
 
-EGI simple binary files can be read in using :func:`mne.io.read_raw_egi`.
-The EGI raw files are simple binary files with a header and can be exported
-from using the EGI Netstation acquisition software.
+EGI simple binary files can be read using :func:`mne.io.read_raw_egi`.
+EGI raw files are simple binary files with a header and can be exported by the
+EGI Netstation acquisition software.
 
 
 .. _import-mff:
 
 EGI MFF (.mff)
 ==============
-These files can also be read with :func:`mne.io.read_raw_egi`.
+
+EGI MFF files can be read with :func:`mne.io.read_raw_egi`.
 
 
 .. _import-set:
 
-EEGLAB set files (.set)
-=======================
+EEGLAB files (.set, .fdt)
+=========================
 
-EEGLAB .set files can be read in using :func:`mne.io.read_raw_eeglab`
-and :func:`mne.read_epochs_eeglab`.
+EEGLAB .set files (which sometimes come with a separate .fdt file) can be read
+using :func:`mne.io.read_raw_eeglab` and :func:`mne.read_epochs_eeglab`.
 
 
 .. _import-nicolet:
 
 Nicolet (.data)
 ===============
+
 These files can be read with :func:`mne.io.read_raw_nicolet`.
 
 
@@ -161,8 +164,8 @@
 eXimia EEG data (.nxe)
 ======================
 
-EEG data from the Nexstim eXimia system can be read in using the
-:func:`mne.io.read_raw_eximia` function.
+EEG data from the Nexstim eXimia system can be read with
+:func:`mne.io.read_raw_eximia`.
 
 
 .. _import-persyst:
@@ -170,15 +173,15 @@
 Persyst EEG data (.lay, .dat)
 =============================
 
-EEG data from the Persyst system can be read in using the
-:func:`mne.io.read_raw_persyst` function.
+EEG data from the Persyst system can be read with
+:func:`mne.io.read_raw_persyst`.
+
+Note that subject metadata may not be properly imported because Persyst
+sometimes changes its specification from version to version. Please let us know
+if you encounter a problem.
 
-Note that not all the subject metadata may be properly read in
-due to the fact that Persyst changes its specification
-sometimes from version to version. Please submit an issue, or
-pull request if you encounter a problem.
 
-Nihon Kohden EEG data (.EEG, .21E, .PNT, .LOG)
+Nihon Kohden EEG data (.eeg, .21e, .pnt, .log)
 ==============================================
 
 EEG data from the Nihon Kohden (NK) system can be read using the
@@ -186,20 +189,38 @@
 
 Files with the following extensions will be read:
 
-- The ``.EEG`` file contains the actual raw EEG data.
-- The ``.PNT`` file contains the metadata related to the recording, such
-  as the measurement date.
-- The ``.LOG`` file contains annotations for the recording.
-- The ``.21E`` file contains the channel and electrode
-  recording system information.
+- The ``.eeg`` file contains the actual raw EEG data.
+- The ``.pnt`` file contains metadata related to the recording such as the
+  measurement date.
+- The ``.log`` file contains annotations for the recording.
+- The ``.21e`` file contains channel and electrode information.
 
-Reading ``.11D``, ``.CMT``, ``.CN2``, and ``.EDF`` files is currently not
+Reading ``.11d``, ``.cmt``, ``.cn2``, and ``.edf`` files is currently not
 supported.
 
-Note that not all the subject metadata may be properly read in
-due to the fact that NK changes the specification
-sometimes from version to version. Please submit an issue, or
-pull request if you encounter a problem.
+Note that not all subject metadata may be properly read because NK changes the
+specification sometimes from version to version. Please let us know if you
+encounter a problem.
+
+
+XDF data (.xdf, .xdfz)
+======================
+
+MNE-Python does not support loading
+`XDF <https://github.com/sccn/xdf/wiki/Specifications>`_ files out of the box,
+because the inherent flexibility of the XDF format makes it difficult to
+provide a one-size-fits-all function. For example, XDF supports signals from
+various modalities recorded with different sampling rates. However, it is
+relatively straightforward to import only a specific stream (such as EEG
+signals) using the `pyxdf <https://github.com/xdf-modules/pyxdf>`_ package.
+See :ref:`ex-read-xdf` for a simple example.
+
+A more sophisticated version, which supports selection of specific streams as
+well as converting marker streams into annotations, is available in
+`MNELAB <https://github.com/cbrnr/mnelab>`_. If you want to use this
+functionality in a script, MNELAB records its history (View - History), which
+contains all commands required to load an XDF file after successfully loading
+that file with the graphical user interface.
 
 
 Setting EEG references
@@ -211,15 +232,16 @@
 the data are assumed to already be properly referenced. See
 :ref:`tut-set-eeg-ref` for more information.
 
+
 Reading electrode locations and head shapes for EEG recordings
 ==============================================================
 
-Some EEG formats (EGI, EDF/EDF+, BDF) neither contain electrode location
-information nor head shape digitization information. Therefore, this
-information has to be provided separately. For that purpose all raw instances
-have a :meth:`mne.io.Raw.set_montage` method to set electrode locations.
+Some EEG formats (e.g., EGI, EDF/EDF+, BDF) contain neither electrode locations
+nor head shape digitization information. Therefore, this information has to be
+provided separately. For that purpose, all raw instances have a
+:meth:`mne.io.Raw.set_montage` method to set electrode locations.
 
-When using the locations of the fiducial points the digitization data
-are converted to the MEG head coordinate system employed in the
-MNE software, see :ref:`coordinate_systems`.
+When using locations of fiducial points, the digitization data are converted to
+the MEG head coordinate system employed in the MNE software, see
+:ref:`coordinate_systems`.
 """  # noqa:E501


Test patch: