Repository: mne-tools/mne-python

Issue:
mne.io.read_raw_brainvision reads incorrect number of datapoints
I am writing a simple time series visualization script basing on MNE. I need to read data from EDF and BrainVision txt files, I have a sample that was exported to both. I discovered a strange behaviour that probably is a bug in `mne.io.read_raw_brainvision`, it is illustrated by

```
#!/usr/bin/env python
import mne

edf = mne.io.read_raw_edf("datasets/example.edf")
bv = mne.io.read_raw_brainvision("datasets/dattxt/example.vhdr")

print("Number of datapoints per channel ", edf.n_times, bv.n_times)
```

which prints in my case `('Number of datapoints per channel ', 831488, 73)`. It can be found (together with sample data) [in this repo](https://github.com/ociepkam/pfb). I would expect that `edf.n_times` and `bv.n_times` be equal. 

Additionally, when I try to read data from `bv` like so

```
data, time = bv[:, :1000]
```

I get

```
Traceback (most recent call last):
  File "./test.py", line 9, in <module>
    data, time = bv[:, :1000]
  File "/home/arccha/.conda/envs/pfb/lib/python2.7/site-packages/mne/io/base.py", line 841, in __getitem__
    verbose=self.verbose)
  File "/home/arccha/.conda/envs/pfb/lib/python2.7/site-packages/mne/io/base.py", line 523, in _read_segment
    cals, mult)
  File "/home/arccha/.conda/envs/pfb/lib/python2.7/site-packages/mne/io/brainvision/brainvision.py", line 116, in _read_segment_file
    _read_segments_c(self, data, idx, fi, start, stop, cals, mult)
  File "/home/arccha/.conda/envs/pfb/lib/python2.7/site-packages/mne/io/brainvision/brainvision.py", line 174, in _read_segments_c
    dtype = _fmt_dtype_dict[raw.orig_format]
TypeError: unhashable type: 'dict'
```

Am I doing something wrong or is it indeed a bug?

EDIT: I've tested this against MNE 0.15.2 and master branch.


Discussion:
I can't read those files at all - maybe something went wrong when you put them in the repo? Can you share these files e.g. via Dropbox?
@cbrnr as this files are rather big I've used [GIT LFS](https://git-lfs.github.com/) to put them in the repository. You need to have it installed in order to clone them.
I can't load the BrainVision file because the `.vhdr` file contains invalid data and marker file names. When I set these fields to `example.dat` and `example.vmkr`, I also get only 73 samples. However, I also cannot load this file properly with SigViewer (I don't see any signals, just markers). How did you create this BrainVision file? Can you open it with e.g. BrainVision Analyzer?
Yes, you’re correct, the `.vhdr` file contained wrong filenames. I’ve fixed it. I was able to load this file sucessfully with BVA 2.1.2.327. It was generated by exporting previously captured file using BVA export option.
I can reproduce the error. I could successfully load this file with EEGLAB and the FileIO plugin, but this took a long time (and produced a warning for each channel: `unknown resolution (i.e. recording units) for channel 73 in example.vhdr`), but the signals look OK. The BioSig toolbox for MATLAB also takes ages, and at some point I aborted the process so I'm not sure if it would have worked.

So yes, it seems like there is a bug in our BrainVision code. I'll look into it when I have more time.
I'm ran into the same issue. In case it helps, the `TypeError` seems to be caused by `raw.orig_format` being a dictionary:
```python
>>> data.orig_format
{'decimalsymbol': '.', 'skiplines': '0', 'skipcolumns': '1'}
```
I looked into the vhdr file, and I imagine this is coming from:
```
[ASCII Infos]
; Decimal symbol for floating point numbers: the header file always uses a dot (.),
; however the data file might use a different one
DecimalSymbol=.
; SkipLines, SkipColumns: leading lines and columns with additional information.
SkipLines=0
SkipColumns=1
```
That apparently happens if we export from BrainVision in text format. If we switch to binary format, then the vhdr file instead contains:
```
[Binary Infos]
BinaryFormat=IEEE_FLOAT_32
```
And `read_raw_brainvision` loads the file without the `TypeError`.
either we raise a meaningful error or we make it work...

I'm not really familiar with this format, but it seems like our code is already prepared for BINARY and ASCII flavors. However, for ASCII it looks like the number of rows in the .dat file is translated to the number of samples - which is wrong for this example file, since the channels are in the rows and the samples are in the columns: https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/brainvision.py#L101

I wonder if this example is another variant of the BrainVision ASCII format, or if this is how all ASCII files are structured. I quickly skimmed our test folder, and all files seem to come in the BINARY flavor. So maybe this was never tested and actually never worked? If not, any ideas how I can determine the row/column mapping in ASCII .dat files?

Furthermore, there is no dedicated reader function for reading ASCII data segments yet - `_read_segments_c` only works with BINARY .dat files as far as I can see.

So the quickest solution is to raise an error and tell people that the ASCII flavor is not supported, and that they should export their files to BINARY format.

Resolving patch:
diff --git a/mne/io/brainvision/brainvision.py b/mne/io/brainvision/brainvision.py
--- a/mne/io/brainvision/brainvision.py
+++ b/mne/io/brainvision/brainvision.py
@@ -94,11 +94,14 @@ def __init__(self, vhdr_fname, montage=None,
         _check_update_montage(info, montage)
         with open(data_filename, 'rb') as f:
             if isinstance(fmt, dict):  # ASCII, this will be slow :(
-                n_skip = 0
-                for ii in range(int(fmt['skiplines'])):
-                    n_skip += len(f.readline())
-                offsets = np.cumsum([n_skip] + [len(line) for line in f])
-                n_samples = len(offsets) - 1
+                if self._order == 'F':  # multiplexed, channels in columns
+                    n_skip = 0
+                    for ii in range(int(fmt['skiplines'])):
+                        n_skip += len(f.readline())
+                    offsets = np.cumsum([n_skip] + [len(line) for line in f])
+                    n_samples = len(offsets) - 1
+                elif self._order == 'C':  # vectorized, channels, in rows
+                    raise NotImplementedError()
             else:
                 f.seek(0, os.SEEK_END)
                 n_samples = f.tell()
@@ -443,6 +446,10 @@ def _get_vhdr_info(vhdr_fname, eog, misc, scale, montage):
             raise NotImplementedError('Datatype %s is not supported' % fmt)
         fmt = _fmt_dict[fmt]
     else:
+        if order == 'C':  # channels in rows
+            raise NotImplementedError('BrainVision files with ASCII data in '
+                                      'vectorized order (i.e. channels in rows'
+                                      ') are not supported yet.')
         fmt = dict((key, cfg.get('ASCII Infos', key))
                    for key in cfg.options('ASCII Infos'))
 


Test patch: