Repository: mne-tools/mne-bids

Issue:
[BUG] _channels_tsv writes default unit for channel type
`mne_bids._channels_tsv` writes a unit that is assumed based on the channel type:

https://github.com/mne-tools/mne-bids/blob/e03f013cfc897d116ce4d67d7bce6204d94a40a7/mne_bids/mne_bids.py#L89

This is problematic because different manufacturers save the raw data in different units. Sometimes there are even differences within single manufacturer data formats. Usually, the units are declared in the dataset - when reading the data with MNE-Python, the data gets automatically scaled to e.g., VOLTS so that all MNE-Python objects consistently work in VOLTS.

This is a problem, because we are then copying the non-mne-modified raw data to the bids directory ... and the units will not match.


Discussion:
hmm ... so what are you proposing? Can you give an example where it fails?
For example the `test_vhdr` in #78 writes this `channels.tsv`:

```
Writing '/tmp/tmp_mne_tempdir_q1hu9plh/sub-01/ses-01/eeg/sub-01_ses-01_task-testing_run-01_channels.tsv'...

  name type units   ...   low_cutoff  high_cutoff  status
0  FP1  EEG     V   ...          0.0        250.0    good
1  FP2  EEG     V   ...          0.0        250.0    good
2   F3  EEG     V   ...          0.0        250.0    good
3   F4  EEG     V   ...          0.0        250.0    good
4   C3  EEG     V   ...          0.0        250.0    good
```

The units are V, however the data is actually in microVolts, see here:

https://github.com/mne-tools/mne-python/blob/5a84ba9f247362da66b209634916485d1c691c13/mne/io/brainvision/tests/data/test.vhdr#L18-L25

Currently just collecting all issues that I encounter while making #78 work. No concrete proposals ... 
okay sounds good! I'd recommend fixing it in a separate PR
it might be possible that this info is in the `_raw_extras` for the system specific io. I think this is mostly an eeg concern? maybe we can list the affected systems, and we can dive in for a solution
> it might be possible that this info is in the _raw_extras for the system specific io

sounds interesting - how can that be accessed?

it's a problem for any system that does not natively save data in volts (so probably all EEG and iEEG data formats, don't know about MEG)
@teonbrooks can you answer to my question? If possible, I'd like to fix this bug before we release
I fear you may have to dig into `mne-python` code for this ...
I don't have time to fix it before 0.1.0, so added it to 0.2.0 ... https://github.com/mne-tools/mne-python/issues/5607 needs to be fixed first (see above)
@sappelhoff this would mean we should use the development version of `mne`, right? I guess it's unavoidable at some point since `mne_bids` is so tightly coupled to `mne` ...
I think that https://github.com/mne-tools/mne-python/pull/5633 is ready to be merged, and then MNE will be released next week or so - so we can use MNE 0.17, right?
I think @teonbrooks wanted to advertise MNE-BIDS at SFN, so I think ideally we'd like to have a release before that. And I don't think MNE is getting released before that ...
okay, fine by me to use master. I was always more inclined to do that - although the arguments against it are still valid ...

the plus side is, that it's making development easier for us. So, let's wait for the mentioned MNE-Python PR to be merged and then I'll open a PR to fix this present issue.
FYI we plan to release mne-python early november

Now that https://github.com/mne-tools/mne-python/pull/5633 was merged, I can start to work on this here :-)
Perfect! May I suggest branching off from #106 so that you don't have a rebasing mess?

Resolving patch:
diff --git a/mne_bids/mne_bids.py b/mne_bids/mne_bids.py
--- a/mne_bids/mne_bids.py
+++ b/mne_bids/mne_bids.py
@@ -131,8 +131,12 @@ def _channels_tsv(raw, fname, overwrite=False, verbose=True):
         ch_type.append(map_chs[_channel_type])
         description.append(map_desc[_channel_type])
     low_cutoff, high_cutoff = (raw.info['highpass'], raw.info['lowpass'])
-    units = [_unit2human.get(ch_i['unit'], 'n/a') for ch_i in raw.info['chs']]
-    units = [u if u not in ['NA'] else 'n/a' for u in units]
+    if raw._orig_units:
+        units = [raw._orig_units.get(ch, 'n/a') for ch in raw.ch_names]
+    else:
+        units = [_unit2human.get(ch_i['unit'], 'n/a')
+                 for ch_i in raw.info['chs']]
+        units = [u if u not in ['NA'] else 'n/a' for u in units]
     n_channels = raw.info['nchan']
     sfreq = raw.info['sfreq']
 
diff --git a/mne_bids/utils.py b/mne_bids/utils.py
--- a/mne_bids/utils.py
+++ b/mne_bids/utils.py
@@ -362,11 +362,11 @@ def _write_json(dictionary, fname, overwrite=False, verbose=False):
 
 
 def _write_tsv(fname, df, overwrite=False, verbose=False):
-    """Write dataframe to a .tsv file"""
+    """Write dataframe to a .tsv file."""
     if op.exists(fname) and not overwrite:
         raise OSError(errno.EEXIST, '"%s" already exists. Please set '
                       'overwrite to True.' % fname)
-    df.to_csv(fname, sep='\t', index=False, na_rep='n/a')
+    df.to_csv(fname, sep='\t', index=False, na_rep='n/a', encoding='utf-8')
 
     if verbose:
         print(os.linesep + "Writing '%s'..." % fname + os.linesep)


Test patch:
diff --git a/mne_bids/tests/test_mne_bids.py b/mne_bids/tests/test_mne_bids.py
--- a/mne_bids/tests/test_mne_bids.py
+++ b/mne_bids/tests/test_mne_bids.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 """Test the MNE BIDS converter.
 
 For each supported file format, implement a test.
@@ -194,6 +195,14 @@ def test_vhdr():
     cmd = ['bids-validator', '--bep006', output_path]
     run_subprocess(cmd, shell=shell)
 
+    # Test that correct channel units are written
+    channels_tsv_name = op.join(output_path, 'sub-' + subject_id,
+                                'ses-' + session_id, 'eeg',
+                                bids_basename + '_channels.tsv')
+    df = pd.read_csv(channels_tsv_name, sep='\t', keep_default_na=False)
+    assert df.loc[df['name'] == 'FP1', 'units'].all() == 'µV'
+    assert df.loc[df['name'] == 'CP5', 'units'].all() == 'n/a'
+
     # create another bids folder with the overwrite command and check
     # no files are in the folder
     data_path = make_bids_folders(subject=subject_id, session=session_id,