Repository: mne-tools/mne-bids

Issue:
problem with overwrite=True for raw ctf data
Hi mne-bids folks!
I am in the process of converting my raw CTF data to BIDS format before analyses.
My raw dir is as such:
```sh
|--------------------------- mASD03_MMNASD-pduret_20181113_03.ds
|------------------------------ mASD03_MMNASD-pduret_20181113_03.acq
|------------------------------ mASD03_MMNASD-pduret_20181113_03.eeg
|------------------------------ bad.segments.bak
|------------------------------ bad.segments
|------------------------------ ChannelGroupSet.cfg
|------------------------------ mASD03_MMNASD-pduret_20181113_03.infods
|------------------------------ MarkerFile.mrk
|------------------------------ mASD03_MMNASD-pduret_20181113_03.hist
|------------------------------ BadChannels
|------------------------------ mASD03_MMNASD-pduret_20181113_03.hc
|------------------------------ mASD03_MMNASD-pduret_20181113_03.res4
|------------------------------ mASD03_MMNASD-pduret_20181113_03.newds
|------------------------------ ClassFile.cls
|------------------------------ MarkerFile.mrk.bak
|------------------------------ mASD03_MMNASD-pduret_20181113_03.infods.bak
|------------------------------ mASD03_MMNASD-pduret_20181113_03.meg4
|------------------------------ hz.ds
|--------------------------------- hz.res4
|--------------------------------- hz.meg4
|--------------------------------- hz.hist
|--------------------------------- hz.hc
|--------------------------------- hz.newds
|--------------------------------- BadChannels
|--------------------------------- hz.acq
|--------------------------------- hz.infods
```

And my output dir is as such:

```sh
--------------------- mMNE
|------------------------ mASD03_MMNASD-pduret_20181113_03.eeg
|------------------------ participants.tsv
|------------------------ participants.json
|------------------------ dataset_description.json
|------------------------ sub-mASD03
|--------------------------- sub-mASD03_scans.tsv
|--------------------------- meg
|------------------------------ sub-mASD03_task-ContextOdd_run-2_channels.tsv
|------------------------------ sub-mASD03_task-ContextOdd_run-2_events.tsv
|------------------------------ sub-mASD03_coordsystem.json
|------------------------------ sub-mASD03_task-ContextOdd_run-2_meg.json
|------------------------------ sub-mASD03_task-ContextOdd_run-2_meg.ds
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.acq
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.eeg
|--------------------------------- bad.segments.bak
|--------------------------------- bad.segments
|--------------------------------- ChannelGroupSet.cfg
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.infods
|--------------------------------- MarkerFile.mrk
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.hist
|--------------------------------- BadChannels
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.hc
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.res4
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.newds
|--------------------------------- ClassFile.cls
|--------------------------------- MarkerFile.mrk.bak
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.infods.bak
|--------------------------------- mASD03_MMNASD-pduret_20181113_03.meg4
|--------------------------------- hz.ds
|------------------------------------ hz.res4
|------------------------------------ hz.meg4
|------------------------------------ hz.hist
|------------------------------------ hz.hc
|------------------------------------ hz.newds
|------------------------------------ BadChannels
|------------------------------------ hz.acq
|------------------------------------ hz.infods
```

As you can see, the data is copied but what's inside the .ds folder does not get renamed correctly. This is for the first iteration of the loop over subj and runs, the script would break and I get the following error:

```py
Copying data files to /sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds
Traceback (most recent call last):

  File "<ipython-input-40-672e36af4084>", line 16, in <module>
    event_id=event_id, overwrite=True)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/site-packages/mne_bids/mne_bids.py", line 736, in write_raw_bids
    copyfile_ctf(raw_fname, bids_fname)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/site-packages/mne_bids/utils.py", line 531, in copyfile_ctf
    sh.copytree(src, dest)

  File "/pbs/home/p/pduret/anaconda3/lib/python3.7/shutil.py", line 359, in copytree
    raise Error(errors)

Error: [('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.acq', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/mASD03_MMNASD-pduret_20181113_03.acq', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.acq'"), ('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.eeg', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/mASD03_MMNASD-pduret_20181113_03.eeg', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/mASD03_MMNASD-pduret_20181113_03.eeg'"), ('/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/bad.segments.bak', '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/mMNE/sub-mASD03/meg/sub-mASD03_task-ContextOdd_run-2_meg.ds/bad.segments.bak', "[Errno 22] Invalid argument: '/sps/crnl/users/pduret/ContextOdd/ContextOdd_Data/Raw_Data/MEG/mASD03_MMNASD-pduret_20181113_03.ds/bad.segments.bak'"), ` .....
```

And so on for all the files in the .ds .
I don't get why this is happening, is there something wrong in my data architecture?

Thank you!


Discussion:
Also, here is the code I am using, I can be wrong here:

```py
for subject, long_name, in subject_ids.items() :
    for run in runs[subject]:
        
        raw_fname = op.join(data_path, f'{long_name}_0{run}.ds')
        raw = mne.io.read_raw_ctf(raw_fname)
        
        if run in [1,6]:
            bids_basename = (f'sub-{subject}_task-ContextOdd_run-RS{run}')
            write_raw_bids(raw, bids_basename, output_path, events_data = None, 
                       event_id=event_id, overwrite=True) 
            
        else:
            events = find_events(raw, shortest_event=2, stim_channel='UPPT001', output='onset')
            bids_basename = (f'sub-{subject}_task-ContextOdd_run-{int(run) - 1}')
            write_raw_bids(raw, bids_basename, output_path, events_data = events, 
                       event_id=event_id, overwrite=True) 
```
Thanks for reporting @paulineduret !

I edited your description so that it's formatted better. I'll take a look later today.

maybe @romquentin would like to pitch in since he wrote the `copyfile_ctf` function in #162 
Hi, 
Yes apparently the function crash before renaming the file (during sh.copytree(src, dest)). 
I am not sure what's going wrong.
@paulineduret could you share a subset of your dataset for me to be able to reproduce the error?
I had a very similar issue when working on [BIDSHandler](https://github.com/Macquarie-MEG-Research/BIDSHandler). I found `shutil` gives errors like that when it tries to move/copy files to a location that would have a file name longer than the maximum value allowed on an operating system.
Not sure if this is the same issue but it may be something to keep in mind...
Hi, thanks for answering folks!

Here are the data from two subjects, to try and run the loop.
https://drive.google.com/drive/folders/1ainSCfThraZu0ZkLwc4iarcy47ESlGh-?usp=sharing
Hi @paulineduret I tried the following:

```py
import mne
from mne_bids import write_raw_bids

subjects = ['mASD03', 'mASD07']
dates = ['20181113', '20181126']
n_runs = [6, 4]

for subject, date, n_run in zip(subjects, dates, n_runs):
    for run in range(1, n_run + 1):

        raw_fname = ('MEGdataTest/%s_MMNASD-pduret_%s_%02d.ds'
                     % (subject, date, run))
        raw = mne.io.read_raw_ctf(raw_fname)
        bids_basename = 'sub-%s_task-ContextOdd_run-%02d' % (subject, run)

        write_raw_bids(raw, bids_basename, 'bids_root', overwrite=True)
```

and this works fine. However, I confirm that there is a problem with overwriting. If you remove the BIDS folder altogether and try again, does it work?
Hi @jasmainak ,
yes I always had an error when overwriting so I deleted the previous BIDS folder each time I tried.

Also, I tried your simpler code, just adjusting the paths, and I still get the same error. I tried with shorter folder names too, same error...

Hello again,
so apparently, my error is rooted on shutil.copy2, precisely when the function calls shutil.copystat, which in turn calls os.getxattr.
This might be a permission problem after all, as all this works fine locally on my computer, but not on the server using linux were I store my data.
Is there something I can do?
@paulineduret for permissions or space/quota issues, you will have to ask the IT person at your place.

However, we did uncover a problem with `overwrite` which should definitely be fixed.
Hi, 
I finally found interesting documentation on my error here:
https://bugs.python.org/issue24564

It is not a permission issue but an issue linked to python and NFS.
Does anyone know how if this getxattr is important or if there is a way to skip this step?

Thx
Hi again!
I did like suggested in the issue and as they did here:
https://github.com/python/cpython/pull/8601

and added errno.EINVAL to the ignored errors in my shutyl.py script (not sure this is legit though!).

But now, it works :)
@paulineduret you have become quite a hacker!

okay, so it seems the clean solution would be to use Python 3.7, right? Apparently, they cannot backport to Python 3.6 anymore. I think this is useful information and we should document this somewhere. 
oh wait, it's not merged yet :(

What if we change the copyfunction in shutil.copytree: https://docs.python.org/3/library/shutil.html#shutil.copytree? Instead of copy2 if we use copy, then it won't use copystat. Could you try this and tell us if it helps?
Hi Mainak,
I already tried that but as it was stated somewhere else, when you have a directory and not a single file, the _getxattr is used anyway, with copy too.
That's why the only thing was to ignore the error.
@paulineduret you are the best person to fix this issue because you hit it :)

okay, one more try. How about reusing the code here: https://github.com/jupyterlab/jupyterlab/pull/5150/files? Feel free to ping me on slack if you need help in trying this.
@jasmainak Hi!
The code you gave works fine when included to the bids module utils.py.
This way I don't have to modify shutils.
Thx!
okay great, I'll make a PR. Just to confirm, it resulted in a valid BIDS directory?

Resolving patch:
diff --git a/mne_bids/fixes.py b/mne_bids/fixes.py
new file mode 100644
--- /dev/null
+++ b/mne_bids/fixes.py
@@ -0,0 +1,17 @@
+"""Temporary fixes/workarounds for bugs in upstream code."""
+
+# Authors: Mainak Jas <mainakjas@gmail.com>
+
+import shutil
+import os.path as op
+
+
+def _copytree(src, dst, **kwargs):
+    """See: https://github.com/jupyterlab/jupyterlab/pull/5150."""
+    try:
+        shutil.copytree(src, dst, **kwargs)
+    except shutil.Error as error:
+        # `copytree` throws an error if copying to + from NFS even though
+        # the copy is successful (see https://bugs.python.org/issue24564)
+        if '[Errno 22]' not in str(error) or not op.exists(dst):
+            raise
diff --git a/mne_bids/mne_bids.py b/mne_bids/mne_bids.py
--- a/mne_bids/mne_bids.py
+++ b/mne_bids/mne_bids.py
@@ -34,6 +34,7 @@
                     copyfile_brainvision, copyfile_eeglab, copyfile_ctf,
                     _infer_eeg_placement_scheme, _parse_bids_filename,
                     _handle_kind)
+
 from .io import _parse_ext, ALLOWED_EXTENSIONS, reader
 from mne_bids.tsv_handler import _from_tsv, _combine, _drop, _contains_row
 
diff --git a/mne_bids/utils.py b/mne_bids/utils.py
--- a/mne_bids/utils.py
+++ b/mne_bids/utils.py
@@ -23,6 +23,7 @@
 from mne.io.pick import pick_types
 
 from .config import BIDS_VERSION
+from .fixes import _copytree
 from .io import _parse_ext
 from .tsv_handler import _to_tsv, _tsv_to_str
 
@@ -528,7 +529,7 @@ def copyfile_ctf(src, dest):
     dest : str
         path to the destination of the new bids folder.
     """
-    sh.copytree(src, dest)
+    _copytree(src, dest)
     # list of file types to rename
     file_types = ('.acq', '.eeg', '.hc', '.hist', '.infods', '.bak',
                   '.meg4', '.newds', '.res4')


Test patch: