Repository: mne-tools/mne-python

Issue:
No exception to reraise in datasets/utils.py
PyCharm is complaining about [line 470 in datasets/utils.py](https://github.com/cbrnr/mne-python/blob/f85de59c9e052e6282c23aeb1f71fc2847ba84cd/mne/datasets/utils.py#L470): "No exception to reraise". What is the intention here? IMO either an explicit error should be raised, or the whole else branch should be deleted.


Discussion:
no opinion. this code what taken orignally from nilearn which took it from sklearn....
you can raise a RuntimeError. 

Is it even covered? I cannot find it here https://codecov.io/gh/massich/mne-python/tree/master/mne
`onerror` is called (presumably) inside an `except` clause and gets called when there is an error, see:

https://docs.python.org/3/library/shutil.html#shutil.rmtree

Confirmed it works as expected with (showing condensed tracebacks):
```
~$ mkdir temp
~$ chmod 500 temp
~$ sudo chown root:root temp
~$ python
>>> import shutil
>>> shutil.rmtree('/home/larsoner/temp/')  # raises
PermissionError: [Errno 13] Permission denied: '/home/larsoner/temp/'
>>> shutil.rmtree('/home/larsoner/temp/', onerror=lambda x, y, z: 0)  # ignored
>>> def reraise(x, y, z):
...     raise
>>> shutil.rmtree('/home/larsoner/temp/', onerror=reraise)  # (re)raises
PermissionError: [Errno 13] Permission denied: '/home/larsoner/temp/'
>>> def reraise_re(x, y, z):
...     raise RuntimeError('foo')
>>> shutil.rmtree('/home/larsoner/temp/', onerror=reraise_re)  # just for fun
RuntimeError: foo
```
So I think what's there is already the right thing to do. Feel free to add some `# noqa` comment or whatever is needed to make PyCharm know it's a false alarm. 
OK thanks for clarifying.
@larsoner I tried to debug this issue, but then stumbled upon something even more mysterious:

Maybe I'm missing something obvious, but I don't understand why I can delete a read-only directory using `rmdir` (at least on macOS, I haven't tried on Linux):

```
cd ~
mkdir temp
sudo chown root:wheel temp
sudo chmod 500 temp
```

Now `ls -ld temp` gives:

```
dr-x------  2 root  wheel  64 Jan 11 13:10 temp
```

But when I execute `rmdir temp` (as a normal user, not using `sudo`), the directory gets removed! Any ideas?
This is really bugging me, therefore I'm reopening. Can anyone tell me why we need the custom `onerror` handler for `shutil.rmtree`? This part of the code is [not covered by a test](https://codecov.io/gh/mne-tools/mne-python/src/master/mne/datasets/utils.py#L449), so it is difficult to verify that we really need it. Also, I can't seem to be able to come up with a test for it, because `shutil.rmtree` happily removes a folder even though the user doesn't have permissions (see my comment before). That's why I propose to remove `onerror` and just run `shutil.rmtree(folder_path)`. I'll create a PR with the change and see how our tests behave.
> Can anyone tell me why we need the custom onerror handler for shutil.rmtree?

The example I provided above shows an example of where it's potentially useful. It not being covered by a test is not sufficient to conclude that it's not useful under some circumstances. I don't see harm in keeping it, I do see potential harm in removing it.
... though I guess it doesn't show why it's useful so much as that it the warning given by PyCharm is a false alarm. For the usefulness we'd have to dig through the blame to see where the code came from to try to figure it out. Feel free to do so but I'm not sure it's worth spending too much time on.
Here you go:

https://github.com/mne-tools/mne-python/commit/9ebb67bc1f413f0843fe42861c1443a69c1e3743

It tries to fix access errors (though we could have known that just from the comment). Perhaps you can't replicate the error on your machine because `onerror` is doing its intended job of working around the perms error?
Which permission errors is this trying to fix? Why don't we raise the OSError to let the user know that there's something wrong with the permissions and let the user fix that?
I think it was because Windows can be overly picky about read-only files. At one point we had read-only files in `testing`, we might still. See e.g.:

https://stackoverflow.com/questions/2656322/shutil-rmtree-fails-on-windows-with-access-is-denied

I don't see why we are expending effort to remove code that was added to deal with a real problem, that otherwise seems like it has not been a problem in four years...?
Mainly because PyCharm marks this as a serious error. I'm not saying that it is necessarily correct, but there's no way to disable this message since it is not originating from its normal code introspection (normally you can right-click the warning and disable it). And if I get into the habit of ignoring particular errors and warnings it will get difficult for me to spot real errors.

Can we somehow change the code? I found that if we `raise exc_info[1]`, PyCharm doesn't generate an error. The traceback looks the same except that the actual line in `onerror` is included.
> The traceback looks the same except that the actual line in onerror is included.

This is less than ideal because it makes the error traceback less useful -- it will look like an `mne` error when it's a `shutil` one. I think we can get around it with something like the folowing (since we are Py3k only now, still needs to be tested):
```
except Exception:
    exc_info = sys.exc_info()
    raise exc_info[1].with_traceback(exc_info[2])
```

And FWIW I'm pretty sure it was the read-only stuff that necessitated this code -- I was on a Windows dev machine at the time IIRC and updating the `testing` dataset is what showed me this problem, which is something we never do in tests (because it would take a long time and extra bandwidth).
Yes, I'm sure there is a good reason for this code - sorry if I gave the impression of thinking otherwise.

Why do you think the message looks like it's an `mne` error? The output of `raise exc_info[1]` is (it seems to be the same for your suggestion):

```
/Users/clemens/Repositories/mne-python/.venv/bin/python /Users/clemens/Library/Preferences/PyCharm2018.3/scratches/scratch.py
Traceback (most recent call last):
  File "/Users/clemens/Library/Preferences/PyCharm2018.3/scratches/scratch.py", line 26, in <module>
    shutil.rmtree("/Users/clemens/temp", onerror=onerror)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 487, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/Users/clemens/Library/Preferences/PyCharm2018.3/scratches/scratch.py", line 23, in onerror
    raise exc_info[1]#.with_traceback(exc_info[2])
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 485, in rmtree
    fd = os.open(path, os.O_RDONLY)
PermissionError: [Errno 13] Permission denied: '/Users/clemens/temp'
```

And the output for `raise` only is:

```
Traceback (most recent call last):
  File "/Users/clemens/Library/Preferences/PyCharm2018.3/scratches/scratch.py", line 26, in <module>
    shutil.rmtree("/Users/clemens/temp", onerror=onerror)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 487, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 485, in rmtree
    fd = os.open(path, os.O_RDONLY)
PermissionError: [Errno 13] Permission denied: '/Users/clemens/temp'
```

In both cases, the `PermissionError` in `shutil.py` is the last message.

Ahh okay I thought it would end up at the bottom not the top.

Does the `raise exc_info[1].with_traceback(exc_info[2])` look shorter? The bare `raise` functionality is nicer so if we can keep it just by doing `raise exc_info[1].with_traceback(exc_info[2])` (or some variant) we should
Nope, it looks exactly the same like `raise exc_info[1]`. I agree that the output of `raise` is a bit nicer, but the new output isn't that bad because it only adds one line in the middle. Importantly, it makes PyCharm shut up 😉!

Resolving patch:
diff --git a/mne/datasets/utils.py b/mne/datasets/utils.py
--- a/mne/datasets/utils.py
+++ b/mne/datasets/utils.py
@@ -466,7 +466,7 @@ def onerror(func, path, exc_info):
             if do:
                 func(path)
             else:
-                raise
+                raise exc_info[1]
         shutil.rmtree(folder_path, onerror=onerror)
 
     logger.info('Decompressing the archive: %s' % archive_name)


Test patch: