Incorrect filepath for dill module
Opened by avivbrokman (closed)

### Describe the bug

I installed the `datasets` package and when I try to `import` it, I get the following error:

```
Traceback (most recent call last):

  File "/var/folders/jt/zw5g74ln6tqfdzsl8tx378j00000gn/T/ipykernel_3805/3458380017.py", line 1, in <module>
    import datasets

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 66, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import Features, Image, Value

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/__init__.py", line 17, in <module>
    from .audio import Audio

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/audio.py", line 12, in <module>
    from ..download.streaming_download_manager import xopen

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/download_manager.py", line 36, in <module>
    from ..utils.py_utils import NestedDataStructure, map_nested, size_str

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 602, in <module>
    class Pickler(dill.Pickler):

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 605, in Pickler
    dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())

AttributeError: module 'dill' has no attribute '_dill'

```

Looking at the github source code for dill, it appears that `datasets` has a bug or is not compatible with the latest `dill`.  Specifically, rather than `dill._dill.XXXX` it should be `dill.dill._dill.XXXX`.  But given the popularity of `datasets` I feel confused about me being the first person to have this issue, so it makes me wonder if I'm misdiagnosing the issue.

### Steps to reproduce the bug

Install `dill` and `datasets` packages and then `import datasets`

### Expected behavior

I expect `datasets` to import.

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 11.0.0
- Pandas version: 1.4.4

Comments:
Hi! The correct path is still `dill._dill.XXXX` in the latest release. What do you get when you run  `python -c "import dill; print(dill.__version__)"` in your environment?

`0.3.6` I feel like that's bad news, because it's probably not the issue.

My mistake, about the wrong path guess.  I think I didn't notice that the first `dill` in the path isn't supposed to be included in the path specification in python.
<img width="146" alt="Screen Shot 2023-01-31 at 12 58 32 PM" src="https://user-images.githubusercontent.com/35349273/215844209-74af6a8f-9bff-4c75-9495-44c658c8e9f7.png">


Hi, @avivbrokman, this issue you report appeared only with old versions of dill. See:
- #288

Are you sure you are in the right Python environment?
- Please note that Jupyter (where I guess you get the error) may have multiple execution backends (IPython kernels) that might be different from the Python environment your are using to get the dill version
  - Have you run `import dill; print(dill.__version__)` in the same Jupyter/IPython that you were using when you got the error while executing `import datasets`?

I'm using spyder, and I am still getting `0.3.6` for `dill`, so unfortunately #288 isn't applicable, I think.  However, I found something odd that I believe is a clue: 

```
import inspect
import dill

inspect.getfile(dill)
>>> '/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/dill/__init__.py'
```

I checked out the directory, and there is no `dill` subdirectory within '/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/dill`, as there should be.  Rather, `_dill.py` is in  '/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/dill` itself. 

 If I run `pip install dill` or `pip install --upgrade dill`, I get the message `Requirement already satisfied: dill in ./opt/anaconda3/lib/python3.9/site-packages (0.3.6)`.  If I run `conda upgrade dill`, I get the message `Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.` a couple of times, followed by

```
Solving environment: failed
Solving environment: / 
Found conflicts! Looking for incompatible packages.
```

And then terminal proceeds to list conflicts between different packages I have.

This is all very strange to me because I recently uninstalled and reinstalled `anaconda`.


As I said above, I guess this is not a problem with `datasets`. I think you have different Python environments: one with the new dill version (the one you get while using pip) and other with the old dill version (the one where you get the AttributeError).

You should update `dill` in the Python environment you are using within spyder.

Please note that the `_dill` module is present in the `dill` package since their 2.8.0 version.