GitHub issue #6134: `datasets` cannot be installed alongside `apache-beam`
State: closed | Created: 2023-08-10T06:54:32Z

### Describe the bug

If one installs `apache-beam` alongside `datasets` (which is required for the [wikipedia](https://huggingface.co/datasets/wikipedia#dataset-summary) dataset) in certain environments (such as a Google Colab notebook), they appear to install successfully, however, actually trying to do something such as importing the `load_dataset` method from `datasets` results in a crashing error.


I think the problem is that `apache-beam` version 2.49.0 requires `dill>=0.3.1.1,<0.3.2`, but the latest version of `multiprocess` (0.70.15) (on which `datasets` depends) requires  `dill>=0.3.7,`, so this is causing the dependency resolver to use an older version of `multiprocess` which leads to the `datasets` crashing since it doesn't actually appear to be compatible with older versions.

### Steps to reproduce the bug

See this [Google Colab notebook](https://colab.research.google.com/drive/1PTeGlshamFcJZix_GiS3vMXX_YzAhGv0?usp=sharing) to easily reproduce the bug.

In some environments, I have been able to reproduce the bug by running the following in Bash:

```bash
$ pip install datasets apache-beam
```

then the following in a Python shell:

```python
from datasets import load_dataset
```

Here is my stacktrace from running on Google Colab:

<details>
<summary>stacktrace</summary>

```
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     20 __version__ = "2.14.4"
     21 
---> 22 from .arrow_dataset import Dataset
     23 from .arrow_reader import ReadInstruction
     24 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     64 
     65 from . import config
---> 66 from .arrow_reader import ArrowReader
     67 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     68 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     28 import pyarrow.parquet as pq
     29 
---> 30 from .download.download_config import DownloadConfig
     31 from .naming import _split_re, filenames_for_dataset_split
     32 from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables

[/usr/local/lib/python3.10/dist-packages/datasets/download/__init__.py](https://localhost:8080/#) in <module>
      7 
      8 from .download_config import DownloadConfig
----> 9 from .download_manager import DownloadManager, DownloadMode
     10 from .streaming_download_manager import StreamingDownloadManager

[/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py](https://localhost:8080/#) in <module>
     33 from ..utils.info_utils import get_size_checksum_dict
     34 from ..utils.logging import get_logger, is_progress_bar_enabled, tqdm
---> 35 from ..utils.py_utils import NestedDataStructure, map_nested, size_str
     36 from .download_config import DownloadConfig
     37 

[/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <module>
     38 import dill
     39 import multiprocess
---> 40 import multiprocess.pool
     41 import numpy as np
     42 from packaging import version

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in <module>
    607 #
    608 
--> 609 class ThreadPool(Pool):
    610 
    611     from .dummy import Process

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in ThreadPool()
    609 class ThreadPool(Pool):
    610 
--> 611     from .dummy import Process
    612 
    613     def __init__(self, processes=None, initializer=None, initargs=()):

[/usr/local/lib/python3.10/dist-packages/multiprocess/dummy/__init__.py](https://localhost:8080/#) in <module>
     85 #
     86 
---> 87 class Condition(threading._Condition):
     88     # XXX
     89     if sys.version_info < (3, 0):

AttributeError: module 'threading' has no attribute '_Condition'
```

</details>

I've also found that attempting to install these `datasets` and `apache-beam` in certain environments (e.g. via pip inside a conda env) simply causes pip to hang indefinitely.

### Expected behavior

I would expect to be able to import methods from `datasets` without crashing. I have tested that this is possible as long as I do not attempt to install `apache-beam`.

### Environment info

Google Colab