Support streaming mlsum dataset
Opened by albertvillanova (closed)

Support streaming mlsum dataset.

This PR:
- pins `fsspec` min version with fixed BlockSizeError: `fsspec[http]>=2021.11.1`
  - https://github.com/fsspec/filesystem_spec/pull/830
- unpins `s3fs==2021.08.1` to align it with `fsspec` requirement: `s3fs>=2021.11.1`
  > s3fs 2021.8.1 requires fsspec==2021.08.1
  - see discussion: https://github.com/huggingface/datasets/pull/2858/files#r700027326
- updates the following requirements to be compatible with the previous ones and one with each other:
  - `aiobotocore==1.4.2` to `aiobotocore>=2.0.1` (required by s3fs>=2021.11.1)
  - `boto3==1.17.106` to `boto3>=1.19.8` (to be compatible with aiobotocore>=2.0.1)
  - `botocore==1.20.106` to `botocore>=1.22.8` (to be compatible with aiobotocore and boto3)

Fix #4572.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

After unpinning `s3fs` and pinning `fsspec[http]>=2021.11.1`, the CI installs
- `fsspec-2022.1.0`
- `s3fs-0.5.1`

and raises the following error:
```
 ImportError while loading conftest '/home/runner/work/datasets/datasets/tests/conftest.py'.
tests/conftest.py:13: in <module>
    import datasets
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/__init__.py:37: in <module>
    from .arrow_dataset import Dataset
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/arrow_dataset.py:62: in <module>
    from .arrow_reader import ArrowReader
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/arrow_reader.py:29: in <module>
    from .download.download_config import DownloadConfig
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/download/__init__.py:10: in <module>
    from .streaming_download_manager import StreamingDownloadManager
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/download/streaming_download_manager.py:20: in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/filesystems/__init__.py:13: in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/filesystems/s3filesystem.py:1: in <module>
    import s3fs
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/s3fs/__init__.py:1: in <module>
    from .core import S3FileSystem, S3File
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/s3fs/core.py:12: in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
E   ImportError: cannot import name 'maybe_sync'
```

The installed `s3fs` version is too old. What about pinning a min version?

Maybe you can try setting the same minimum version as fsspec ? `s3fs>=2021.11.1`

Yes, I have checked that they both require to have the same version. 

The issue then was coming from aiobotocore, boto3, botocore. I have changed them from strict to min version requirements.
> s3fs 2021.11.1 depends on aiobotocore~=2.0.1

I have updated all min versions so that they are compatible one with each other. I'm pushing again...

Thanks !

Nice!