devopsmarc/my-issues-dataset huggingface dataset:
pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic) 

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 The batch size fix doesn't work... not for #741 and not for this dataset I'm trying (DGS corpus)
Loading the DGS corpus takes 400GB of RAM, which is fine with me as my machine is large enough
pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic) 

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 Not yet, I've been pretty busy with the dataset sprint lately but this is something that's been asked several times already. So I'll definitely work on this as soon as I'm done with the sprint and with the RAM issue you reported.
pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic) 

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 Hi @lhoestq,
Any chance you have some updates on the supporting `ArrayXD` as a subtype or support of dynamic sized arrays?

e.g.:
`datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))`
`Array3D(shape=(None, 137, 2), dtype="float32")`
pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic) 

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 Hi ! We haven't worked in this lately and it's not in our very short-term roadmap since it requires a bit a work to make it work with arrow. Though this will definitely be added at one point.
pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic) 

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type> 
 Thanks for diving into this !

Indeed focusing on making the first dimensions dynamic make total sense (and users could still re-order their dimensions to match this constraint).
Your code looks great :) I think it can even be extended to support several dynamic dimensions if we want to.

Feel free to open a PR to include these changes, then we can update our test suite to make sure it works in all use cases.
In particular I think we might need a few tweaks to allow it to be converted to pandas (though I haven't tested yet):

```python
from datasets import Dataset, Features, Array3D

# this works
matrix = [[1, 0], [0, 1]]
features = Features({"a": Array3D(dtype="int32", shape=(1, 2, 2))})
d = Dataset.from_dict({"a": [[matrix], [matrix]]})
print(d.to_pandas())

# this should work as well
matrix = [[1, 0], [0, 1]]
features = Features({"a": Array3D(dtype="int32", shape=(None, 2, 2))})
d = Dataset.from_dict({"a": [[matrix], [matrix] * 2]})
print(d.to_pandas())
```

I'll be happy to help you on this :)
Downloading/caching only a part of a datasets' dataset. 
 Hi,
I want to use the validation data *only* (of natural question).
I don't want to have the whole dataset cached in my machine, just the dev set.
Is this possible? I can't find a way to do it in the docs.

Thank you,
Sapir 
 I think it would be a very helpful feature, because sometimes one only wants to evaluate models on the dev set, and the whole training data may be many times bigger.
This makes the task impossible with limited memory resources.
Add SQA 
 ## Adding a Dataset
- **Name:** SQA (Sequential Question Answering) by Microsoft. 
- **Description:** The SQA dataset was created to explore the task of answering sequences of inter-related questions on HTML tables. It has 6,066 sequences with 17,553 questions in total.
- **Paper:** https://www.microsoft.com/en-us/research/publication/search-based-neural-structured-learning-sequential-question-answering/
- **Data:** https://www.microsoft.com/en-us/download/details.aspx?id=54253
- **Motivation:** currently, the [Tapas](https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html) algorithm by Google AI is being added to the Transformers library (see https://github.com/huggingface/transformers/pull/8113). It would be great to use that model in combination with this dataset, on which it achieves SOTA results (average question accuracy of 0.71).

Note 1: this dataset actually consists of 2 types of files: 
1) TSV files, containing the questions, answer coordinates and answer texts (for training, dev and test)
2) a folder of csv files, which contain the actual tabular data

Note 2: if you download the dataset straight from the download link above, then you will see that the `answer_coordinates` and `answer_text` columns are string lists of string tuples and strings respectively, which is not ideal. It would be better to make them true Python lists of tuples and strings respectively (using `ast.literal_eval`), before uploading them to the HuggingFace hub.

Adding this would be great! Then we could possibly also add [WTQ (WikiTable Questions)](https://github.com/ppasupat/WikiTableQuestions) and [TabFact (Tabular Fact Checking)](https://github.com/wenhuchen/Table-Fact-Checking) on which TAPAS also achieves state-of-the-art results. Note that the TAPAS algorithm requires these datasets to first be converted into the SQA format.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).
 
 I’ll take this one to test the workflow for the sprint next week cc @yjernite @lhoestq
boolq does not load  
 Hi
I am getting these errors trying to load boolq thanks 

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    data = AutoTask().get("boolq").get_dataset("train", n_obs=10)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 42, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 38, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 150, in download_custom
    get_from_cache(url, cache_dir=cache_dir, local_files_only=True, use_etag=False)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 472, in get_from_cache
    f"Cannot find the requested files in the cached path at {cache_path} and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path at /idiap/home/rkarimi/.cache/huggingface/datasets/eaee069e38f6ceaa84de02ad088c34e63ec97671f2cd1910ddb16b10dc60808c and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.
 
 Hi ! It runs on my side without issues. I tried
```python
from datasets import load_dataset
load_dataset("boolq")
```

What version of datasets and tensorflow are your runnning ?
Also if you manage to get a minimal reproducible script (on google colab for example) that would be useful.
boolq does not load  
 Hi
I am getting these errors trying to load boolq thanks 

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    data = AutoTask().get("boolq").get_dataset("train", n_obs=10)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 42, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 38, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 150, in download_custom
    get_from_cache(url, cache_dir=cache_dir, local_files_only=True, use_etag=False)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 472, in get_from_cache
    f"Cannot find the requested files in the cached path at {cache_path} and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path at /idiap/home/rkarimi/.cache/huggingface/datasets/eaee069e38f6ceaa84de02ad088c34e63ec97671f2cd1910ddb16b10dc60808c and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.
 
 hey
i do the exact same commands. for me it fails i guess might be issues with
caching maybe?
thanks
best
rabeeh

On Tue, Nov 24, 2020, 10:24 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hi ! It runs on my side without issues. I tried
>
> from datasets import load_datasetload_dataset("boolq")
>
> What version of datasets and tensorflow are your runnning ?
> Also if you manage to get a minimal reproducible script (on google colab
> for example) that would be useful.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/879#issuecomment-732769114>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCGGDR2FUMRKZTIY5CTSRN3VXANCNFSM4T7R3U6A>
> .
>
boolq does not load  
 Hi
I am getting these errors trying to load boolq thanks 

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    data = AutoTask().get("boolq").get_dataset("train", n_obs=10)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 42, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 38, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 150, in download_custom
    get_from_cache(url, cache_dir=cache_dir, local_files_only=True, use_etag=False)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 472, in get_from_cache
    f"Cannot find the requested files in the cached path at {cache_path} and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path at /idiap/home/rkarimi/.cache/huggingface/datasets/eaee069e38f6ceaa84de02ad088c34e63ec97671f2cd1910ddb16b10dc60808c and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.
 
 Could you check if it works on the master branch ?
You can use `load_dataset("boolq", script_version="master")` to do so.
We did some changes recently in boolq to remove the TF dependency and we changed the way the data files are downloaded in https://github.com/huggingface/datasets/pull/881
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 > neat feature

I dint get these clearly, can you please elaborate like how to work on these
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 It could maybe work almost out of the box just by using `cached_path` in the text/csv/json scripts, no?
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 Thanks thomwolf and julien-c

I'm still confusion on what you guys said, 

I have solved the problem as follows:

1. read the csv file using pandas from s3 
2. Convert to dictionary key as column name and values as list column data
3. convert it to Dataset using 
`from datasets import Dataset`
`train_dataset = Dataset.from_dict(train_dict)`
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 We were brainstorming around your use-case.

Let's keep the issue open for now, I think this is an interesting question to think about.
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 > We were brainstorming around your use-case.
> 
> Let's keep the issue open for now, I think this is an interesting question to think about.

Sure thomwolf, Thanks for your concern
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 I agree it would be cool to have that feature. Also that's good to know that pandas supports this.
For the moment I'd suggest to first download the files locally as thom suggested and then load the dataset by providing paths to the local files
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 Any updates on this issue?
I face a similar issue. I have many parquet files in S3 and I would like to train on them. 
To be honest I even face issues with only getting the last layer embedding out of them.
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 Hi dorlavie, 
You can find one solution that i have mentioned above, that can help you. 
And there is one more solution also which is downloading files locally
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 > Hi dorlavie,
> You can find one solution that i have mentioned above, that can help you.
> And there is one more solution also which is downloading files locally

mahesh1amour, thanks for the fast reply

Unfortunately, in my case I can not read with pandas. The dataset is too big (50GB). 
In addition, due to security concerns I am not allowed to save the data locally
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 @dorlavie could use `boto3` to download the data to your local machine and then load it with `dataset`

boto3 example [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html)
```python
import boto3

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
```

datasets example [documentation](https://huggingface.co/docs/datasets/loading_datasets.html)

```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])
```
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 Thanks @philschmid for the suggestion.
As I mentioned in the previous comment, due to security issues I can not save the data locally.
I need to read it from S3 and process it directly.

I guess that many other people try to train / fit those models on huge datasets (e.g entire Wiki), what is the best practice in those cases?
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 If I understand correctly you're not allowed to write data on disk that you downloaded from S3 for example ?
Or is it the use of the `boto3` library that is not allowed in your case ?
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 @lhoestq yes you are correct.
I am not allowed to save the "raw text" locally - The "raw text" must be saved only on S3.
I am allowed to save the output of any model locally. 
It doesn't matter how I do it boto3/pandas/pyarrow, it is forbidden
Loading Data From S3 Path in Sagemaker 
 In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load 
 @dorlavie are you using sagemaker for training too? Then you could use S3 URI, for example `s3://my-bucket/my-training-data` and pass it within the `.fit()` function when you start the sagemaker training job. Sagemaker would then download the data from s3 into the training runtime and you could load it from disk

**sagemaker start training job**
```python
pytorch_estimator.fit({'train':'s3://my-bucket/my-training-data','eval':'s3://my-bucket/my-evaluation-data'})
```

**in the train.py script**
```python
from datasets import load_from_disk

train_dataset = load_from_disk(os.environ['SM_CHANNEL_TRAIN'])
```

I have created an example of how to use transformers and datasets with sagemaker. 
https://github.com/philschmid/huggingface-sagemaker-example/tree/main/03_huggingface_sagemaker_trainer_with_data_from_s3

The example contains a jupyter notebook `sagemaker-example.ipynb` and an `src/` folder. The sagemaker-example is a jupyter notebook that is used to create the training job on AWS Sagemaker. The `src/` folder contains the `train.py`, our training script, and `requirements.txt` for additional dependencies.
DataLoader(datasets) become more and more slowly within iterations 
 Hello, when I for loop my dataloader, the loading speed is becoming more and more slowly!
```
dataset = load_from_disk(dataset_path)  # around 21,000,000 lines

lineloader = tqdm(DataLoader(dataset, batch_size=1))
for idx, line in enumerate(lineloader):
     # do some thing for each line
```
In the begining, the loading speed is around 2000it/s, but after 1 minutes later, the speed is much slower, just around 800it/s.

And when I set `num_workers=4` in DataLoader, the loading speed is much lower, just 130it/s.

Could you please help me with this problem?
Thanks a lot! 
 Hi ! Thanks for reporting.
Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
It would be nice to know whether it comes from the dataloader or not
DataLoader(datasets) become more and more slowly within iterations 
 Hello, when I for loop my dataloader, the loading speed is becoming more and more slowly!
```
dataset = load_from_disk(dataset_path)  # around 21,000,000 lines

lineloader = tqdm(DataLoader(dataset, batch_size=1))
for idx, line in enumerate(lineloader):
     # do some thing for each line
```
In the begining, the loading speed is around 2000it/s, but after 1 minutes later, the speed is much slower, just around 800it/s.

And when I set `num_workers=4` in DataLoader, the loading speed is much lower, just 130it/s.

Could you please help me with this problem?
Thanks a lot! 
 > Hi ! Thanks for reporting.
> Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> It would be nice to know whether it comes from the dataloader or not

I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`,  around 20000it/s.
imdb dataset cannot be loaded  
 Hi
I am trying to load the imdb train dataset

`dataset = datasets.load_dataset("imdb", split="train")`

getting following errors, thanks for your help 
```
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=32660064, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='test', num_bytes=26476338, num_examples=20316, dataset_name='imdb')}, {'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
>>> dataset = datasets.load_dataset("imdb", split="train")

```
 
 It looks like there was an issue while building the imdb dataset.
Could you provide more information about your OS and the version of python and `datasets` ?

Also could you try again with 
```python
dataset = datasets.load_dataset("imdb", split="train", download_mode="force_redownload")
```
to make sure it's not a corrupted file issue ?