DatasetGenerationError when load glue benchmark datasets from `load_dataset` 
Opened by yusx-swapp (closed)

### Describe the bug


Package version: datasets-2.14.4

When I run the codes:

```
from datasets import load_dataset

dataset = load_dataset("glue", "ax")
```


I got the following errors:


---------------------------------------------------------------------------
SchemaInferenceError                      Traceback (most recent call last)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1949, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1948 num_shards = shard_id + 1
-> 1949 num_examples, num_bytes = writer.finalize()
   1950 writer.close()

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/arrow_writer.py:598, in ArrowWriter.finalize(self, close_stream)
    597         self.stream.close()
--> 598     raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
    599 logger.debug(
    600     f"Done writing {self._num_examples} {self.unit} in {self._num_bytes} bytes {self._path if self._path else ''}."
    601 )

SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[5], line 3
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("glue", "ax")

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/load.py:2136, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2133 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   2135 # Download and prepare data
-> 2136 builder_instance.download_and_prepare(
   2137     download_config=download_config,
   2138     download_mode=download_mode,
   2139     verification_mode=verification_mode,
   2140     try_from_hf_gcs=try_from_hf_gcs,
   2141     num_proc=num_proc,
   2142     storage_options=storage_options,
   2143 )
   2145 # Build dataset for splits
   2146 keep_in_memory = (
   2147     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   2148 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:954, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    952     if num_proc is not None:
    953         prepare_split_kwargs["num_proc"] = num_proc
--> 954     self._download_and_prepare(
    955         dl_manager=dl_manager,
    956         verification_mode=verification_mode,
    957         **prepare_split_kwargs,
    958         **download_and_prepare_kwargs,
    959     )
    960 # Sync info
    961 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1049, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1045 split_dict.add(split_generator.split_info)
   1047 try:
   1048     # Prepare split will record examples associated to the split
-> 1049     self._prepare_split(split_generator, **prepare_split_kwargs)
   1050 except OSError as e:
   1051     raise OSError(
   1052         "Cannot find data file. "
   1053         + (self.manual_download_instructions or "")
   1054         + "\nOriginal error:\n"
   1055         + str(e)
   1056     ) from None

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1813, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1811 job_id = 0
   1812 with pbar:
-> 1813     for job_id, done, content in self._prepare_split_single(
   1814         gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1815     ):
   1816         if done:
   1817             result = content

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1958, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1956     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1957         e = e.__context__
-> 1958     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1960 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset

### Steps to reproduce the bug

from datasets import load_dataset

dataset = load_dataset("glue", "ax")

### Expected behavior

When generating the train split:
Generating train split:
0/0 [00:00<?, ? examples/s]

It raise the error:
DatasetGenerationError: An error occurred while generating the dataset


### Environment info

datasets-2.14.4.
Python 3.10

Comments:
I've tried clear the .cache file, doesn't work.

This issue happens on AWS sagemaker

This issue can happen if there is a directory named "glue" relative to the Python script with the `load_dataset` call (similar issue to this one: https://github.com/huggingface/datasets/issues/5228). Is this the case?

> This issue can happen if there is a directory named "glue" relative to the Python script with the `load_dataset` call (similar issue to this one: #5228). Is this the case?

Thats correct!
Sorry for my late response.