Dataset lib seems to broke after fssec lib update
Opened by sleepingcat4 (closed)

### Describe the bug

I am facing an issue since today where HF's dataset is acting weird and in some instances failure to recognise a valid dataset entirely, I think it is happening due to recent change in `fsspec` lib as using this command fixed it for me in one-time: `!pip install -U datasets huggingface_hub fsspec`


### Steps to reproduce the bug

from datasets import load_dataset

def download_hf():
  dataset_name = input("Enter the dataset name: ")
  subset_name = input("Enter subset name: ")
  ds = load_dataset(dataset_name, name=subset_name)
  for split in ds:
    ds[split].to_pandas().to_csv(f"{subset_name}.csv", index=False)

download_hf()

### Expected behavior

```
Downloading readme: 100%
 1.55k/1.55k [00:00<00:00, 121kB/s]
Downloading data files: 100%
 1/1 [00:00<00:00,  2.06it/s]

Downloading data:   0%|          | 0.00/54.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 54.2k/54.2k [00:00<00:00, 121kB/s]
Extracting data files: 100%
 1/1 [00:00<00:00, 35.17it/s]
Generating test split: 
 140/0 [00:00<00:00, 2628.62 examples/s]
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[<ipython-input-2-12ab305b0e77>](https://localhost:8080/#) in <cell line: 0>()
      8     ds[split].to_pandas().to_csv(f"{subset_name}.csv", index=False)
      9 
---> 10 download_hf()

2 frames
[/usr/local/lib/python3.11/dist-packages/datasets/builder.py](https://localhost:8080/#) in as_dataset(self, split, run_post_process, verification_mode, ignore_verifications, in_memory)
   1171         is_local = not is_remote_filesystem(self._fs)
   1172         if not is_local:
-> 1173             raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
   1174         if not os.path.exists(self._output_dir):
   1175             raise FileNotFoundError(

NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```
OR
```
Traceback (most recent call last):
  File "e:\Fuck\download-data\mcq_dataset.py", line 10, in <module>
    download_hf()
  File "e:\Fuck\download-data\mcq_dataset.py", line 6, in download_hf
    ds = load_dataset(dataset_name, name=subset_name)
  File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
  File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\load.py", line 1917, in dataset_module_factory    
    raise e1 from None
  File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\load.py", line 1867, in dataset_module_factory    
    raise DatasetNotFoundError(f"Dataset '{path}' doesn't exist on the Hub or cannot be accessed.") from e
datasets.exceptions.DatasetNotFoundError: Dataset 'dataset repo_id' doesn't exist on the Hub or cannot be accessed.
```

### Environment info

colab and 3.10 local system

Comments:
Hi, can you try updating `datasets` ? Colab still installs `datasets` 2.x by default, instead of 3.x

It would be cool to also report this to google colab, they have a GitHub repo for this IIRC

@lhoestq I have updated it to `datasets==3.6.0` and now there's an entirely different issue on colab while locally its fine. 

```
/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(
README.md: 100%
 2.88k/2.88k [00:00<00:00, 166kB/s]
suno.jsonl.zst: 100%
 221M/221M [00:05<00:00, 48.6MB/s]
Generating train split: 
 18633/0 [00:01<00:00, 13018.92 examples/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1870                     try:
-> 1871                         writer.write_table(table)
   1872                     except CastError as cast_error:

17 frames
TypeError: Couldn't cast array of type
struct<id: string, type: string, infill: bool, source: string, continue_at: double, infill_dur_s: double, infill_end_s: double, infill_start_s: double, include_future_s: double, include_history_s: double, infill_context_end_s: double, infill_context_start_s: int64>
to
{'id': Value(dtype='string', id=None), 'type': Value(dtype='string', id=None), 'infill': Value(dtype='bool', id=None), 'source': Value(dtype='string', id=None), 'continue_at': Value(dtype='float64', id=None), 'include_history_s': Value(dtype='float64', id=None)}

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1896             if isinstance(e, DatasetGenerationError):
   1897                 raise
-> 1898             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1899 
   1900         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

@lhoestq opps sorry the dataset was in .zst which was causing this error rather than being a datasets library fault. After upgrading dataset version Colab is working fine.