dalssy/data_del huggingface dataset:
GitHub issue: imagenet-1k downloads over and over
Author: seann999 (NONE)
State: closed | Created: 2023-11-06 02:58:58

### Describe the bug

What could be causing this?
```
$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset("imagenet-1k")
Downloading builder script: 100%|██████████| 4.72k/4.72k [00:00<00:00, 7.51MB/s]
Downloading readme: 100%|███████████████████| 85.4k/85.4k [00:00<00:00, 510kB/s]
Downloading extra modules: 100%|████████████| 46.4k/46.4k [00:00<00:00, 300kB/s]
Downloading data: 100%|████████████████████| 29.1G/29.1G [19:36<00:00, 24.8MB/s]
Downloading data: 100%|████████████████████| 29.3G/29.3G [08:38<00:00, 56.5MB/s]
Downloading data: 100%|████████████████████| 29.0G/29.0G [09:26<00:00, 51.2MB/s]
Downloading data: 100%|████████████████████| 29.2G/29.2G [09:38<00:00, 50.6MB/s]
Downloading data: 100%|███████████████████▉| 29.2G/29.2G [09:37<00:00, 44.1MB/s^Downloading data:   0%|                     | 106M/29.1G [00:05<23:49, 20.3MB/s]
```

### Steps to reproduce the bug

See above commands/code

### Expected behavior

imagenet-1k is downloaded

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.2.0-34-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- PyArrow version: 14.0.0
- Pandas version: 1.5.2
GitHub issue: Add CheXpert dataset for vision
Author: SauravMaheshkar (NONE)
State: open | Created: 2023-11-04 15:36:11
Labels: name: enhancement, name: dataset request

### Feature request

### Name 

**CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison**

### Paper

https://arxiv.org/abs/1901.07031

### Data

https://stanfordaimi.azurewebsites.net/datasets/8cbd9ed4-2eb9-4565-affc-111cf4f7ebe2

### Motivation

CheXpert is one of the fundamental models in medical image classification and can serve as a viable pre-training dataset for radiology classification or low-scale ablation / exploratory studies.

This could also serve as a good pre-training dataset for Kaggle competitions.

### Your contribution

Would love to make a PR and pre-process / get this into 🤗
GitHub issue: Add my dataset
Author: keyur536 (NONE)
State: closed | Created: 2023-11-02 20:59:52

## medical data

**Description:**
This dataset, named "medical data," is a collection of text data from various sources, carefully curated and cleaned for use in natural language processing (NLP) tasks. It consists of a diverse range of text, including articles, books, and online content, covering topics from science to literature.

**Citation:**
If applicable, please include a citation for this dataset to give credit to the original sources or contributors.

**Key Features:**
- Language: The text is primarily in English, but it may include content in other languages as well.
- Use Cases: This dataset is suitable for text classification, language modeling, sentiment analysis, and other NLP tasks.

**Usage:**
To access this dataset, use the `load_your_dataset` function provided in the `your_dataset.py` script within this repository. You can specify the dataset split you need, such as "train," "test," or "validation," to get the data for your specific task.

**Contributors:**
- [Keyur Chaudhari]

**Contact:**
If you have any questions or need assistance regarding this dataset, please feel free to contact [keyurchaudhari536@gmail.com].

Please note that this dataset is shared under a specific license, which can be found in the [LICENSE](link to your dataset's license) file. Make sure to review and adhere to the terms of the license when using this dataset for your projects.
GitHub issue: Fix for continuation behaviour on broken dataset archives due to starving download connections via HTTP-GET
Author: RuntimeRacer (NONE)
State: open | Created: 2023-11-02 17:28:23

This PR proposes a (slightly hacky) fix for an Issue that can occur when downloading large dataset parts over unstable connections.
The underlying issue is also being discussed in https://github.com/huggingface/datasets/issues/5594.

Issue Symptoms & Behaviour:
- Download of a large archive file during dataset download via HTTP-GET fails.
- An silent net exception (which I was unable to identify) is thrown within the `tqdm` download progress.
- Due to missing exception catch code, the above process just continues processing, assuming `http_get` completed successfully.
- Pending Archive file gets renamed to remove the `.incomplete` extension, despite not all data has been downloaded. 
- Also, for reasons I did not investigate, there seems to be no real integrity check for the downloaded files; or it does not detect this problem. This is especially problematic, since the downloader script won't retry downloading this archive after CRC-Checking, even if it is being manually restarted / executed again after running into errors on extraction.

Fix proposal: Adding a retry mechanic for HTTP-GET downloads, which adds the following behaviour:
- Download Progress Thread checks for download size validity in case the HTTP connection starves mid download. If the check fails, a RuntimeError is thrown
- Cache Downloader code with retry mechanic monitors for an exception thrown by the download progress thread, and retries download with updated `resume_size`.
- Cache Downloader will not mark incomplete files which have thrown an exception during download, and exceeded retries, as complete.
GitHub issue: Avoid redundant warning when encoding NumPy array as `Image`
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-02 16:37:58

Avoid a redundant warning in `encode_np_array` by removing the identity check as NumPy `dtype`s can be equal without having identical `id`s.

Additionally, fix "unreachable" checks in `encode_np_array`.
GitHub issue: Caching problem when deleting a dataset 
Author: clefourrier (MEMBER)
State: closed | Created: 2023-11-02 10:15:58

### Describe the bug

Pushing a dataset with n + m features to a repo which was deleted, but contained n features, will fail.

### Steps to reproduce the bug

1. Create a dataset with n features per row
2. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)`
3. Go on the hub, delete the repo at `YOUR_PATH`
4. Update your local dataset to have n + m features per row
5. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)` will fail because of a mismatch in features number


### Expected behavior

Step 5 should work or display a message to indicate the cache has not been cleared

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue: CI is broken: TypeError: Couldn't cast array
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-02 09:37:06

See: https://github.com/huggingface/datasets/actions/runs/6730567226/job/18293518039
```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```
GitHub issue: `Dataset.from_generator` should not try to download from HF GCS
Author: yundai424 (CONTRIBUTOR)
State: closed | Created: 2023-11-01 17:36:17

### Describe the bug

When using [`Dataset.from_generator`](https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/arrow_dataset.py#L1072) with `streaming=False`, the internal logic will call [`download_and_prepare`](https://github.com/huggingface/datasets/blob/main/src/datasets/io/generator.py#L47) which will attempt to download from HF GCS which is redundant, because user has already provided the generator from which the data should be drawn. 

If someone attempts to call `Dataset.from_generator` from an environment that doesn't have external internet access (for example internal production machine) and doesn't set `HF_DATASETS_OFFLINE=1`, this will result in process being stuck at building connection.

### Steps to reproduce the bug

```python
import datasets
def gen():
    for _ in range(100):
        yield {"text": "dummy text"}

dataset = datasets.Dataset.from_generator(gen)
```

A minimum example executed on any environment that doesn't have access to HF GCS can result in the error

### Expected behavior

`try_from_hf_gcs` should be set to False here https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/io/generator.py#L51

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
GitHub issue: TensorDataset format does not work with Trainer from transformers
Author: jinzzasol (NONE)
State: closed | Created: 2023-11-01 10:09:54

### Describe the bug

The model was built to do fine tunning on BERT model for relation extraction.

trainer.train() returns an error message ```TypeError: vars() argument must have __dict__ attribute``` when it has `train_dataset` generated from `torch.utils.data.TensorDataset`

However, in the document, the required data format is `torch.utils.data.TensorDataset`. 
![image](https://github.com/huggingface/datasets/assets/49014051/36fa34ac-3127-4c64-9580-9ab736136d83)

Transformers trainer is supposed to accept the train_dataset in the format of torch.utils.data.TensorDataset, but it returns error message *"TypeError: vars() argument must have __dict__ attribute"*

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-5df728c929a2> in <cell line: 1>()
----> 1 trainer.train()
      2 trainer.evaluate(test_dataset)

9 frames
/usr/local/lib/python3.10/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    107 
    108     if not isinstance(features[0], Mapping):
--> 109         features = [vars(f) for f in features]
    110     first = features[0]
    111     batch = {}

TypeError: vars() argument must have __dict__ attribute
```

### Steps to reproduce the bug

Create train_dataset using `torch.utils.data.TensorDataset`, for instance,

```train_dataset = torch.utils.data.TensorDataset(train_input_ids, train_attention_masks, train_labels)```

Feed this `train_dataset` to your trainer and run trainer.train

```
trainer = Trainer(model,
                  training_args,
                  train_dataset=train_dataset,
                  eval_dataset=dev_dataset,
                  compute_metrics=compute_metrics,
                  )
```

### Expected behavior

Trainer should start training

### Environment info

It is running on Google Colab

- `datasets` version: 2.14.6
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue: Multi process map did not load cache file correctly
Author: enze5088 (NONE)
State: closed | Created: 2023-11-01 06:36:54

### Describe the bug

When I  was training model on Multiple GPUs by DDP, the dataset is tokenized multiple times after main process. 
![1698820541284](https://github.com/huggingface/datasets/assets/14285786/0b2fe054-54d8-4e00-96e6-6ca5b69e662b)

![1698820501568](https://github.com/huggingface/datasets/assets/14285786/dd62bf6f-a58f-41bf-9848-ea4fb3b62b9b)

Code is modified from  [run_clm.py](https://github.com/huggingface/transformers/blob/7d8ff3629b2725ec43ace99c1a6e87ac1978d433/examples/pytorch/language-modeling/run_clm.py#L484)


### Steps to reproduce the bug

```
block_size = data_args.block_size
IGNORE_INDEX = -100
Ignore_Input = False

def tokenize_function(examples):
    sources = []
    targets = []
    for instruction, inputs, output in zip(examples['instruction'], examples['input'], examples['output']):

        source = instruction + inputs 

        target = f"{output}{tokenizer.eos_token}"

        sources.append(source)
        targets.append(target)

    tokenized_sources = tokenizer(sources, return_attention_mask=False)

    tokenized_targets = tokenizer(targets, return_attention_mask=False,
                                  add_special_tokens=False
                                  )

    all_input_ids = []
    all_labels = []
    for s, t in zip(tokenized_sources['input_ids'], tokenized_targets['input_ids']):
        if len(s) > block_size and Ignore_Input == False:
            # print(s)
            continue
        input_ids = torch.LongTensor(s + t)[:block_size]
        if Ignore_Input:
            labels = torch.LongTensor([IGNORE_INDEX] * len(s) + t)[:block_size]
        else:
            labels = input_ids
        assert len(input_ids) == len(labels)
        all_input_ids.append(input_ids)
        all_labels.append(labels)

    results = {
        'input_ids': all_input_ids,
        'labels': all_labels,

    }
    return results



with training_args.main_process_first(desc="dataset map tokenization ", local=False):
    # print('local_rank',training_args.local_rank)
    if not data_args.streaming:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset ",
        )
    else:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            remove_columns=column_names,
            desc="Running tokenizer on dataset "
        )
```

### Expected behavior

This code should only tokenize the dataset in the main process, and the other processes load the dataset after waiting

### Environment info

transformers == 4.34.1
datasets == 2.14.5
GitHub issue: with_format() function returns bytes instead of PIL images even when image column is not part of "columns"
Author: leot13 (NONE)
State: closed | Created: 2023-10-31 11:10:48

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6
GitHub issue: Parquet size grows exponential for categorical data
Author: aseganti (NONE)
State: closed | Created: 2023-10-31 10:29:02

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4
GitHub issue:  Add support for `Sequence(Audio/Image)` feature in `push_to_hub`
Author: Laurent2916 (CONTRIBUTOR)
State: closed | Created: 2023-10-27 14:39:57
Labels: name: enhancement

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature
GitHub issue: Stuck in "Resolving data files..."
Author: Luciennnnnnn (NONE)
State: open | Created: 2023-10-27 12:01:51

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
GitHub issue: Mounting datasets cache fails due to absolute paths.
Author: charliebudd (NONE)
State: closed | Created: 2023-10-27 08:20:27

### Describe the bug

Creating a datasets cache and mounting this into, for example, a docker container, renders the data unreadable due to absolute paths written into the cache.

### Steps to reproduce the bug

1. Create a datasets cache by downloading some data
2. Mount the dataset folder into a docker container or remote system.
3. (Edit) Set `HF_HOME` or `HF_DATASET_CACHE` to point to the mounted cache.
4. Attempt to access the data from within the docker container.
5. An error is thrown saying no file exists at \<absolute path to original cache location\>

### Expected behavior

The data is loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
GitHub issue: Allow passing a multiprocessing context to functions that support `num_proc`
Author: bryant1410 (CONTRIBUTOR)
State: open | Created: 2023-10-27 02:31:16
Labels: name: enhancement

### Feature request

Allow specifying [a multiprocessing context](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) to functions that support `num_proc` or use multiprocessing pools. For example, the following could be done:

```python
dataset = dataset.map(_func, num_proc=2, mp_context=multiprocess.get_context("spawn"))
```

Or at least the multiprocessing start method ("fork", "spawn", "fork_server" or `None`):

```python
dataset = dataset.map(_func, num_proc=2, mp_start_method="spawn")
```

Another option could be passing the `pool` as an argument.

### Motivation

By default, `multiprocess` (the `multiprocessing`-fork library that this repo uses) uses the "fork" start method for multiprocessing pools (for the default context). It could be changed by using `set_start_method`. However, this conditions the multiprocessing start method from all processing in a Python program that uses the default context, because [you can't call that function more than once](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:~:text=set_start_method()%20should%20not%20be%20used%20more%20than%20once%20in%20the%20program.). My proposal is to allow using a different multiprocessing context, not to condition the whole Python program.

One reason to change the start method is that "fork" (the default) makes child processes likely deadlock if thread pools were created before (and also this is not supported by POSIX). For example, this happens when using PyTorch because OpenMP threads are used for CPU intra-op parallelism, which is enabled by default (e.g., for context see [`torch.set_num_threads`](https://pytorch.org/docs/stable/generated/torch.set_num_threads.html)). This can also be fixed by setting `torch.set_num_threads(1)` (or similarly by other methods) but this conditions the whole Python program as it can only be set once to guarantee its behavior (similarly to). There are noticeable performance differences when setting this number to 1 even when using GPU(s). Using, e.g., a "spawn" start method would solve this issue.

For more context, see:

* https://discuss.huggingface.co/t/dataset-map-stuck-with-torch-set-num-threads-set-to-2-or-larger/37984
* https://discuss.huggingface.co/t/using-num-proc-1-in-dataset-map-hangs/44310

### Your contribution

I'd be happy to review a PR that makes such a change. And if you really don't have the bandwidth for it, I'd consider creating one.
GitHub issue: More hub centric docs
Author: lhoestq (MEMBER)
State: closed | Created: 2023-10-26 16:54:46

Let's have more hub-centric documentation in the datasets docs

Tutorials
- Add “Configure the dataset viewer” page
- Change order:
  - Overview
    - and more focused on the Hub rather than the library
  - Then all the hub related things
    - and mention how to read/write with other tools like pandas
  - Then all the datasets lib related things in a subsection

Also:
- Rename “know your dataset” page to “Explore your dataset”
- Remove “Evaluate Predictions” page since it's 'evaluate' stuff (or move to legacy section ?)

TODO:
- [ ] write the “Configure the dataset viewer” page
GitHub issue: load_dataset save_to_disk load_from_disk error
Author: brisker (NONE)
State: closed | Created: 2023-10-26 03:47:06

### Describe the bug

datasets version： 2.10.1
I `load_dataset `and `save_to_disk` sucessfully on windows10( **and I `load_from_disk(/LLM/data/wiki)` succcesfully on windows10**), and I copy the dataset `/LLM/data/wiki`
into a ubuntu system, but when I `load_from_disk(/LLM/data/wiki)` on ubuntu,  something weird happens:


```
load_from_disk('/LLM/data/wiki')
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1874, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1309, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1543, in load_from_disk
    fs_token_paths = fsspec.get_fs_token_paths(dataset_path, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 610, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 325, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    raise ValueError(f"Protocol not known: {protocol}")
ValueError: Protocol not known: /LLM/data/wiki
```
It seems that something went wrong on the arrow file?
How can I solve this , since currently I can not save_to_disk on ubuntu system

### Steps to reproduce the bug

datasets version： 2.10.1

### Expected behavior

datasets version： 2.10.1

### Environment info

datasets version： 2.10.1
GitHub issue: Error loading wikitext data    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
Author: Ahmed-Roushdy (NONE)
State: closed | Created: 2023-10-25 21:55:31

I was trying to load  the wiki dataset, but i got this error 

    traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/load.py", line 1804, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/builder.py", line 1108, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
GitHub issue: Different objects are returned from calls that should be returning the same kind of object.
Author: phalexo (NONE)
State: open | Created: 2023-10-25 17:08:39

### Describe the bug

        1.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
        2.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.
GitHub issue: Can't load ds = load_dataset("imdb")
Author: vivianc2 (NONE)
State: closed | Created: 2023-10-25 13:29:51

### Describe the bug

I did `from datasets import load_dataset, load_metric` and then `ds = load_dataset("imdb")` and it gave me the error:
ExpectedMoreDownloadedFiles: {'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'}

I tried doing `ds = load_dataset("imdb",download_mode="force_redownload")` as well as reinstalling dataset. I still face this problem.

### Steps to reproduce the bug

1. from datasets import load_dataset, load_metric
2. ds = load_dataset("imdb")

### Expected behavior

It should load and give me this when I run `ds`

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-164-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.16.2
- PyArrow version: 13.0.0
- Pandas version: 2.0.2
GitHub issue: Incorrect example code in 'Create a dataset' docs
Author: rwood-97 (NONE)
State: closed | Created: 2023-10-24 11:01:21

### Describe the bug

On [this](https://huggingface.co/docs/datasets/create_dataset) page, the example code for loading in images and audio is incorrect.

Currently, examples are:

``` python
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")
```
and 
``` python
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I'm pretty sure the imports are wrong and should be:

``` python 
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I am happy to update this if this is right but just wanted to check before making any changes.

### Steps to reproduce the bug

Go to https://huggingface.co/docs/datasets/create_dataset

### Expected behavior

N/A

### Environment info

N/A
GitHub issue: Fix UnboundLocalError if preprocessing returns an empty list
Author: cwallenwein (CONTRIBUTOR)
State: closed | Created: 2023-10-24 08:38:43

If this tokenization function is used with IterableDatasets and no sample is as big as the context length, `input_batch` will be an empty list.
```
def tokenize(batch, tokenizer, context_length):
    outputs = tokenizer(
        batch["text"],
        truncation=True,
        max_length=context_length,
        return_overflowing_tokens=True,
        return_length=True
    )
    input_batch = []
    for length, input_ids in zip(outputs["length"], outputs["input_ids"]):
        if length == context_length:
            input_batch.append(input_ids)
    return {"input_ids": input_batch}

dataset.map(tokenize, batched=True, batch_size=batch_size, fn_kwargs={"context_length": context_length, "tokenizer": tokenizer}, remove_columns=dataset.column_names)
```

This will throw the following error: UnboundLocalError: local variable 'batch_idx' referenced before assignment, because the for loop was not executed a single time

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```

Some of the possible solutions

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
try:
     current_idx += batch_idx + 1
except:
     current_idx += 1
```

or 

```
batch_idx = 0
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```
GitHub issue: support squad structure datasets using a YAML parameter
Author: MajdTannous1 (NONE)
State: open | Created: 2023-10-23 17:55:37
Labels: name: enhancement

### Feature request

Since the squad structure is widely used, I think it could be beneficial to support it using a YAML parameter.
could you implement automatic data loading of squad-like data using squad JSON format, to read it from JSON files and view it in the correct squad structure.
The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers

### Motivation

Dataset repo requires arbitrary Python code execution

### Your contribution

The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers
train and dev sets in squad structure JSON files
GitHub issue: Pin supported upper version of fsspec
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-23 10:44:16

Pin upper version of `fsspec` to avoid disruptions introduced by breaking changes (and the need of urgent patch releases with hotfixes) on each release on their side. See:
- #6331
- #6210 
- #5731
- #5617
- #5447

I propose that we explicitly test, introduce fixes and support each new `fsspec` version release.

CC: @LysandreJik
GitHub issue: datasets.filesystems: fix is_remote_filesystems
Author: ap-- (CONTRIBUTOR)
State: closed | Created: 2023-10-23 09:17:54

Close #6330, close #6333.

`fsspec.implementations.LocalFilesystem.protocol`
was changed from `str` "file" to `tuple[str,...]` ("file", "local") in `fsspec>=2023.10.0`

This commit supports both styles.
GitHub issue: Support fsspec 2023.10.0
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-23 09:14:53

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:
- #6331

Related to issue:
- #6330

As @ZachNagengast suggested, the issue might be related to:
- https://github.com/fsspec/filesystem_spec/pull/1381
GitHub issue: Replace deprecated license_file in setup.cfg
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-23 09:05:26

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
```
GitHub issue: Temporarily pin fsspec < 2023.10.0
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-23 08:51:50

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```
GitHub issue: Latest fsspec==2023.10.0 issue with streaming datasets
Author: ZachNagengast (CONTRIBUTOR)
State: closed | Created: 2023-10-22 20:57:10

### Describe the bug

Loading a streaming dataset with this version of fsspec fails with the following error:

`NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.`

I suspect the issue is with this PR 

https://github.com/fsspec/filesystem_spec/pull/1381

### Steps to reproduce the bug

1. Upgrade fsspec to version `2023.10.0`
2. Attempt to load a streaming dataset e.g. `load_dataset("laion/gpt4v-emotion-dataset", split="train", streaming=True)`
3. Observe the following exception:

```
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/load.py", line 2146, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/builder.py", line 1318, in as_streaming_dataset
    raise NotImplementedError(
NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
```

### Expected behavior

Should stream the dataset as normal.

### Environment info

datasets@main
fsspec==2023.10.0
GitHub issue: شبکه های متن به گفتار ابتدا متن داده شده را به بازنمایی میانی
Author: shabnam706 (NONE)
State: closed | Created: 2023-10-22 11:07:46

شبکه های متن به گفتار ابتدا متن داده شده را به بازنمایی میانی
GitHub issue: FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`
Author: yzhangcs (CONTRIBUTOR)
State: closed | Created: 2023-10-21 12:27:03

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`
GitHub issue: Conversion to Arrow fails due to wrong type heuristic
Author: jphme (NONE)
State: closed | Created: 2023-10-20 23:20:58

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values. 

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.


### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
GitHub issue: Loading dataset from large GCS bucket very slow since 2.14
Author: jbcdnr (NONE)
State: open | Created: 2023-10-20 12:59:55

### Describe the bug

Since updating to >2.14 we have very slow access to our parquet files on GCS when loading a dataset (>30 min vs 3s). Our GCS bucket has many objects and resolving globs is very slow. I could track down the problem to this change:
https://github.com/huggingface/datasets/blame/bade7af74437347a760830466eb74f7a8ce0d799/src/datasets/data_files.py#L348 
The underlying implementation with gcsfs is really slow. Could you go back to the old way if we are simply giving the parquet files and no glob pattern?

Thank you.




### Steps to reproduce the bug

Load a dataset from a GCS bucket that has many files.

### Expected behavior

Used to be fast (3s) in 2.13

### Environment info

datasets==2.14.5
GitHub issue: Fix regex `get_data_files` formatting for base paths
Author: ZachNagengast (CONTRIBUTOR)
State: closed | Created: 2023-10-19 19:45:10

With this pr https://github.com/huggingface/datasets/pull/6309, it is formatting the entire base path into regex, which results in the undesired formatting error `doesn't match the pattern` because of the line in `glob_pattern_to_regex`: `.replace("//", "/")`:
- Input: `hf://datasets/...`
- Output: `hf:/datasets/...`

This fix will only convert the `split_pattern` to regex and keep the `base_path` unchanged.

cc @albertvillanova hopefully this still works with your implementation
GitHub issue: Dataset slice splits can't load training and validation at the same time
Author: timlac (NONE)
State: closed | Created: 2023-10-19 16:09:22

### Describe the bug

According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:

`train_test_ds = datasets.load_dataset("bookcorpus", split="train+test")`

to load the train and test sets from the dataset. 

However executing the equivalent code:

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

only yields the following output:

> Dataset({
>     features: ['file', 'audio', 'label'],
>     num_rows: 54175
> })

Where loading the dataset without the split argument yields:

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     validation: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 6798
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

Thus, the API seems to be broken in this regard. 

This is a bit annoying since I want to be able to use the split argument with `split="train[:10%]+test[:10%]"` to have smaller dataset to work with when validating my model is working correctly. 

### Steps to reproduce the bug

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

### Expected behavior

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

### Environment info

```
import datasets
print(datasets.__version__)
```
> 2.14.5

```
import sys
print(sys.version)
```
> 3.9.17 (main, Jul  5 2023, 20:41:20) 
> [GCC 11.2.0]
GitHub issue: Datasets.map is severely broken
Author: phalexo (NONE)
State: open | Created: 2023-10-19 12:19:33

### Describe the bug

Regardless of how many cores I used, I have 16 or 32 threads, map slows down to a crawl at around 80% done, lingers maybe until 97% extremely slowly and NEVER finishes the job. It just hangs.

After watching this for 27 hours I control-C out of it. Until the end one process appears to be doing something, but it never ends.

I saw some comments about fast tokenizers using Rust and all and tried different variations. NOTHING works.



### Steps to reproduce the bug

Running it without breaking the dataset into parts results in the same behavior. The loop was an attempt to see if this was a RAM issue. 

for idx in range(100):
    dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
    dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False, num_proc=1, remove_columns=["text", "meta"])
    dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")


### Expected behavior

I expect map to run at more or less the same speed it starts with and FINISH its processing. 

### Environment info

Python 3.8, same with 3.10 makes no difference.
Ubuntu 20.04,
GitHub issue: Deterministic set hash
Author: lhoestq (MEMBER)
State: closed | Created: 2023-10-19 12:19:13

Sort the items in a set according to their `datasets.fingerprint.Hasher.hash` hash to get a deterministic hash of sets.

This is useful to get deterministic hashes of tokenizers that use a trie based on python sets.

reported in https://github.com/huggingface/datasets/issues/3847
GitHub issue: sentiment140 dataset unavailable
Author: AndreasKarasenko (NONE)
State: closed | Created: 2023-10-19 11:25:21

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3    
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
GitHub issue: Fix loading Hub datasets with CSV metadata file
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-19 10:21:34

Currently, the reading of the metadata file infers the file extension (.jsonl or .csv) from the passed filename. However, downloaded files from the Hub don't have file extension. For example:
- the original file: `hf://datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5916a4-16977085077831/metadata.jsonl`
- corresponds to the downloaded path: `/tmp/pytest-of-username/pytest-46/cache/datasets/downloads/9f5374dbb470f711f6b89d66a5eec1f19cc96324b26bcbebe29138bda6cb20e6`, which does not have extension

In the case where the metadata file does not have an extension, the reader assumes it is a JSONL file, thus the reported error when trying to read a CSV file as a JSONL one: `ArrowInvalid: JSON parse error: Invalid value. in row 0`

This behavior was introduced by:
- #4837

This PR extracts the metadata file extension from the original filename (instead of the downloaded one) and passes it as a parameter to the read_metadata function.

Fix #6315.
GitHub issue: Hub datasets with CSV metadata raise ArrowInvalid: JSON parse error: Invalid value. in row 0
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-10-19 10:11:29
Labels: name: bug

When trying to load a Hub dataset that contains a CSV metadata file, it raises an `ArrowInvalid` error:
```
E   pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0

pyarrow/error.pxi:100: ArrowInvalid
```

See: https://huggingface.co/datasets/lukarape/public_small_papers/discussions/1
GitHub issue: Fix commit message formatting in multi-commit uploads
Author: qgallouedec (MEMBER)
State: closed | Created: 2023-10-19 07:53:56

Currently, the commit message keeps on adding:

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00000-of-00002) (part 00001-of-00002)`

Introduced in https://github.com/huggingface/datasets/pull/6269

This PR fixes this issue to have

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00001-of-00002)`
GitHub issue: docs: resolving namespace conflict, refactored variable 
Author: smty2018 (CONTRIBUTOR)
State: closed | Created: 2023-10-18 16:10:59

In docs of about_arrow.md, in the below example code
![image](https://github.com/huggingface/datasets/assets/74114936/fc70e152-e15f-422e-949a-1c4c4c9aa116)
The variable name 'time' was being used in a way that could potentially lead to a namespace conflict with Python's built-in 'time' module. It is not a good convention and can lead to unintended variable shadowing for any user re-using the example code.
To ensure code clarity, and prevent potential naming conflicts renamed the variable 'time' to 'elapsed_time' in the example  code.
GitHub issue: Add return_file_name in load_dataset
Author: juliendenize (NONE)
State: closed | Created: 2023-10-17 13:36:57

Proposition to fix #5806.

Added an optional parameter `return_file_name` in the dataset builder config. When set to `True`, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:
- for arrow-based: a column is concatenated after the table is cast.
- for folder-based: `dataset.info.features` has the entry `file_name` and the original file name is passed to the `sample_metadata` dictionary. 

The difference in behavior might be a concern, also I do not know whether the `file_name` should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.