jamesLeeeeeee/datasets-github-issues huggingface dataset:
Multi process map did not load cache file correctly
Author: enze5088 | Pull request: no | State: closed

### Describe the bug

When I  was training model on Multiple GPUs by DDP, the dataset is tokenized multiple times after main process. 
![1698820541284](https://github.com/huggingface/datasets/assets/14285786/0b2fe054-54d8-4e00-96e6-6ca5b69e662b)

![1698820501568](https://github.com/huggingface/datasets/assets/14285786/dd62bf6f-a58f-41bf-9848-ea4fb3b62b9b)

Code is modified from  [run_clm.py](https://github.com/huggingface/transformers/blob/7d8ff3629b2725ec43ace99c1a6e87ac1978d433/examples/pytorch/language-modeling/run_clm.py#L484)


### Steps to reproduce the bug

```
block_size = data_args.block_size
IGNORE_INDEX = -100
Ignore_Input = False

def tokenize_function(examples):
    sources = []
    targets = []
    for instruction, inputs, output in zip(examples['instruction'], examples['input'], examples['output']):

        source = instruction + inputs 

        target = f"{output}{tokenizer.eos_token}"

        sources.append(source)
        targets.append(target)

    tokenized_sources = tokenizer(sources, return_attention_mask=False)

    tokenized_targets = tokenizer(targets, return_attention_mask=False,
                                  add_special_tokens=False
                                  )

    all_input_ids = []
    all_labels = []
    for s, t in zip(tokenized_sources['input_ids'], tokenized_targets['input_ids']):
        if len(s) > block_size and Ignore_Input == False:
            # print(s)
            continue
        input_ids = torch.LongTensor(s + t)[:block_size]
        if Ignore_Input:
            labels = torch.LongTensor([IGNORE_INDEX] * len(s) + t)[:block_size]
        else:
            labels = input_ids
        assert len(input_ids) == len(labels)
        all_input_ids.append(input_ids)
        all_labels.append(labels)

    results = {
        'input_ids': all_input_ids,
        'labels': all_labels,

    }
    return results



with training_args.main_process_first(desc="dataset map tokenization ", local=False):
    # print('local_rank',training_args.local_rank)
    if not data_args.streaming:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset ",
        )
    else:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            remove_columns=column_names,
            desc="Running tokenizer on dataset "
        )
```

### Expected behavior

This code should only tokenize the dataset in the main process, and the other processes load the dataset after waiting

### Environment info

transformers == 4.34.1
datasets == 2.14.5
with_format() function returns bytes instead of PIL images even when image column is not part of "columns"
Author: leot13 | Pull request: no | State: closed

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6
Parquet size grows exponential for categorical data
Author: aseganti | Pull request: no | State: closed

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4
Add support for `Sequence(Audio/Image)` feature in `push_to_hub`
Author: Laurent2916 | Pull request: no | State: closed

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature
Stuck in "Resolving data files..."
Author: Luciennnnnnn | Pull request: no | State: open

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
Mounting datasets cache fails due to absolute paths.
Author: charliebudd | Pull request: no | State: closed

### Describe the bug

Creating a datasets cache and mounting this into, for example, a docker container, renders the data unreadable due to absolute paths written into the cache.

### Steps to reproduce the bug

1. Create a datasets cache by downloading some data
2. Mount the dataset folder into a docker container or remote system.
3. (Edit) Set `HF_HOME` or `HF_DATASET_CACHE` to point to the mounted cache.
4. Attempt to access the data from within the docker container.
5. An error is thrown saying no file exists at \<absolute path to original cache location\>

### Expected behavior

The data is loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
Allow passing a multiprocessing context to functions that support `num_proc`
Author: bryant1410 | Pull request: no | State: open

### Feature request

Allow specifying [a multiprocessing context](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) to functions that support `num_proc` or use multiprocessing pools. For example, the following could be done:

```python
dataset = dataset.map(_func, num_proc=2, mp_context=multiprocess.get_context("spawn"))
```

Or at least the multiprocessing start method ("fork", "spawn", "fork_server" or `None`):

```python
dataset = dataset.map(_func, num_proc=2, mp_start_method="spawn")
```

Another option could be passing the `pool` as an argument.

### Motivation

By default, `multiprocess` (the `multiprocessing`-fork library that this repo uses) uses the "fork" start method for multiprocessing pools (for the default context). It could be changed by using `set_start_method`. However, this conditions the multiprocessing start method from all processing in a Python program that uses the default context, because [you can't call that function more than once](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:~:text=set_start_method()%20should%20not%20be%20used%20more%20than%20once%20in%20the%20program.). My proposal is to allow using a different multiprocessing context, not to condition the whole Python program.

One reason to change the start method is that "fork" (the default) makes child processes likely deadlock if thread pools were created before (and also this is not supported by POSIX). For example, this happens when using PyTorch because OpenMP threads are used for CPU intra-op parallelism, which is enabled by default (e.g., for context see [`torch.set_num_threads`](https://pytorch.org/docs/stable/generated/torch.set_num_threads.html)). This can also be fixed by setting `torch.set_num_threads(1)` (or similarly by other methods) but this conditions the whole Python program as it can only be set once to guarantee its behavior (similarly to). There are noticeable performance differences when setting this number to 1 even when using GPU(s). Using, e.g., a "spawn" start method would solve this issue.

For more context, see:

* https://discuss.huggingface.co/t/dataset-map-stuck-with-torch-set-num-threads-set-to-2-or-larger/37984
* https://discuss.huggingface.co/t/using-num-proc-1-in-dataset-map-hangs/44310

### Your contribution

I'd be happy to review a PR that makes such a change. And if you really don't have the bandwidth for it, I'd consider creating one.
More hub centric docs
Author: lhoestq | Pull request: yes | State: closed

Let's have more hub-centric documentation in the datasets docs

Tutorials
- Add “Configure the dataset viewer” page
- Change order:
  - Overview
    - and more focused on the Hub rather than the library
  - Then all the hub related things
    - and mention how to read/write with other tools like pandas
  - Then all the datasets lib related things in a subsection

Also:
- Rename “know your dataset” page to “Explore your dataset”
- Remove “Evaluate Predictions” page since it's 'evaluate' stuff (or move to legacy section ?)

TODO:
- [ ] write the “Configure the dataset viewer” page
load_dataset save_to_disk load_from_disk error
Author: brisker | Pull request: no | State: closed

### Describe the bug

datasets version： 2.10.1
I `load_dataset `and `save_to_disk` sucessfully on windows10( **and I `load_from_disk(/LLM/data/wiki)` succcesfully on windows10**), and I copy the dataset `/LLM/data/wiki`
into a ubuntu system, but when I `load_from_disk(/LLM/data/wiki)` on ubuntu,  something weird happens:


```
load_from_disk('/LLM/data/wiki')
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1874, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1309, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1543, in load_from_disk
    fs_token_paths = fsspec.get_fs_token_paths(dataset_path, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 610, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 325, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    raise ValueError(f"Protocol not known: {protocol}")
ValueError: Protocol not known: /LLM/data/wiki
```
It seems that something went wrong on the arrow file?
How can I solve this , since currently I can not save_to_disk on ubuntu system

### Steps to reproduce the bug

datasets version： 2.10.1

### Expected behavior

datasets version： 2.10.1

### Environment info

datasets version： 2.10.1
Error loading wikitext data    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
Author: Ahmed-Roushdy | Pull request: no | State: closed

I was trying to load  the wiki dataset, but i got this error 

    traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/load.py", line 1804, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/builder.py", line 1108, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
Different objects are returned from calls that should be returning the same kind of object.
Author: phalexo | Pull request: no | State: open

### Describe the bug

        1.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
        2.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.
Can't load ds = load_dataset("imdb")
Author: vivianc2 | Pull request: no | State: closed

### Describe the bug

I did `from datasets import load_dataset, load_metric` and then `ds = load_dataset("imdb")` and it gave me the error:
ExpectedMoreDownloadedFiles: {'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'}

I tried doing `ds = load_dataset("imdb",download_mode="force_redownload")` as well as reinstalling dataset. I still face this problem.

### Steps to reproduce the bug

1. from datasets import load_dataset, load_metric
2. ds = load_dataset("imdb")

### Expected behavior

It should load and give me this when I run `ds`

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-164-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.16.2
- PyArrow version: 13.0.0
- Pandas version: 2.0.2
Incorrect example code in 'Create a dataset' docs
Author: rwood-97 | Pull request: no | State: closed

### Describe the bug

On [this](https://huggingface.co/docs/datasets/create_dataset) page, the example code for loading in images and audio is incorrect.

Currently, examples are:

``` python
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")
```
and 
``` python
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I'm pretty sure the imports are wrong and should be:

``` python 
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I am happy to update this if this is right but just wanted to check before making any changes.

### Steps to reproduce the bug

Go to https://huggingface.co/docs/datasets/create_dataset

### Expected behavior

N/A

### Environment info

N/A
Fix UnboundLocalError if preprocessing returns an empty list
Author: cwallenwein | Pull request: yes | State: closed

If this tokenization function is used with IterableDatasets and no sample is as big as the context length, `input_batch` will be an empty list.
```
def tokenize(batch, tokenizer, context_length):
    outputs = tokenizer(
        batch["text"],
        truncation=True,
        max_length=context_length,
        return_overflowing_tokens=True,
        return_length=True
    )
    input_batch = []
    for length, input_ids in zip(outputs["length"], outputs["input_ids"]):
        if length == context_length:
            input_batch.append(input_ids)
    return {"input_ids": input_batch}

dataset.map(tokenize, batched=True, batch_size=batch_size, fn_kwargs={"context_length": context_length, "tokenizer": tokenizer}, remove_columns=dataset.column_names)
```

This will throw the following error: UnboundLocalError: local variable 'batch_idx' referenced before assignment, because the for loop was not executed a single time

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```

Some of the possible solutions

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
try:
     current_idx += batch_idx + 1
except:
     current_idx += 1
```

or 

```
batch_idx = 0
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```
support squad structure datasets using a YAML parameter
Author: MajdTannous1 | Pull request: no | State: open

### Feature request

Since the squad structure is widely used, I think it could be beneficial to support it using a YAML parameter.
could you implement automatic data loading of squad-like data using squad JSON format, to read it from JSON files and view it in the correct squad structure.
The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers

### Motivation

Dataset repo requires arbitrary Python code execution

### Your contribution

The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers
train and dev sets in squad structure JSON files
Pin supported upper version of fsspec
Author: albertvillanova | Pull request: yes | State: closed

Pin upper version of `fsspec` to avoid disruptions introduced by breaking changes (and the need of urgent patch releases with hotfixes) on each release on their side. See:
- #6331
- #6210 
- #5731
- #5617
- #5447

I propose that we explicitly test, introduce fixes and support each new `fsspec` version release.

CC: @LysandreJik
Support fsspec 2023.10.0
Author: albertvillanova | Pull request: no | State: closed

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:
- #6331

Related to issue:
- #6330

As @ZachNagengast suggested, the issue might be related to:
- https://github.com/fsspec/filesystem_spec/pull/1381
Replace deprecated license_file in setup.cfg
Author: albertvillanova | Pull request: yes | State: closed

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
```
Temporarily pin fsspec < 2023.10.0
Author: albertvillanova | Pull request: yes | State: closed

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```
Latest fsspec==2023.10.0 issue with streaming datasets
Author: ZachNagengast | Pull request: no | State: closed

### Describe the bug

Loading a streaming dataset with this version of fsspec fails with the following error:

`NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.`

I suspect the issue is with this PR 

https://github.com/fsspec/filesystem_spec/pull/1381

### Steps to reproduce the bug

1. Upgrade fsspec to version `2023.10.0`
2. Attempt to load a streaming dataset e.g. `load_dataset("laion/gpt4v-emotion-dataset", split="train", streaming=True)`
3. Observe the following exception:

```
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/load.py", line 2146, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/builder.py", line 1318, in as_streaming_dataset
    raise NotImplementedError(
NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
```

### Expected behavior

Should stream the dataset as normal.

### Environment info

datasets@main
fsspec==2023.10.0