noamaanMulla-03/datasets-issues huggingface dataset:
Support multiprocessing for download/extract nestedly

### Feature request

Current multiprocessing for download/extract is not done nestedly. For example, when processing SlimPajama, there is only 3 processes (for train/test/val), while there are many files inside these 3 folders

```
Downloading data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]

Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s][A
Extracting data files #2:   0%|          | 0/1 [00:00<?, ?obj/s][A[A
```

### Motivation

speedup dataset loading

### Your contribution

I can help test the feature

Labels: name: enhancement
Dataloader stuck on multiple GPUs

### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1


What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo.

Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.
`dataset.filter` ALWAYS removes the first item from the dataset when using batched=True

### Describe the bug

If you call batched=True when calling `filter`, the first item is _always_ filtered out, regardless of the filter condition.



### Steps to reproduce the bug

Here's a minimal example:
```python
def filter_batch_always_true(batch, indices):
    print("First index being passed into this filter function: ", indices[0])
    return indices  # Keep all indices

data = {"value": list(range(10))}
dataset = Dataset.from_dict(data)

filtered_dataset = dataset.filter(filter_batch_always_true, with_indices=True, batched=True)
print("Length of original dataset: ", len(dataset))
print("Length of filtered_dataset: ", len(filtered_dataset))
print("Is equal to original? ", len(filtered_dataset) == len(dataset))
print("First item of filtered dataset: ", filtered_dataset[0])
print("Last item of filtered dataset: ", filtered_dataset[-1])
```
prints:
```
First index being passed into this filter function:  0
Length of original dataset:  10
Length of filtered_dataset:  9
Is equal to original?  False
First item of filtered dataset:  {'value': 1}
Last item of filtered dataset:  {'value': 9}
```

### Expected behavior

Filter should respect the filter condition.

### Environment info


- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- PyArrow version: 10.0.1
- Pandas version: 2.0.2



`filter` treats the function's output as a (selection) mask - `True` keeps the sample, and `False` drops it. In your case, `bool(0)` evaluates to `False`, so dropping the first sample is the correct behavior.

Oh gosh! 🤦  I totally misunderstood the API! My apologies!
Support buffer shuffle for to_tf_dataset

### Feature request

I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model. 
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset. 
tf.data.Dataset support buffer shuffle by default. 
shuffle(
    buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)

### Motivation

I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset. 

### Your contribution

NA

Labels: name: enhancement

cc @Rocketknight1 

Hey! You can implement this yourself, just:

1) Create the dataset with `to_tf_dataset()` with `shuffle=False`
2) Add an `unbatch()` at the end (or use batch_size=1)
3) Add a `shuffle()` to the resulting dataset with your desired buffer size
4) Add a `batch()` at the end again to re-batch your dataset.

Note that the way we construct datasets in `to_tf_dataset()`, we don't actually shuffle the entire dataset in-memory, using `tf.data.Dataset.shuffle()`! Instead, we shuffle an index array and then load from the dataset with that. This means that shuffling with `tf.data.Dataset.shuffle()` will probably be slower and use more memory than our approach - I don't think adding the option for smaller shuffle buffers will actually save you memory on this!

Thanks for your reply! @Rocketknight1 
"We don't actually shuffle the entire dataset in-memory, using tf.data.Dataset.shuffle()! Instead, we shuffle an index array and then load from the dataset with that."
In such case, there will be random access to dataset data during shuffling. When the dataset is large, the performance can be X10 times slow. I have tried many ways with to_tf_dataset() trying to achieve comparable performance with tf.data.Dataset().shuffle(buffer_size).batch(). But the performance with to_tf_dataset() is still slow.
Don't use take on dataset table in pyarrow 1.0.x

Fix #615 


I tried lower batch sizes and it didn't accelerate filter (quite the opposite actually).
The slow-down also appears for pyarrow 0.17.1 for some reason, not sure it comes from these changes

I just checked the benchmarks of other PRs and some of them had 300s (!!) for filter. This needs some investigation..

Merging this one since it's not the cause of the the slow down

@lhoestq What might be the reason for the slowdown? When I was training large batchsize, the slowdown was obvious, and my original assumption was to increase write_batch_size
Data alteration when loading dataset with unspecified inner sequence length

### Describe the bug

When a dataset saved with a specified inner sequence length is loaded without specifying that length, the original data is altered and becomes inconsistent.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Value, Sequence, load_dataset

# Repository ID
repo_id = "my_repo_id"

# Define features with a specific length of 3 for each inner sequence
specified_features = Features({"key": Sequence(Sequence(Value("float32"), length=3))})

# Create a dataset with the specified features
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
dataset = Dataset.from_dict({"key": data}, features=specified_features)

# Push the dataset to the hub
dataset.push_to_hub(repo_id)

# Define features without specifying the length
unspecified_features = Features({"key": Sequence(Sequence(Value("float32")))})

# Load the dataset from the hub with this new feature definition
dataset = load_dataset(f"qgallouedec/{repo_id}", split="train", features=unspecified_features)

# The obtained data is altered
print(dataset.to_dict())  # {'key': [[[1.0], [2.0]], [[3.0], [4.0]]]}
```


### Expected behavior

```python
print(dataset.to_dict())  # {'key': [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]}
```

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


While this issue may seem specific, it led to a silent problem in my workflow that took days to diagnose. If this feature is not intended to be supported, an error should be raised when encountering this configuration to prevent such issues.

Thanks for reporting! This is a MRE:

```python
import pyarrow as pa
from datasets.table import cast_array_to_feature
from datasets import Sequence, Value
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
arr = pa.array(data, pa.list_(pa.list_(pa.float32(), 3)))
cast_array_to_feature(arr, Sequence(Sequence(Value("float32"))))
```

I've opened a PR with a fix.
Tokenization with multiple workers is too slow

I am trying to tokenize a few million documents with multiple workers but the tokenization process is taking forever. 

Code snippet:

```
raw_datasets.map(
            encode_function,
            batched=False,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            remove_columns=[name for name in raw_datasets["train"].column_names if name not in ["input_ids", "labels", "attention_mask"]],
            desc="Tokenizing data",
        )
```

Details:

```
transformers==4.28.0.dev0
datasets==4.28.0.dev0
preprocessing_num_workers==48
```
tokenizer == decapoda-research/llama-7b-hf





[This](https://huggingface.co/docs/datasets/nlp_process#map) is the most performant way to tokenize a dataset (`batched=True, num_proc=None, return_tensors="np"`) 

If`tokenizer.is_fast` returns `True`, `num_proc` must be `None/1` to benefit from the fast tokenizers' parallelism (the fast tokenizers are implemented in Rust, and Rust multi-threading doesn't work well with Python multi-processing)
[Feature Request] Check the length of batch before writing so that empty batch is allowed

### Use Case
I use `dataset.map(process_fn, batched=True)` to process the dataset, with data **augmentations or filtering**. However, when all examples within a batch is filtered out, i.e. **an empty batch is returned**, the following error will be thrown:
  ```
  ValueError: Schema and number of arrays unequal
  ```
This is because the empty batch does not comply with the schema of other batches. I think an empty batch should be allowed to facilitate coding (one does not need to assign an empty list manually for all keys.)

A simple fix is to check the length of `batch` before writing:
```
if len(batch):
  writer.write_batch(batch)
```
instead of 

https://github.com/huggingface/datasets/blob/74d60213dcbd7c99484c62ce1d3dfd90a1df0770/src/datasets/arrow_dataset.py#L3493



I think this error means you filter all examples within an (input) batch by deleting its columns. In that case, to avoid the error, you can set the column value to an empty list (`input_batch["col"] = []`) instead.
Offset overflow while doing regex on a text column

### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0


Hi! This looks like an Arrow bug, but it can be avoided by reducing the `writer_batch_size`.

(`ds = ds.map(get_text_caption, writer_batch_size=100)` in Colab runs without issues)


@mariosasko I ran into this problem with load_dataset. What should I do

@AisingioroHao0  You can also pass the `writer_batch_size` parameter to `load_dataset`, e.g., `load_dataset("mnist", writer_batch_size=100)`

@mariosasko How do I determine the optimal size of write_batch_size? My training is sometimes fast and sometimes slow. Is it because write_batch_size is too small? Each batch of the current dataloader should be the same size. I preprocessed the dataset using map

@aihao2000 It's unlikely `writer_batch_size` is the problem. You can use the following code to profile the training loop (e.g., on a subset of data) and find slow parts:
```python
import cProfile, pstats

with cProfile.Profile() as profiler:
    ... # training loop code

stats = pstats.Stats(profiler).sort_stats("cumtime")
stats.print_stats()
```


@nishanthcgit ok，thanks.Recently I found dataset.with_transform to be faster and more stable with multiple processes

@mariosasko Is the larger the num_proc of load_dataset within the number of cpu cores, the better? Then the num_proc of data_loader is the number of cpu cores/number of training processes
Create all YAML dataset_info

Following https://github.com/huggingface/datasets/pull/4926

Creates all the `dataset_info` YAML fields in the dataset cards
The JSON are also updated using the simplified backward compatible format added in https://github.com/huggingface/datasets/pull/4926

Needs https://github.com/huggingface/datasets/pull/4926 to be merged first

Labels: name: dataset contribution

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5018). All of your documentation changes will be reflected on that endpoint.

Closing since https://github.com/huggingface/datasets/pull/4974 removed all the datasets scripts.

Individual PRs must be opened on the Hugging face Hub to add the YAML metadata
Add COCO dataset

This PR adds the MS COCO dataset. Compared to the [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/object_detection/coco.py) script, this implementation adds 8 additional configs to cover the tasks other than object detection.

Some notes:
* the data exposed by TFDS is contained in the `2014`, `2015`, `2017` and `2017_panoptic_segmentation` configs here
* I've updated `encode_nested_example` for easier handling of missing values (cc @lhoestq @albertvillanova; will add tests if you are OK with the changes in `features.py`)
* this implementation should fix https://github.com/huggingface/datasets/pull/3377#issuecomment-985559427

TODOs:
- [x] dataset card
- [ ] dummy data

cc @merveenoyan 

Closes #2526 

Labels: name: dataset contribution

The CI failures other than a missing dummy data file and missing fields in the card are unrelated to this PR. 

Thanks a lot for this great work and fixing TFDS based script @mariosasko 🤗 will generate the dummy dataset and write the model card tomorrow!

@mariosasko I added the dataset card, I'm on the dummy data rn. 

@merveenoyan Let me know if you need any help with the dummy data.

I plan to split the current script/dataset into 4 smaller scripts/datasets to make sure they are properly indexed by Papers With Code later on. In this format:
* the `*_image_captioning` configs will form the [COCO Captions](https://paperswithcode.com/sota/image-captioning-on-coco-captions) dataset (also present in TFDS, but only the 2017 version)
* the `stuff_segmentation` config will form the [COCO Stuff](https://paperswithcode.com/dataset/coco-stuff) dataset
* the `desnepose` config will form the [DensePose-COCO](https://paperswithcode.com/dataset/densepose) dataset
* the rest will be [COCO](https://paperswithcode.com/dataset/coco) (+ will add the `minival` and the `valminusminival` splits to COCO 2014)

Also, if I find the time, I'll add preprocessing examples that rely on `pycocotools` to the README files.

@mariosasko I feel like we can just push main COCO and add Captions + Stuff later, WDYT?

_The documentation is not available anymore as the PR was closed or merged._

Thanks for your contribution, @mariosasko and @merveenoyan. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add evaluation metadata to wmt16

Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

Labels: name: dataset contribution

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4557). All of your documentation changes will be reflected on that endpoint.

> Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

yes :)

As discussed with @lewtun, we are closing this PR, because it requires first the task names to be aligned between AutoTrain and datasets.
Add Pascal VOC dataset

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._

Some CI fails are unrelated to your PR and fixed on master, feel free to merge master into your branch :)

Thanks @nateraw for the addition of this dataset.

I would suggest to transfer it to the Hugging Face Hub, under a "pascal" organization namespace: "pascal/voc".

What do you think?

FYI I think this dataset is also available at (internal) https://huggingface.co/datasets/HuggingFaceM4/pascal_voc

@lhoestq @albertvillanova what do you think best path forward is? No idea when I'll get to looking at this again, but would be nice to know plan so when I find time I can just get it done in one sitting. 

My (not strong) opinion on this:
- as we are removing dataset scripts from GitHub, this dataset should be created directly on the Hub
- I proposed doing it under some kind of "official" org namespace, like pascal or pascal2; other suggestions are welcome
- the link given by @lhoestq might serve as inspiration for your implementation (I think yours misses data about action classification): their implementation comprises tasks: classification/detection, segmentation, action classification, person layout; it misses other tasks though

What do you think?
Add ImageNet toy datasets from fastai

Adds the ImageNet toy datasets from FastAI: Imagenette, Imagewoof and Imagewang.

TODOs:
* [ ] add dummy data
* [ ] add dataset card
* [ ] generate `dataset_info.json`

Labels: name: dataset contribution

Thanks for your contribution, @mariosasko. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Multi Domain Sentiment Analysis Dataset (MDSA)


Labels: name: dataset contribution

can you also run `make style` to format the code ?

I'll come back to this one in sometime :) @lhoestq 

Also if you would use `xml.etree.ElementTree` to parse the XML it would be awesome, because right now you're using an external dependency `xmltodict `

> Also if you would use xml.etree.ElementTree to parse the XML it would be awesome, because right now you're using an external dependency xmltodict

Its pseudo xml so elementtree fails. xmltodict seems to be working quite good for this. do we have examples of pseudo xml datasets?

for the other pseudo xml the text is parsed manually

Can you add `xmltodict` to the test dependencies in setup.py please to fix the CI please ?

Also can you add the dataset card with the tags and run `make style` ?

Hi :) have you had a chance to fix the test dependency and apply `make style` ?

Feel fee to ping me when it's ready for a review

Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
add reuters21578 dataset

This PR adds the `Reuters_21578` dataset https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html  
#353 

The datasets is a lit of `.sgm` files which are a bit different from xml file indeed `xml.etree` couldn't be used to read files. I consider them as text file (to avoid using external library)  and read line by line (maybe there is a better way to do, happy to get your opinion on it)

In the Readme file 3 ways to split the dataset are given.:

- The Modified Lewis ("ModLewis") Split: train, test and unused-set

- The Modified Apte ("ModApte") Split : train, test and unused-set

- The Modified Hayes ("ModHayes") Split: train and test

Here I consider the last one as the readme file highlight that this split provides the ability to compare results with those of the 2 first splits.



> Awesome !
> Good job on parsing these files :O
> 
> Do you think it would be hard to get the two other split configurations ?

It shouldn't be that hard, I think I can consider different config names for each split 

> > Awesome !
> > Good job on parsing these files :O
> > Do you think it would be hard to get the two other split configurations ?
> 
> It shouldn't be that hard, I think I can consider different config names for each split

Yes that would be perfect

closing this PR and opening a new one to fix the circle CI problems
Column indexing hotfix

As observed for example in #469 , currently `__getitem__` does not convert the data to the dataset format when indexing by column. This is a hotfix that imitates functional 0.3.0. code. In the future it'd probably be nice to have a test there.


Looks good to me as well but we'll want to add a test indeed.
You can add one if you have time @TevenLeScao.
Otherwise, we'll do it when we are back with Quentin. 

I fixed it in #494
[Reproductibility] Allow to pin versions of datasets/metrics

Repurpose the `version` attribute in datasets and metrics to let the user pin a specific version of datasets and metric scripts:
```
dataset = nlp.load_dataset('squad', version='1.0.0')
metric = nlp.load_metric('squad', version='1.0.0')
```

Notes:
- version number are the release version of the library
- currently only possible for canonical datasets/metrics, ie. integrated in the GitHub repo of the library


Closing this one in favor of #584
Add MATINF dataset

@lhoestq The command to create metadata failed. I guess it's because the zip is not downloaded from a remote address? How to solve that? Also the CI fails and I don't know how to fix that :(


Hi ! sorry for the late response

Could you try to rebase from master ? We changed the named of the library last week so you have to include this change in your code.

Can you give me more details about the error you get when running the cli command ?

Note that in case of a manual download you have to specify the directory where you downloaded the data with `--data_dir <path/to/the/directory>`

I fucked up the Git rebase lol. Closing it.