jamesLeeeeeee/datasets-github-issues huggingface dataset:
IterableDataset and Dataset return different batch sizes when using Trainer with multiple GPUs
Author: kheyer | Pull request: no | State: closed

### Describe the bug

I am training a Roberta model using 2 GPUs and the `Trainer` API with a batch size of 256.

Initially I used a standard `Dataset`, but had issues with slow data loading. After reading [this issue](https://github.com/huggingface/datasets/issues/2252), I swapped to loading my dataset as contiguous shards and passing those to an `IterableDataset`. I observed an unexpected drop in GPU memory utilization, and found the batch size returned from the model had been cut in half.

When using `Trainer` with 2 GPUs and a batch size of 256, `Dataset` returns a batch of size 512 (256 per GPU), while `IterableDataset` returns a batch size of 256 (256 total). My guess is `IterableDataset` isn't accounting for multiple cards.


### Steps to reproduce the bug

```python
import datasets
from datasets import IterableDataset

from transformers import RobertaConfig
from transformers import RobertaTokenizerFast
from transformers import RobertaForMaskedLM

from transformers import DataCollatorForLanguageModeling
from transformers import Trainer, TrainingArguments

use_iterable_dataset = True
def gen_from_shards(shards):
    for shard in shards:
        for example in shard:
            yield example

dataset = datasets.load_from_disk('my_dataset.hf')

if use_iterable_dataset:
    n_shards = 100
    shards = [dataset.shard(num_shards=n_shards, index=i) for i in range(n_shards)]
    dataset = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards})

tokenizer = RobertaTokenizerFast.from_pretrained("./my_tokenizer", max_len=160, use_fast=True)

config = RobertaConfig(
    vocab_size=8248,
    max_position_embeddings=256,
    num_attention_heads=8,
    num_hidden_layers=6,
    type_vocab_size=1)

model = RobertaForMaskedLM(config=config)

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

training_args = TrainingArguments(
    per_device_train_batch_size=256
    # other args removed for brevity 
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()
```

### Expected behavior

Expected `Dataset` and `IterableDataset` to have the same batch size behavior. If the current behavior is intentional, the batch size printout at the start of training should be updated. Currently, both dataset classes result in `Trainer` printing the same total batch size, even though the batch size sent to the GPUs are different.

### Environment info

datasets 2.7.1
transformers 4.25.1
PyTorch BatchSampler still loads from Dataset one-by-one
Author: davidgilbertson | Pull request: no | State: closed

### Describe the bug

In [the docs here](https://huggingface.co/docs/datasets/use_with_pytorch#use-a-batchsampler), it mentions the issue of the Dataset being read one-by-one, then states that using a BatchSampler resolves the issue.

I'm not sure if this is a mistake in the docs or the code, but it seems that the only way for a Dataset to be passed a list of indexes  by PyTorch (instead of one index at a time) is to define a `__getitems__` method (note the plural) on the Dataset object, and since the HF Dataset doesn't have this, PyTorch executes [this line of code](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/_utils/fetch.py#L58), reverting to fetching one-by-one.

### Steps to reproduce the bug

You can put a breakpoint in `Dataset.__getitem__()` or just print the args from there and see that it's called multiple times for a single `next(iter(dataloader))`, even when using the code from the docs:
```py
from torch.utils.data.sampler import BatchSampler, RandomSampler
batch_sampler = BatchSampler(RandomSampler(ds), batch_size=32, drop_last=False)
dataloader = DataLoader(ds, batch_sampler=batch_sampler)
```



### Expected behavior

The expected behaviour would be for it to fetch batches from the dataset, rather than one-by-one.

To demonstrate that there is room for improvement: once I have a HF dataset `ds`, if I just add this line:
```py
ds.__getitems__ = ds.__getitem__
```
...then the time taken to loop over the dataset improves considerably (for wikitext-103, from one minute to 13 seconds with batch size 32). Probably not a big deal in the grand scheme of things, but seems like an easy win.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
Increase chunk size for speeding up file downloads
Author: Narsil | Pull request: yes | State: open

Original fix: https://github.com/huggingface/huggingface_hub/pull/1267
Not sure this function is actually still called though.

I haven't done benches on this. Is there a dataset where files are hosted on the hub through cloudfront so we can have the same setup as in `hf_hub` ?
`load_dataset` has ~4 seconds of overhead for cached data
Author: davidgilbertson | Pull request: no | State: open

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.
TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset
Author: vmuel | Pull request: no | State: closed

### Describe the bug

Hi, 

Thanks for the amazing work on the library!

**Describe the bug**

I think I might have noticed a small bug in the filter method. 
Having loaded a dataset using `load_dataset`,  when I try to filter out empty entries with `batched=True`, I get a TypeError.

### Steps to reproduce the bug

```
train_dataset = train_dataset.filter(
            function=lambda example: example["image"] is not None, 
            batched=True,
            batch_size=10)
```
Error message:

```
File .../lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
...
-> 5666 indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
   5667 if indices_mapping is not None:
   5668     indices_array = pa.array(indices_array, type=pa.uint64())

TypeError: 'bool' object is not iterable
```

**Removing batched=True allows to bypass the issue.** 

### Expected behavior

According to the doc, "[batch_size corresponds to the] number of examples per batch provided to function if batched = True", so we shouldn't need to remove the batchd=True arg?

source: https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.Dataset.filter

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
Add a `reduce` method
Author: zhangir-azerbayev | Pull request: no | State: closed

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend.
Update audio installation doc page
Author: polinaeterna | Pull request: no | State: closed

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327 
So we should update the doc page. But first investigate [this issue](5488).
Push_to_hub in a pull request
Author: lhoestq | Pull request: no | State: closed

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw 

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR
Do not add index column by default when exporting to CSV
Author: albertvillanova | Pull request: yes | State: closed

As pointed out by @merveenoyan, default behavior of `Dataset.to_csv` adds the index as an additional column without name.

This PR changes the default behavior, so that now the index column is not written.

To add the index column, now you need to pass `index=True` and also `index_label=<name of the index colum>` to name that column.

CC: @merveenoyan
Pin dill lower version
Author: albertvillanova | Pull request: yes | State: closed

Pin `dill` lower version compatible with `datasets`.

Related to:
- #5487
- #288

Note that the required `dill._dill` module was introduced in dill-2.8.0, however we have heuristically tested that datasets can only be installed with dill>=3.0.0 (otherwise pip hangs indefinitely while preparing metadata for multiprocess-0.70.7)
Error loading MP3 files from CommonVoice
Author: kradonneoh | Pull request: no | State: closed

### Describe the bug

When loading a CommonVoice dataset with `datasets==2.9.0` and `torchaudio>=0.12.0`, I get an error reading the audio arrays: 

```python
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/datasets/features/audio.py in _decode_mp3(self, path_or_file)
    310             try:  # try torchaudio anyway because sometimes it works (depending on the os and os packages installed)
--> 311                 array, sampling_rate = self._decode_mp3_torchaudio(path_or_file)
    312             except RuntimeError:

~/.local/lib/python3.8/site-packages/datasets/features/audio.py in _decode_mp3_torchaudio(self, path_or_file)
    351 
--> 352         array, sampling_rate = torchaudio.load(path_or_file, format="mp3")
    353         if self.sampling_rate and self.sampling_rate != sampling_rate:

~/.local/lib/python3.8/site-packages/torchaudio/backend/soundfile_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    204     """
--> 205     with soundfile.SoundFile(filepath, "r") as file_:
    206         if file_.format != "WAV" or normalize:

~/.local/lib/python3.8/site-packages/soundfile.py in __init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    654                                          format, subtype, endian)
--> 655         self._file = self._open(file, mode_int, closefd)
    656         if set(mode).issuperset('r+') and self.seekable():

~/.local/lib/python3.8/site-packages/soundfile.py in _open(self, file, mode_int, closefd)
   1212             err = _snd.sf_error(file_ptr)
-> 1213             raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1214         if mode_int == _snd.SFM_WRITE:

LibsndfileError: Error opening <_io.BytesIO object at 0x7fa539462090>: File contains data in an unknown format.

```

I assume this is because there's some issue with the mp3 decoding process. I've verified that I have `ffmpeg>=4` (on a Linux distro), which appears to be the fallback backend for `torchaudio,` (at least according to #4889).

### Steps to reproduce the bug

```python
dataset = load_dataset("mozilla-foundation/common_voice_11_0", "be", split="train")
dataset[0]
```

### Expected behavior

Similar behavior to `torchaudio<0.12.0`, which doesn't result in a `LibsndfileError`

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.1
Incorrect filepath for dill module
Author: avivbrokman | Pull request: no | State: closed

### Describe the bug

I installed the `datasets` package and when I try to `import` it, I get the following error:

```
Traceback (most recent call last):

  File "/var/folders/jt/zw5g74ln6tqfdzsl8tx378j00000gn/T/ipykernel_3805/3458380017.py", line 1, in <module>
    import datasets

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 66, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import Features, Image, Value

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/__init__.py", line 17, in <module>
    from .audio import Audio

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/features/audio.py", line 12, in <module>
    from ..download.streaming_download_manager import xopen

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/download/download_manager.py", line 36, in <module>
    from ..utils.py_utils import NestedDataStructure, map_nested, size_str

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 602, in <module>
    class Pickler(dill.Pickler):

  File "/Users/avivbrokman/opt/anaconda3/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 605, in Pickler
    dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())

AttributeError: module 'dill' has no attribute '_dill'

```

Looking at the github source code for dill, it appears that `datasets` has a bug or is not compatible with the latest `dill`.  Specifically, rather than `dill._dill.XXXX` it should be `dill.dill._dill.XXXX`.  But given the popularity of `datasets` I feel confused about me being the first person to have this issue, so it makes me wonder if I'm misdiagnosing the issue.

### Steps to reproduce the bug

Install `dill` and `datasets` packages and then `import datasets`

### Expected behavior

I expect `datasets` to import.

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 11.0.0
- Pandas version: 1.4.4
Adding `sep` to TextConfig 
Author: omar-araboghli | Pull request: no | State: open

I have a local a `.txt` file that follows the `CONLL2003` format which I need to load using `load_script`. However, by using `sample_by='line'`, one can only split the dataset into lines without splitting each line into columns. Would it be reasonable to add a `sep` argument in combination with `sample_by='paragraph'` to parse a paragraph into an array for each column ? If so, I am happy to contribute!

## Environment
* `python 3.8.10`
* `datasets 2.9.0`

## Snippet of `train.txt`

```txt
Distribution	NN	O	O
and	NN	O	O
dynamics	NN	O	O
of	NN	O	O
electron	NN	O	B-RP
complexes	NN	O	I-RP
in	NN	O	O
cyanobacterial	NN	O	B-R
membranes	NN	O	I-R

The	NN	O	O
occurrence	NN	O	O
of	NN	O	O
prostaglandin	NN	O	B-R
F2α	NN	O	I-R
in	NN	O	O
Pharbitis	NN	O	B-R
seedlings	NN	O	I-R
grown	NN	O	O
under	NN	O	O
short	NN	O	B-P
days	NN	O	I-P
or	NN	O	I-P
days	NN	O	I-P

```

## Current Behaviour 
```python
# defining 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'] here would fail with `ValueError: Length of names (4) does not match length of arrays (1)`
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='line')
dataset['train']['tokens'][0]
>>> 'Distribution\tNN\tO\tO'
```

## Expected Behaviour / Suggestion
```python
# suppose we defined 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='paragraph', sep='\t')
dataset['train']['tokens'][0]
>>> ['Distribution', 'and', 'dynamics', ... ]

dataset['train']['ner_tags'][0]
>>> ['O', 'O', 'O', ... ]
```
Add section in tutorial for IterableDataset
Author: stevhliu | Pull request: yes | State: closed

Introduces an `IterableDataset` and how to access it in the tutorial section. It also adds a brief next step section at the end to provide a path for users who want more explanation and a path for users who want something more practical and learn how to preprocess these dataset types. It'll complement the awesome new doc introduced in:

- #5410
Update docs for `nyu_depth_v2` dataset
Author: awsaf49 | Pull request: yes | State: closed

This PR will fix the issue mentioned in #5461. Here is brief overview,

## Bug:

Discrepancy between depth map of `nyu_depth_v2` dataset [here](https://huggingface.co/docs/datasets/main/en/depth_estimation) and actual depth map. Depth values somehow got **discretized/clipped** resulting in depth maps that are different from actual ones. Here is a side-by-side comparison,

![image](https://user-images.githubusercontent.com/36858976/214381162-1d9582c2-6750-4114-a01a-61ca1cd5f872.png)


## Fix:

When I first loaded the datasets from HF I noticed it was 30GB but in DenseDepth data is only 4GB with dtype=uint8. This means data from fast-depth (before loading to HF) must have high precision. So when I tried to dig deeper by directly loading depth_map from `h5py`, I found depth_map from `h5py`  came with `float32`. But when the data is processed in HF with `datasets.Image()` it was directly converted to `uint8` from `float32`  hence the **discretized** depth map.
https://github.com/huggingface/datasets/blob/c78559cacbb0ca6e0bc8bfc313cc0359f8c23ead/src/datasets/features/image.py#L91-L93


cc: @sayakpaul @lhoestq
Reload features from Parquet metadata
Author: lhoestq | Pull request: no | State: closed

The idea would be to allow this :
```python 
ds.to_parquet("my_dataset/ds.parquet")
reloaded = load_dataset("my_dataset")
assert ds.features == reloaded.features 
```
And it should also work with Image and Audio types (right now they're reloaded as a dict type)

This can be implemented by storing and reading the feature types in the parquet metadata, as we do for arrow files.
Load a cached dataset as iterable
Author: lhoestq | Pull request: no | State: open

The idea would be to allow something like 

```python 
ds = load_dataset("c4", "en", as_iterable=True)
```

To be used to train models. It would load an IterableDataset from the cached Arrow files.

Cc @stas00 

Edit : from the discussions we may load from cache when streaming=True
Dataset scan time is much slower than using native arrow
Author: jonny-cyberhaven | Pull request: no | State: closed

### Describe the bug

I'm basically running the same scanning experiment from the tutorials https://huggingface.co/course/chapter5/4?fw=pt except now I'm comparing to a native pyarrow version.

I'm finding that the native pyarrow approach is much faster (2 orders of magnitude). Is there something I'm missing that explains this phenomenon?

### Steps to reproduce the bug

https://colab.research.google.com/drive/11EtHDaGAf1DKCpvYnAPJUW-LFfAcDzHY?usp=sharing

### Expected behavior

I expect scan times to be on par with using pyarrow directly.

### Environment info

standard colab environment
Column project operation on `datasets.Dataset`
Author: daskol | Pull request: no | State: closed

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.
Add num_test_batches option
Author: amyeroberts | Pull request: yes | State: closed

`to_tf_dataset` calls can be very costly because of the number of test batches drawn during `_get_output_signature`. The test batches are draw in order to estimate the shapes when creating the tensorflow dataset. This is necessary when the shapes can be irregular, but not in cases when the tensor shapes are the same across all samples. This PR adds an option to change the number of batches drawn, so the user can speed this conversion up. 

Running the following, and modifying `num_test_batches`
```
import time
from datasets import load_dataset
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator()
dataset = load_dataset("beans")
dataset = dataset["train"].with_format("np")
start = time.time()
dataset = dataset.to_tf_dataset(
    columns=["image"],
    label_cols=["label"],
    batch_size=8,
    collate_fn=data_collator,
    num_test_batches=NUM_TEST_BATCHES,
)
end = time.time()
print(end - start)
```

NUM_TEST_BATCHES=200:  0.8197s
NUM_TEST_BATCHES=50:    0.3070s
NUM_TEST_BATCHES=2:      0.1417s
NUM_TEST_BATCHES=1:      0.1352s