kardwalker/my_dataset huggingface dataset:
GitHub issue #5338: `map()` stops every 1000 steps
State: closed | Created: 2022-12-07T19:09:40Z

### Describe the bug

I am passing the following `prepare_dataset` function to `Dataset.map` (code is inspired from [here](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py#L454))
```python3
def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array 
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]

    # encode target text to label ids 
    batch["labels"] = tokenizer(batch[text_column]).input_ids
    return batch
...
train_ds = train_ds.map(prepare_dataset)
```

Here is the exact code I am running https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets/blob/main/train.py#L70-L71

It starts using all the cores (I am not sure why because I did not pass `num_proc`)
then progress bar stops at every 1k steps. (starts using a single core)
then come back to using all the cores again.

link to [screen record](https://youtu.be/jPQpQQGp6Gc)

Can someone explain this process and maybe provide a way to improve this pipeline? cc: @lhoestq 

### Steps to reproduce the bug

1. load the dataset
2. create a Whisper processor
3. create a `prepare_dataset` function
4. pass the function to `dataset.map(prepare_dataset)`

### Expected behavior

- Use a single core per a function
- not to stop at some point?

### Environment info

- `datasets` version: 2.7.1.dev0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue #5337: Support webdataset format
State: closed | Created: 2022-12-07T11:32:25Z

Webdataset is an efficient format for iterable datasets. It would be nice to support it in `datasets`, as discussed in https://github.com/rom1504/img2dataset/issues/234.

In particular it would be awesome to be able to load one using `load_dataset` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.

It terms of implementation, we can have something similar to the Parquet loader.
I also think it's fine to have webdataset as an optional dependency.
GitHub issue #5336: Set `IterableDataset.map` param `batch_size` typing as optional
State: closed | Created: 2022-12-06T17:08:10Z

This PR solves #5325 

~Indeed we're using the typing for optional values as `Union[type, None]` as it's similar to how Python 3.10 handles optional values as `type | None`, instead of using `Optional[type]`.~

~Do we want to start using `Union[type, None]` for type-hinting optional values or just keep on using `Optional`?~ -> Keeping `Optional` still for consistency with the rest of the code in `datasets`

Also we now allow `batch_size` to be `None` for `IterableDataset.map` and `IterableDataset.filter`e.g. `MappedExamplesIterable` as `map` is internally instantiating those and propagating the `batch_size` param so if it can be `None` for `map` it should also do so for `MappedExamplesIterable`, as well as for `FilteredExamplesIterable` when calling `IterableDataset.filter`.

## TODOs

- [x] Add integration tests
- [x] Handle scenario where `batched=True` and `batch_size=None` or `batch_size<=0`
GitHub issue #5334: Clean up docstrings
State: closed | Created: 2022-12-05T20:56:08Z

As raised by @polinaeterna in #5324, some of the docstrings are a bit of a mess because it has both Markdown and Sphinx syntax. This PR fixes the docstring for `DatasetBuilder`. 

I'll start working on cleaning up the rest of the docstrings and removing the old Sphinx syntax (let me know if you prefer one big PR with all the cleaned changes or multiple smaller ones)! 🧼
GitHub issue #5332: Passing numpy array to ClassLabel names causes ValueError
State: closed | Created: 2022-12-05T12:59:03Z

### Describe the bug

If a numpy array is passed to the names argument of ClassLabel, creating a dataset with those features causes an error.

### Steps to reproduce the bug

https://colab.research.google.com/drive/1cV_es1PWZiEuus17n-2C-w0KEoEZ68IX

TLDR:

If I define my classes as:
```
my_classes = np.array(['one', 'two', 'three'])
```

Then this errors:

```py
features = Features({'value': Value('string'), 'label': ClassLabel(names=my_classes)})
dataset = Dataset.from_list(my_data, features=features)
```
```
ValueError                                Traceback (most recent call last)
[<ipython-input-8-a8a9d53ec82f>](https://localhost:8080/#) in <module>
----> 1 dataset = Dataset.from_list(my_data, features=features)

11 frames
[/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in _asdict_inner(obj)
    183             for f in fields(obj):
    184                 value = _asdict_inner(getattr(obj, f.name))
--> 185                 if not f.init or value != f.default or f.metadata.get("include_in_asdict_even_if_is_default", False):
    186                     result[f.name] = value
    187             return result

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

But this works:
```
features2 = Features({'value': Value('string'), 'label': ClassLabel(names=list(my_classes))})
dataset2 = Dataset.from_list(my_data, features=features2)
```


### Expected behavior

If I provide a numpy array of class names, I would expect either an error that the names list is the wrong type, or for it to be cast internally.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.10
- Python version: 3.8.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Additionally:
- Numpy version: 1.23.5
GitHub issue #5329: Clarify imagefolder is for small datasets
State: closed | Created: 2022-12-01T21:47:29Z

Based on feedback from [here](https://github.com/huggingface/datasets/issues/5317#issuecomment-1334108824), this PR adds a note to the `imagefolder` loading and creating docs that `imagefolder` is designed for small scale image datasets.
GitHub issue #5325: map(...batch_size=None) for IterableDataset
State: closed | Created: 2022-12-01T15:43:42Z

### Feature request

Dataset.map(...) allows batch_size to be None. It would be nice if IterableDataset did too.

### Motivation

Although it may seem a bit of a spurious request given that `IterableDataset` is meant for larger than memory datasets, but there are a couple of reasons why this might be nice.

One is that load_dataset(...) can return either IterableDataset or Dataset. mypy will then complain if batch_size=None even if we know it is Dataset. Of course we can do:

    assert isinstance(d, datasets.DatasetDict)

But it is a mild inconvenience. What's more annoying is that whenever we use something like e.g. `combine_datasets(...)`, we end up with the union again, and so have to do the assert again.

Another is that we could actually end up with an IterableDataset small enough for memory in normal/correct usage, e.g. by filtering a massive IterableDataset.

For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

### Your contribution

Not this time.
GitHub issue #5324: Fix docstrings and types in documentation that appears on the website
State: open | Created: 2022-12-01T15:34:53Z

While I was working on https://github.com/huggingface/datasets/pull/5313 I've noticed that we have a mess in how we annotate types and format args and return values in the code. And some of it is displayed in the [Reference section](https://huggingface.co/docs/datasets/package_reference/builder_classes) of the documentation on the website.
 
Would be nice someday, maybe before releasing datasets 3.0.0, to unify it......
GitHub issue #5322: Raise error for `.tar` archives in the same way as for `.tar.gz` and `.tgz` in `_get_extraction_protocol`
State: closed | Created: 2022-12-01T15:19:28Z

Currently `download_and_extract` doesn't throw an error when it is used with files with `.tar` extension in streaming mode because `_get_extraction_protocol` doesn't do it (like it does for  `tar.gz` and `tgz`). `_get_extraction_protocol` returns formatted url as if we support tar protocol but we don't.
That means that in dataset scripts `.tar` files would be attempted to load and fail during examples generation (after `download_and_extract` execution). So this PR raises error for `tar` files too.
GitHub issue #5320: [Extract] Place the lock file next to the destination directory
State: closed | Created: 2022-12-01T13:55:49Z

Previously it was placed next to the archive to extract, but the archive can be in a read-only directory as noticed in https://github.com/huggingface/datasets/issues/5295

Therefore I moved the lock location to be next to the destination directory, which is required to have write permissions
GitHub issue #5318: Origin/fix missing features error
State: closed | Created: 2022-12-01T06:18:39Z

This fixes the problem of when the dataset_load function reads a function with "features" provided but some read batches don't have columns that later show up. For instance, the provided "features" requires columns A,B,C but only columns B,C show. This fixes this by adding the column A with nulls.
GitHub issue #5317: `ImageFolder` performs poorly with large datasets
State: open | Created: 2022-12-01T00:04:21Z

### Describe the bug

While testing image dataset creation, I'm seeing significant performance bottlenecks with imagefolders when scanning a directory structure with large number of images.


## Setup
* Nested directories (5 levels deep)
* 3M+ images
* 1 `metadata.jsonl` file


## Performance Degradation Point 1

Degradation occurs because [`get_data_files_patterns`](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L231-L243) runs the exact same scan for many different types of patterns, and there doesn't seem to be a way to easily limit this. It's controlled by the definition of [`ALL_DEFAULT_PATTERNS`](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L82-L85). 

One scan with 3M+ files takes about 10-15 minutes to complete on my setup, so having those extra scans really slows things down – from 10 minutes to 60+. Most of the scans return no matches, but they still take a significant amount of time to complete – hence the poor performance.

As a side effect, when this scan is run on 3M+ image files, Python also consumes up to 12 GB of RAM, which is not ideal.


## Performance Degradation Point 2

The second performance bottleneck is in [`PackagedDatasetModuleFactory.get_module`](https://github.com/huggingface/datasets/blob/d7dfbc83d68e87ba002c5eb2555f7a932e59038a/src/datasets/load.py#L707-L711), which calls `DataFilesDict.from_local_or_remote`. 

It runs for a long time (60min+), consuming significant amounts of RAM – even more than the point 1 above. Based on `iostat -d 2`, it performs **zero** disk operations, which to me suggests that there is a code based bottleneck there that could be sorted out.

### Steps to reproduce the bug

```python
from datasets import load_dataset
import os
import huggingface_hub

dataset = load_dataset(
  'imagefolder',
  data_dir='/some/path',
  # just to spell it out:
  split=None,
  drop_labels=True,
  keep_in_memory=False
)

dataset.push_to_hub('account/dataset', private=True)
```

### Expected behavior

While it's certainly possible to write a custom loader to replace `ImageFolder` with, it'd be great if the off-the-shelf `ImageFolder` would by default have a setup that can scale to large datasets.

Or perhaps there could be a dedicated loader just for large datasets that trades off flexibility for performance? As in, maybe you have to define explicitly how you want it to work rather than it trying to guess your data structure like `_get_data_files_patterns()` does?

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-4.14.296-222.539.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 10.0.1
- Pandas version: 1.3.5
GitHub issue #5316: Bug in sample_by="paragraph"
State: closed | Created: 2022-11-30T19:24:13Z

### Describe the bug

I think [this line](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/text/text.py#L96) is wrong and should be `batch = f.read(self.config.chunksize)`. Otherwise it will never terminate because even when `f` is finished reading, `batch` will still be truthy from the last iteration.

### Steps to reproduce the bug

```
> cat test.txt
a b c

d e f
````

```python
>>> import datasets
>>> datasets.load_dataset("text", data_files={"train":"test.txt"}, sample_by="paragraph")
```

This will go on forever.

### Expected behavior

Terminates very quickly. 

### Environment info

`version = "2.6.1"` but I think the bug is still there on main.
GitHub issue #5314: Datasets: classification_report() got an unexpected keyword argument 'suffix'
State: closed | Created: 2022-11-30T14:01:03Z

https://github.com/huggingface/datasets/blob/main/metrics/seqeval/seqeval.py


> import datasets
predictions = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = datasets.load_metric("seqeval")
results = seqeval.compute(predictions=predictions, references=references)
print(list(results.keys()))
print(results["overall_f1"])
print(results["PER"]["f1"])

It raises the error:

> TypeError: classification_report() got an unexpected keyword argument 'suffix'

For context, versions on my pip list -v

> datasets                 1.12.1  
seqeval                  1.2.2
GitHub issue #5313: Fix description of streaming in the docs
State: closed | Created: 2022-11-29T18:00:28Z

We say that "the data is being downloaded progressively" which is not true, it's just streamed, so I fixed it. Probably I missed some other places where it is written? 

Also changed docstrings for `StreamingDownloadManager`'s `download` and `extract` to reflect the same, as these docstrings are displayed in the documentation cc @lhoestq
GitHub issue #5312: Add DatasetDict.to_pandas
State: closed | Created: 2022-11-29T16:30:02Z

From discussions in https://github.com/huggingface/datasets/issues/5189, for tabular data it doesn't really make sense to have to do
```python
df = load_dataset(...)["train"].to_pandas()
```
because many datasets are not split.

In this PR I added `to_pandas` to `DatasetDict` which returns the DataFrame:

If there's only one split, you don't need to specify the split name:
```python
df = load_dataset(...).to_pandas()
```
EDIT: and if a dataset has multiple splits:
```python
df = load_dataset(...).to_pandas(splits=["train", "test"])
# or
df = load_dataset(...).to_pandas(splits="all")

# raises an error because you need to select the split(s) to convert
load_dataset(...).to_pandas()
```

I do have one question though @merveenoyan @adrinjalali @mariosasko:

Should we raise an error if there are multiple splits and ask the user to choose one explicitly ?
GitHub issue #5311: Add `features` param to `IterableDataset.map`
State: closed | Created: 2022-11-29T11:08:34Z

## Description

As suggested by @lhoestq in #3888, we should be adding the param `features` to `IterableDataset.map` so that the features can be preserved (not turned into `None` as that's the default behavior) whenever the user passes those as param, so as to be consistent with `Dataset.map`, as it provides the `features` param so that those are not inferred by default, but specified by the user, and later validated by `ArrowWriter`.

This is internally handled already by the functions relying on `IterableDataset.map` such as `rename_column`, `rename_columns`, and `remove_columns` as described in #5287.

## Usage Example

```python
from datasets import load_dataset, Features

ds = load_dataset("rotten_tomatoes", split="validation", streaming=True)
print(ds.info.features)
ds = ds.map(
    lambda x: {"target": x["label"]},
    features=Features(
        {"target": ds.info.features["label"], "label": ds.info.features["label"], "text": ds.info.features["text"]}
    ),
)
print(ds.info.features)
```
GitHub issue #5310: Support xPath for Windows pathnames
State: closed | Created: 2022-11-29T09:20:47Z

This PR implements a string representation of `xPath`, which is valid for local paths (also windows) and remote URLs.

Additionally, some `os.path` methods are fixed for remote URLs on Windows machines.

Now, on Windows machines:
```python
In [2]: str(xPath("C:\\dir\\file.txt"))
Out[2]: 'C:\\dir\\file.txt'
In [3]: str(xPath("http://domain.com/file.txt"))
Out[3]: 'http://domain.com/file.txt'
```
GitHub issue #5304: timit_asr doesn't load the test split.
State: closed | Created: 2022-11-26T10:18:22Z

### Describe the bug

When I use the function ```timit = load_dataset('timit_asr', data_dir=data_dir)```, it only loads train split, not test split.

I tried to change the directory and filename to lower case to upper case for the test split, but it does not work at all.

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 0
    })
})
```

The directory structure of both splits are same. (DIALECT_REGION / SPEAKER_CODE / DATA_FILES)

### Steps to reproduce the bug

1. just use ```timit = load_dataset('timit_asr', data_dir=data_dir)```

### Expected behavior

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 1680
    })
})
```

### Environment info

- ubuntu 20.04
- python 3.9.13
- datasets 2.7.1
GitHub issue #5303: Skip dataset verifications by default
State: closed | Created: 2022-11-25T18:39:09Z

Skip the dataset verifications (split and checksum verifications, duplicate keys check) by default unless a dataset is being tested (`datasets-cli test/run_beam`). The main goal is to avoid running the checksum check in the default case due to how expensive it can be for large datasets.

PS: Maybe we should deprecate `ignore_verifications`, which is `True` now by default, and give it a different name?
GitHub issue #5301: Return a split Dataset in load_dataset
State: closed | Created: 2022-11-25T16:35:54Z

...instead of a DatasetDict.

```python
# now supported
ds = load_dataset("squad")
ds[0]  
for example in ds:
    pass

# still works
ds["train"]
ds["validation"]

# new
ds.splits  # Dict[str, Dataset] | None

# soon to be supported (not in this PR)
ds = load_dataset("dataset_with_no_splits")
ds[0]
for example in ds:
    pass
```

I implemented `Dataset.__getitem__` and `IterableDataset.__getitem__` to be able to get a split from a dataset.
The splits are defined by the `ds.info.splits` dictionary.

Therefore a dataset is a table that optionally has some splits defined in the dataset info. And a split dataset is the concatenation of all its splits.

I made as little breaking changes as possible. Notable breaking changes:
- `load_dataset("potato").keys() / .items() / .values() /` don't work anymore, since we don't return a dict
- same for `for split_name in load_dataset("potato")`, since we now iterate on the examples
- ..

TODO:
- [x] Update push_to_hub
- [x] Update save_to_disk/load_from_disk
- [ ] check for other breaking changes
- [ ] fix existing tests
- [ ] add new tests
- [ ] docs

This is related to https://github.com/huggingface/datasets/issues/5189, to extend `load_dataset` to return datasets without splits
GitHub issue #5298: Bug in xopen with Windows pathnames
State: closed | Created: 2022-11-25T15:21:32Z

Currently, `xopen` function has a bug with local Windows pathnames:

From its implementation:
```python
def xopen(file: str, mode="r", *args, **kwargs):
    file = _as_posix(PurePath(file))
    main_hop, *rest_hops = file.split("::")
    if is_local_path(main_hop):
        return open(file, mode, *args, **kwargs)
```

On a Windows machine, if we pass the argument:
```python
xopen("C:\\Users\\USERNAME\\filename.txt")
```
it returns
```python
open("C:/Users/USERNAME/filename.txt")
```
GitHub issue #5296: Bug in xjoin with Windows pathnames
State: closed | Created: 2022-11-25T13:29:33Z

Currently, `xjoin` function has a bug with local Windows pathnames: instead of returning the OS-dependent join pathname, it always returns it in POSIX format.

```python
from datasets.download.streaming_download_manager import xjoin

path = xjoin("C:\\Users\\USERNAME", "filename.txt")
```

Join path should be:
```python
"C:\\Users\\USERNAME\\filename.txt"
```
However it is:
```python
"C:/Users/USERNAME/filename.txt"
```
GitHub issue #5292: Missing documentation build for versions 2.7.1 and 2.6.2 
State: closed | Created: 2022-11-24T09:42:10Z

After the patch releases [2.7.1](https://github.com/huggingface/datasets/releases/tag/2.7.1) and [2.6.2](https://github.com/huggingface/datasets/releases/tag/2.6.2), the online docs were not properly built (the build_documentation workflow was not triggered).

There was a fix by:
- #5291

However, both documentations were built from main branch, instead of their corresponding version branch.

We are rebuilding them.
GitHub issue #5291: [build doc] for v2.7.1 & v2.6.2
State: closed | Created: 2022-11-24T08:54:47Z

Do NOT merge. Using this PR to build docs for [v2.7.1](https://github.com/huggingface/datasets/pull/5291/commits/f4914af20700f611b9331a9e3ba34743bbeff934) & [v2.6.2](https://github.com/huggingface/datasets/pull/5291/commits/025f85300a0874eeb90a20393c62f25ac0accaa0)
GitHub issue #5289: Added support for JXL images.
State: open | Created: 2022-11-23T23:16:33Z

JPEG-XL is the most advanced of the next-generation of image codecs, supporting both lossless and lossy files — with better compression and quality than PNG and JPG respectively.  It has reduced the disk sizes and bandwidth required for many of the datasets I use.

Pillow does not yet support JXL, but there's a plugin as a separate Python library that does (`pip install jxlpy`), and I've tested that this change works as expected when the plugin is imported.

Dataset used for testing, you must `git pull` as loading it from Python won't work until `datasets-server` is also changed to support JXL files:
https://huggingface.co/datasets/texturedesign/td01_natural-ground-textures

The case where the plugin is not imported first raises an error:
```
PIL.UnidentifiedImageError: cannot identify image file 'td01/train/set01/01_145523.jxl'
```

In order to enable support for JXL even before pillow supports this, should this exception be handled with a better error message?  I'd expect/hope JXL support to follow in one of the pillow quarterly releases in the next 6-9 months.
GitHub issue #5288: Lossy json serialization - deserialization of dataset info 
State: open | Created: 2022-11-23T17:20:15Z

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead. 

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
    
test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}, 
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```


### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3
GitHub issue #5287: Fix methods using `IterableDataset.map` that lead to `features=None`
State: closed | Created: 2022-11-23T15:33:25Z

As currently `IterableDataset.map` is setting the `info.features` to `None` every time as we don't know the output of the dataset in advance, `IterableDataset` methods such as `rename_column`, `rename_columns`, and `remove_columns`. that internally use `map` lead to the features being `None`.

This PR is related to #3888, #5245, and #5284 

## ✅ Current solution

The code in this PR is basically making sure that if the features were there since the beginning and a `rename_column`/`rename_columns` happens, those are kept and the rename is applied to the `Features` too. Also, if the features were not there before applying `rename_column`, `rename_columns` or `remove_columns`, a batch is prefetched and the features are being inferred (that could potentially be part of `IterableDataset.__init__` in case the `info.features` value is `None`).

## 💡 Ideas

Some ideas were proposed in https://github.com/huggingface/datasets/issues/3888, but probably the most consistent solution even though it may take some time is to actually do the type inferencing during the `IterableDataset.__init__` in case the provided `info.features` is `None`, otherwise, we can just use the provided features.

Additionally, as mentioned at https://github.com/huggingface/datasets/issues/3888, we could also include a `features` parameter to the `map` function, but that's probably more tedious.

Also thanks to @lhoestq for sharing some ideas in both https://github.com/huggingface/datasets/issues/3888 and https://github.com/huggingface/datasets/issues/5245 :hugs:
GitHub issue #5284: Features of IterableDataset set to None by remove column
State: closed | Created: 2022-11-23T10:54:59Z

### Describe the bug

The `remove_column` method of the IterableDataset sets the dataset features to None.

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
print("Original features: ", dataset.features.keys())

# define features to remove: we KEEP audio and text
COLUMNS_TO_REMOVE = ['chapter_id', 'speaker_id', 'file', 'id']

dataset = dataset.remove_columns(COLUMNS_TO_REMOVE)

# check processed features, uh-oh!
print("Processed features: ", dataset.features)

# streaming the first audio sample still works
print("First sample:", next(iter(ds)))
```

**Print Output:**

```
Original features:  dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
Processed features:  None
First sample: {'audio': {'path': '2277-149896-0000.flac', 'array': array([ 0.00186157,  0.0005188 ,  0.00024414, ..., -0.00097656,
       -0.00109863, -0.00146484]), 'sampling_rate': 16000}, 'text': "HE WAS IN A FEVERED STATE OF MIND OWING TO THE BLIGHT HIS WIFE'S ACTION THREATENED TO CAST UPON HIS ENTIRE FUTURE"}
```

### Expected behavior

The features should be those **not** removed by the `remove_column` method, i.e. audio and text.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

(Running on Google Colab for a blog post: https://colab.research.google.com/drive/1ySCQREPZEl4msLfxb79pYYOWjUZhkr9y#scrollTo=8pRDGiVmH2ml)

cc @polinaeterna @lhoestq
GitHub issue #5281: Support cloud storage in load_dataset
State: open | Created: 2022-11-22T14:00:10Z

Would be nice to be able to do

```python
data_files=["s3://..."]  # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
```

The idea would be to use `fsspec` as in `download_and_prepare` and `save_to_disk`.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

https://github.com/huggingface/datasets/issues/3490
https://github.com/huggingface/datasets/issues/5244
[forum](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/2)
GitHub issue #5280: Import error
State: closed | Created: 2022-11-22T12:56:43Z

https://github.com/huggingface/datasets/blob/cd3d8e637cfab62d352a3f4e5e60e96597b5f0e9/src/datasets/__init__.py#L28

Hy, 
I have error at the above line. I have python version 3.8.13, the message says I need python>=3.7, which is True, but I think the if statement not working properly (or the message wrong)
GitHub issue #5278: load_dataset does not read jsonl metadata file properly
State: closed | Created: 2022-11-22T10:24:46Z

### Describe the bug

Hi, I'm following [this page](https://huggingface.co/docs/datasets/image_dataset) to create a dataset of images and captions via an image folder and a metadata.json file, but I can't seem to get the dataloader to recognize the "text" column. It just spits out "image" and "label" as features. 

Below is code to reproduce my exact example/problem.

### Steps to reproduce the bug

```ruby 
dataset_link="19Unu89Ih_kP6zsE7f9Mkw8dy3NwHopRF"
id = dataset_link
output = 'Godardv01.zip'
gdown.download(id=id, output=output, quiet=False)

ds = load_dataset("imagefolder", data_dir="/kaggle/working/Volumes/TOSHIBA/Godard_imgs/Volumes/TOSHIBA/Godard_imgs/Full/train", split="train", drop_labels=False)
print(ds)
```

### Expected behavior

I would expect that it returned "image" and "text" columns from the code above.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 5.0.0
- Pandas version: 1.3.5
GitHub issue #5276: Bug in downloading common_voice data and snall chunk of it to one's own hub
State: closed | Created: 2022-11-22T08:17:53Z

### Describe the bug

I'm trying to load the common voice dataset. Currently there is no implementation to download just par tof the data, and I need just one part of it, without downloading the entire dataset



Help please?


![image](https://user-images.githubusercontent.com/48530104/203260511-26df766f-6013-4eaf-be26-8aa13794def2.png)


### Steps to reproduce the bug

So here is what I have done:
1. Download common_voice data 
2. Trim part of it and publish it to my own repo.
3. Download data from my own repo, but am getting this error.

### Expected behavior

There shouldn't be an error in downloading part of the data and publishing it to one's own repo

### Environment info

common_voice 11
GitHub issue #5274: load_dataset possibly broken for gated datasets?
State: closed | Created: 2022-11-21T21:59:53Z

### Describe the bug

When trying to download the [winoground dataset](https://huggingface.co/datasets/facebook/winoground), I get this error unless I roll back the version of huggingface-hub:

```
[/usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in validate_repo_id(repo_id)
    165     if repo_id.count("/") > 1:
    166         raise HFValidationError(
--> 167             "Repo id must be in the form 'repo_name' or 'namespace/repo_name':"
    168             f" '{repo_id}'. Use `repo_type` argument if needed."
    169         )

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/facebook/winoground'. Use `repo_type` argument if needed
```

### Steps to reproduce the bug

Install requirements:

```
pip install transformers
pip install datasets
# It works if you uncomment the following line, rolling back huggingface hub:
# pip install huggingface-hub==0.10.1
```

Then:

```
from datasets import load_dataset
auth_token = ""  # Replace with an auth token, which you can get from your huggingface account: Profile -> Settings -> Access Tokens -> New Token
winoground = load_dataset("facebook/winoground", use_auth_token=auth_token)["test"]
```





### Expected behavior

Downloading of the datset

### Environment info

Just a google colab; see here: https://colab.research.google.com/drive/15wwOSte2CjTazdnCWYUm2VPlFbk2NGc0?usp=sharing
GitHub issue #5273: download_mode="force_redownload" does not refresh cached dataset
State: open | Created: 2022-11-21T14:12:43Z

### Describe the bug

`load_datasets` does not refresh dataset when features are imported from external file, even with `download_mode="force_redownload"`. The bug is not limited to nested fields, however it is more likely to occur with nested fields.

### Steps to reproduce the bug

To reproduce the bug 3 files are needed: `dataset.py` (contains dataset loading script), `schema.py` (contains features of dataset) and `main.py` (to run `load_datasets`)

`dataset.py`
```python
import datasets
from schema import features

class NewDataset(datasets.GeneratorBasedBuilder):

    def _info(self):
        return datasets.DatasetInfo(
            features=features
        )

    def _split_generators(self, dl_manager):
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN
            )
        ]

    def _generate_examples(self):
        data = [
            {"id": 0, "nested": []},
            {"id": 1, "nested": []}
        ]
        for key, example in enumerate(data):
            yield key, example
```

`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"text": datasets.Value("string")}
        ]
    }
)
```

`main.py`
```python
import datasets
a = datasets.load_dataset("dataset.py")
print(a["train"].info.features)
```

Now if `main.py` is run it prints the following correct output: `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`. However, if f.e. the label of the feature "text" is changed to something else, f.e. to


`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"textfoo": datasets.Value("string")}
        ]
    }
)
```

 `main.py` still prints `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`, even if run with `download_mode="force_redownload"`. The only fix is to delete the folder in the cache.

### Expected behavior

The cached dataset is deleted and refreshed when using `load_datasets` with `download_mode="force_redownload"`.

### Environment info

- `datasets` version: 2.7.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyArrow version: 10.0.0
- Pandas version: 1.3.5
GitHub issue #5272: Use pyarrow Tensor dtype
State: open | Created: 2022-11-20T15:18:41Z

### Feature request

I was going the discussion of converting tensors to lists.
Is there a way to leverage pyarrow's Tensors for nested arrays / embeddings?

For example:
```python
import pyarrow as pa
import numpy as np
x = np.array([[2, 2, 4], [4, 5, 100]], np.int32)
pa.Tensor.from_numpy(x, dim_names=["dim1","dim2"])
```

[Apache docs](https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html)

Maybe this belongs into the pyarrow features / repo.

### Motivation

Working with big data, we need to make sure to use the best data structures and IO out there

### Your contribution

Can try to a PR if code changes necessary
GitHub issue #5268: Sharded save_to_disk + multiprocessing
State: closed | Created: 2022-11-18T18:50:01Z

Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263 
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351
GitHub issue #5266: Specify arguments as keywords in librosa.reshape to avoid future errors 
State: closed | Created: 2022-11-18T14:58:47Z

Fixes a warning and future deprecation from `librosa.reshape`:
```
FutureWarning: Pass orig_sr=16000, target_sr=48000 as keyword args. From version 0.10 passing these as positional arguments will result in an error

   array = librosa.resample(array, sampling_rate, self.sampling_rate, res_type="kaiser_best")
```
GitHub issue #5265: Get an IterableDataset from a map-style Dataset
State: closed | Created: 2022-11-18T14:54:40Z

This is useful to leverage iterable datasets specific features like:
- fast approximate shuffling
- lazy map, filter etc.

Iterating over the resulting iterable dataset should be at least as fast at iterating over the map-style dataset.

Here are some ideas regarding the API:
```python
# 1. 
# - consistency with load_dataset(..., streaming=True)
# - gives intuition that map/filter/etc. are done on-the-fly
ids = ds.stream()

# 2.
# - more explicit on the output type
# - but maybe sounds like a conversion tool rather than a step in a processing pipeline
ids = ds.as_iterable_dataset()
```
GitHub issue #5264: `datasets` can't read a Parquet file in Python 3.9.13
State: closed | Created: 2022-11-18T14:44:01Z

### Describe the bug

I have an error when trying to load this [dataset](https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj) (it's private but I can add you to the bigcode org). `datasets` can't read one of the parquet files in the Java subset

```python
from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", split="train", revision="v1.1.a1", use_auth_token=True)
````
```
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```
It seems to be an issue with new Python versions, Because it works in these two environements:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.4.0-131-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```

```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.12
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```
But not in this:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```

### Steps to reproduce the bug

Load the dataset in python 3.9.13

### Expected behavior

Load the dataset without the pyarrow error.

### Environment info

```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```
GitHub issue #5262: AttributeError: 'Value' object has no attribute 'names'
State: closed | Created: 2022-11-18T13:58:42Z

Hello

I'm trying to build a model for custom token classification 
I already followed the token classification course on huggingface

while adapting the code to my work, this message occures : 
'Value' object has no attribute 'names'

Here's my code:
`raw_datasets` 
generates
DatasetDict({
    train: Dataset({
        features: ['isDisf', 'pos', 'tokens', 'id'],
        num_rows: 14
    })
})

`raw_datasets["train"][3]["isDisf"]`
generates 
['B_RM', 'I_RM', 'I_RM', 'B_RP', 'I_RP', 'O', 'O']

`dis_feature = raw_datasets["train"].features["isDisf"]
dis_feature`
generates
Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)

and
`label_names = dis_feature.feature.names
label_names`
generates
AttributeError                            Traceback (most recent call last)
[<ipython-input-28-972fd54a869a>](https://localhost:8080/#) in <module>
----> 1 label_names = dis_feature.feature.names
      2 label_names

AttributeError: 'Value' object has
AttributeError: 'Value' object has no attribute 'names'

Thank you for your help
GitHub issue #5261: Add PubTables-1M
State: open | Created: 2022-11-18T07:56:36Z

### Name

PubTables-1M

### Paper

https://openaccess.thecvf.com/content/CVPR2022/html/Smock_PubTables-1M_Towards_Comprehensive_Table_Extraction_From_Unstructured_Documents_CVPR_2022_paper.html

### Data

https://github.com/microsoft/table-transformer

### Motivation

Table Transformer is now available in 🤗 Transformer, and it was trained on PubTables-1M. It's a large dataset for table extraction and structure recognition in unstructured documents.
GitHub issue #5260: consumer-finance-complaints dataset not loading
State: open | Created: 2022-11-17T20:10:26Z

### Describe the bug

Error during dataset loading

### Steps to reproduce the bug

```
>>> import datasets
>>> cf_raw = datasets.load_dataset("consumer-finance-complaints")
Downloading builder script: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.42k/8.42k [00:00<00:00, 3.33MB/s]
Downloading metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.60k/5.60k [00:00<00:00, 2.90MB/s]
Downloading readme: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.6k/16.6k [00:00<00:00, 510kB/s]
Downloading and preparing dataset consumer-finance-complaints/default to /root/.cache/huggingface/datasets/consumer-finance-complaints/default/0.0.0/30e483d37fb4b25bb98cad1bfd2dc48f6ed6d1f3371eb4568c625a61d1a79b69...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 511M/511M [00:04<00:00, 103MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/skunk-pod-storage-lee-2emartie-40ibm-2ecom-pvc/anaconda3/envs/datasets/lib/python3.8/site-packages/datasets/load.py", line 1741, in load_dataset
    builder_instance.download_and_prepare(
  File "/skunk-pod-storage-lee-2emartie-40ibm-2ecom-pvc/anaconda3/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 822, in download_and_prepare
    self._download_and_prepare(
  File "/skunk-pod-storage-lee-2emartie-40ibm-2ecom-pvc/anaconda3/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 1555, in _download_and_prepare
    super()._download_and_prepare(
  File "/skunk-pod-storage-lee-2emartie-40ibm-2ecom-pvc/anaconda3/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 931, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/skunk-pod-storage-lee-2emartie-40ibm-2ecom-pvc/anaconda3/envs/datasets/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=1605177353, num_examples=2455765, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=2043641693, num_examples=3079747, shard_lengths=[721000, 656000, 788000, 846000, 68747], dataset_name='consumer-finance-complaints')}]
```

### Expected behavior

dataset should load 

### Environment info

>>> datasets.__version__
'2.7.0'

Python 3.8.10

"Ubuntu 20.04.4 LTS"
GitHub issue #5258: Restore order of split names in dataset_info for canonical datasets
State: closed | Created: 2022-11-17T15:13:15Z

After a bulk edit of canonical datasets to create the YAML `dataset_info` metadata, the split names were accidentally sorted alphabetically. See for example:
- https://huggingface.co/datasets/bc2gm_corpus/commit/2384629484401ecf4bb77cd808816719c424e57c

Note that this order is the one appearing in the preview of the datasets.

I'm making a bulk edit to align the order of the splits appearing in the metadata info with the order appearing in the loading script.

Related to:
- #5202
GitHub issue #5255: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI
State: closed | Created: 2022-11-17T03:22:22Z

### Name

NYUDepth

### Paper

http://cs.nyu.edu/~silberman/papers/indoor_seg_support.pdf

### Data

https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

### Motivation

Depth estimation is an important problem in computer vision. We have a couple of Depth Estimation models on Hub as well:

* [GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)
* [DPT](https://huggingface.co/docs/transformers/model_doc/dpt)

Would be nice to have a dataset for depth estimation. These datasets usually have three things: input image, depth map image, and depth mask (validity mask to indicate if a reading for a pixel is valid or not). Since we already have [semantic segmentation datasets on the Hub](https://huggingface.co/datasets?task_categories=task_categories:image-segmentation&sort=downloads), I don't think we need any extended utilities to support this addition. 

Having this dataset would also allow us to author data preprocessing guides for depth estimation, particularly like the ones we have for other tasks ([example](https://huggingface.co/docs/datasets/image_classification)). 

Ccing @osanseviero @nateraw @NielsRogge 

Happy to work on adding it.
GitHub issue #5252: Support for decoding Image/Audio types in map when format type is not default one
State: closed | Created: 2022-11-16T15:02:13Z

Add support for decoding the `Image`/`Audio` types in `map` for the formats (Numpy, TF, Jax, PyTorch) other than the default one (Python).

Additional improvements:
* make `Dataset`'s "iter" API  cleaner by removing `_iter` and replacing `_iter_batches` with `iter(batch_size)` (also implemented for `IterableDataset`)
* iterate over arrow tables in `map` to avoid `_getitem` calls, which are much slower than `__iter__`/`iter(batch_size)`, when the `format_type` is not Python
* fix `_iter_batches` (now named `iter`) when `drop_last_batch=True` and `pyarrow<=8.0.0` is installed
* lazily extract and decode arrow data in the default format 

TODO:
* [x] update the `iter` benchmark in the docs (the `BeamBuilder` cannot load the preprocessed datasets from our bucket, so wait for this to be fixed (cc @lhoestq))

Fix https://github.com/huggingface/datasets/issues/3992, fix https://github.com/huggingface/datasets/issues/3756