liuqingwen/github-issues huggingface dataset:
`load_dataset` has ~4 seconds of overhead for cached data
Labels: name: enhancement
description: New feature or request

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.

Comments:
Hi ! To skip the verification step that checks if newer data exist, you can enable offline mode with `HF_DATASETS_OFFLINE=1`.

Although I agree this step should be much faster for datasets hosted on the HF Hub - we could just compare the commit hash from the local data and the remote git repository. We're not been leveraging the git commit hashes, since the library was built before we even had git repositories for each dataset on HF.

Thanks @lhoestq, for memory when I recorded those times I had `HF_DATASETS_OFFLINE` set.
TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset

### Describe the bug

Hi, 

Thanks for the amazing work on the library!

**Describe the bug**

I think I might have noticed a small bug in the filter method. 
Having loaded a dataset using `load_dataset`,  when I try to filter out empty entries with `batched=True`, I get a TypeError.

### Steps to reproduce the bug

```
train_dataset = train_dataset.filter(
            function=lambda example: example["image"] is not None, 
            batched=True,
            batch_size=10)
```
Error message:

```
File .../lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
...
-> 5666 indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
   5667 if indices_mapping is not None:
   5668     indices_array = pa.array(indices_array, type=pa.uint64())

TypeError: 'bool' object is not iterable
```

**Removing batched=True allows to bypass the issue.** 

### Expected behavior

According to the doc, "[batch_size corresponds to the] number of examples per batch provided to function if batched = True", so we shouldn't need to remove the batchd=True arg?

source: https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.Dataset.filter

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments:
Hi! Instead of a single boolean, your filter function should return an iterable (of booleans) in the batched mode like so:
```python
train_dataset = train_dataset.filter(
            function=lambda batch: [image is not None for image in batch["image"]], 
            batched=True,
            batch_size=10)
```

PS: You can make this operation much faster by operating directly on the arrow data to skip the decoding part:
```python
train_dataset = train_dataset.with_format("arrow")
train_dataset = train_dataset.filter(
            function=lambda table: table["image"].is_valid().to_pylist(), 
            batched=True,
            batch_size=100)
train_dataset = train_dataset.with_format(None)
```

Thank a lot!
Add a `reduce` method
Labels: name: enhancement
description: New feature or request

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend. 

Comments:
Hi! Sure, feel free to open a PR, so we can see the API you have in mind.

I would like to give it a go! #self-assign
Update audio installation doc page
Labels: name: documentation
description: Improvements or additions to documentation

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327 
So we should update the doc page. But first investigate [this issue](5488). 

Comments:
Totally agree, the docs should be in sync with our code.

Indeed to avoid confusing users, I think we should have updated the docs at the same time as this PR:
- #5167

@albertvillanova yeah sure I should have, but I forgot back then, sorry for that 😶

No, @polinaeterna, nothing to be sorry about.

My comment was for all of us datasets team, as a reminder: when making a PR, but also when reviewing some other's PR, we should not forget to update the corresponding docstring and doc pages. It is something we can improve if we help each other in reminding about it... :hugs: 

@polinaeterna I think we can close this issue now as we no longer use `torchaudio` for decoding.
Push_to_hub in a pull request
Labels: name: enhancement
description: New feature or request

name: good first issue
description: Good for newcomers

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw 

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR

Comments:
Assigned to myself and will get to it in the next week, but if someone finds this issue annoying and wants to submit a PR before I do, just ping me here and I'll reassign :). 

I would like to be assigned to this issue, @nateraw . #self-assign
Reload features from Parquet metadata
Labels: name: enhancement
description: New feature or request

name: good second issue
description: Issues a bit more difficult than "Good First" issues

The idea would be to allow this :
```python 
ds.to_parquet("my_dataset/ds.parquet")
reloaded = load_dataset("my_dataset")
assert ds.features == reloaded.features 
```
And it should also work with Image and Audio types (right now they're reloaded as a dict type)

This can be implemented by storing and reading the feature types in the parquet metadata, as we do for arrow files.

Comments:
I'd be happy to have a look, if nobody else has started working on this yet @lhoestq. 

It seems to me that for the `arrow` format features are currently attached as metadata [in `datasets.arrow_writer`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/arrow_writer.py#L412)  and retrieved from the metadata at `load_dataset` time using [`datasets.features.features.from_arrow_schema`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/features/features.py#L1602). 

This will need to be replicated for `parquet` via calls to [this api](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html) from `io.parquet.ParquetWriter` and `io.parquet.ParquetReader` [respectively](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/io/parquet.py#L104).

Any other important considerations?


Thanks @MFreidank ! That's correct :)

Reading the metadata to infer the features can be ideally done in the `parquet.py` file in `packaged_builder` when a parquet file is read. You can cast the arrow table to the schema you get from the features.arrow_schema

#self-assign
Column project operation on `datasets.Dataset`
Labels: name: duplicate
description: This issue or pull request already exists

name: enhancement
description: New feature or request

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.

Comments:
Hi ! This would be a nice addition indeed :) This sounds like a duplicate of https://github.com/huggingface/datasets/issues/5468

> Not sure. Some of my PRs are still open and some do not have any discussions.

Sorry to hear that, feel free to ping me on those PRs
Allow opposite of remove_columns on Dataset and DatasetDict
Labels: name: enhancement
description: New feature or request

name: good first issue
description: Good for newcomers

### Feature request

In this blog post https://huggingface.co/blog/audio-datasets, I noticed the following code:

```python
COLUMNS_TO_KEEP = ["text", "audio"]
all_columns = gigaspeech["train"].column_names
columns_to_remove = set(all_columns) - set(COLUMNS_TO_KEEP)

gigaspeech = gigaspeech.remove_columns(columns_to_remove)
```

This kind of thing happens a lot when you don't need to keep all columns from the dataset. It would be more convenient (and less error prone) if you could just write:

```python
gigaspeech = gigaspeech.keep_columns(["text", "audio"])
```

Internally, `keep_columns` could still call `remove_columns`, but it expresses more clearly what the user's intent is.

### Motivation

Less code to write for the user of the dataset.

### Your contribution

-

Comments:
Hi! I agree it would be nice to have a method like that. Instead of `keep_columns`, we can name it `select_columns` to be more aligned with PyArrow's naming convention (`pa.Table.select`).

Hi, I am a newbie to open source and would like to contribute. @mariosasko can I take up this issue ?

Hey, I also want to work on this issue I am a newbie to open source. 

This sounds related to https://github.com/huggingface/datasets/issues/5474

I'm fine with `select_columns`, or we could also override `select` to also accept a list of columns maybe ?

@lhoestq, I am planning to add a member function to the dataset class to perform the selection operation. Do you think its the right way to proceed? or there is a better option ?

Unless @mariosasko thinks otherwise, I think it can go in `Dataset.select()` :)
Though some parameters like keep_in_memory, indices_cache_file_name or writer_batch_size wouldn't when selecting columns, so we would need to update the docstring as well

If someone wants to give it a shot, feel free to comment `#self-assign` and it will assign the issue to you.

Feel free to ping us here if you have questions or if we can help :)

I would rather have this functionality as a separate method. IMO it's always better to be explicit than to have an API where a single method can do different/uncorrelated things (somewhat reminds me of Pandas, and there is probably a good reason why PyArrow is more rigid in this aspect).

In the end I also think it would be nice to have it as a separate method, this way we can also have it for `IterableDataset` (which can't have `select` for indices)
Save and resume the state of a DataLoader
Labels: name: enhancement
description: New feature or request

name: generic discussion
description: Generic discussion on the library

It would be nice when using `datasets` with a PyTorch DataLoader to be able to resume a training from a DataLoader state (e.g. to resume a training that crashed)

What I have in mind (but lmk if you have other ideas or comments):

For map-style datasets, this requires to have a PyTorch Sampler state that can be saved and reloaded per node and worker.

For iterable datasets, this requires to save the state of the dataset iterator, which includes:
- the current shard idx and row position in the current shard
- the epoch number
- the rng state
- the shuffle buffer

Right now you can already resume the data loading of an iterable dataset by using `IterableDataset.skip` but it takes a lot of time because it re-iterates on all the past data until it reaches the resuming point.

cc @stas00 @sgugger 

Comments:
Something that'd be nice to have is "manual update of state". One of the learning from training LLMs is the ability to skip some batches whenever we notice huge spike might be handy.

Your outline spec is very sound and clear, @lhoestq - thank you!

@thomasw21, indeed that would be a wonderful extra feature. In Megatron-Deepspeed we manually drained the dataloader for the range we wanted. I wasn't very satisfied with the way we did it, since its behavior would change if you were to do multiple range skips. I think it should remember all the ranges it skipped and not just skip the last range - since otherwise the data is inconsistent (but we probably should discuss this in a separate issue not to derail this much bigger one).
Support fsspec 2023.1.0 in CI
Labels: name: enhancement
description: New feature or request

Once we find out the root cause of:
- #5445 

we should revert the temporary pin on fsspec introduced by:
- #5447

Comments:
CI tests are broken: AttributeError: 'mappingproxy' object has no attribute 'target'
Labels: name: bug
description: Something isn't working

CI tests are broken, raising `AttributeError: 'mappingproxy' object has no attribute 'target'`. See: https://github.com/huggingface/datasets/actions/runs/3966497597/jobs/6797384185
```
...
ERROR tests/test_streaming_download_manager.py::TestxPath::test_xpath_rglob[mock://top_level-date=2019-10-0[1-4]/*-expected_paths4] - AttributeError: 'mappingproxy' object has no attribute 'target'
===== 2076 passed, 19 skipped, 15 warnings, 47 errors in 115.54s (0:01:55) =====
```

Comments:
OneDrive Integrations  with HF Datasets 
Labels: name: enhancement
description: New feature or request

### Feature request

First of all , I would like to thank all community who are developed DataSet storage and make it free available  
How to integrate our Onedrive account or any other possible storage clouds (like google drive,...) with the **HF** datasets section.
For example, if  I have **50GB** on my **Onedrive** account and I want to move between drive and Hugging face repo or vis versa 

### Motivation

make the dataset section more flexible with other possible storage 
like the integration between Google Collab and Google drive the storage

### Your contribution

Can be done using Hugging face CLI 

Comments:
Hi! 

We use [`fsspec`](https://github.com/fsspec/filesystem_spec) to integrate with storage providers. You can find more info (and the usage examples) in [our docs](https://huggingface.co/docs/datasets/v2.8.0/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage).

[`gdrivefs`](https://github.com/fsspec/gdrivefs) makes it possible to use Google Drive as a storage service in Datasets, but this is not the case for OneDrive, since its[ Python SDK](https://github.com/OneDrive/onedrive-sdk-python) is not integrated with `fsspec`. Can you please request the integration with `fsspec` in their repo to address this limitation?

I'm closing this issue as implementing a fsspec-compliant OneDrive filesystem is not our responsibility.
[dataset request] Add Common Voice 12.0
Labels: name: enhancement
description: New feature or request

### Feature request

Please add the common voice 12_0 datasets. Apart from English, a significant amount of audio-data has been added to the other minor-language datasets.

### Motivation

The dataset link:
https://commonvoice.mozilla.org/en/datasets


Comments:
@polinaeterna any tentative date on when the Common Voice 12.0 dataset will be added ?
Support latest Docker image in CI benchmarks
Labels: name: enhancement
description: New feature or request

Once we find out the root cause of:
- #5431 

we should revert the temporary pin on the Docker image version introduced by:
- #5432

Comments:
Sorry, it was us:[^1] https://github.com/iterative/cml/pull/1317 & https://github.com/iterative/cml/issues/1319#issuecomment-1385599559; should be fixed with [v0.18.17](https://github.com/iterative/cml/releases/tag/v0.18.17).

[^1]: More or less, see https://github.com/yargs/yargs/issues/873.

Opened https://github.com/huggingface/datasets/pull/5436 unpinning again the container image.

Hi @0x2b3bfa0, thanks a lot for the investigation, the context about the the root cause and for fixing it!!

We are reviewing your PR to unpin the container image.
CI benchmarks are broken: Unknown arguments: runnerPath, path
Labels: name: maintenance
description: Maintenance tasks

Our CI benchmarks are broken, raising `Unknown arguments` error: https://github.com/huggingface/datasets/actions/runs/3932397079/jobs/6724905161
```
Unknown arguments: runnerPath, path
```

Stack trace:
```
100%|██████████| 500/500 [00:01<00:00, 338.98ba/s]
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add dvc.lock

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
cml send-comment <markdown file>

Global Options:
  --log                    Logging verbosity
          [string] [choices: "error", "warn", "info", "debug"] [default: "info"]
  --driver                 Git provider where the repository is hosted
    [string] [choices: "github", "gitlab", "bitbucket"] [default: infer from the
                                                                    environment]
  --repo                   Repository URL or slug
                                  [string] [default: infer from the environment]
  --driver-token, --token  CI driver personal/project access token (PAT)
                                  [string] [default: infer from the environment]
  --help                   Show help                                   [boolean]

Options:
  --target                    Comment type (`commit`, `pr`, `commit/f00bar`,
                              `pr/42`, `issue/1337`),default is automatic (`pr`
                              but fallback to `commit`).                [string]
  --watch                     Watch for changes and automatically update the
                              comment                                  [boolean]
  --publish                   Upload any local images found in the Markdown
                              report                   [boolean] [default: true]
  --publish-url               Self-hosted image server URL
                                     [string] [default: "https://asset.cml.dev/"]
  --publish-native, --native  Uses driver's native capabilities to upload assets
                              instead of CML's storage; not available on GitHub
                                                                       [boolean]
  --watermark-title           Hidden comment marker (used for targeting in
                              subsequent `cml comment update`); "{workflow}" &
                              "{run}" are auto-replaced   [string] [default: ""]

Unknown arguments: runnerPath, path
Error: Process completed with exit code 1.
```

Issue reported to iterative/cml:
- iterative/cml#1319

Comments:
Support Apache Beam >= 2.44.0
Labels: name: enhancement
description: New feature or request

Once we find out the root cause of:
- #5426 

we should revert the temporary pin on apache-beam introduced by:
- #5429

Comments:
Some of the shard files now have 0 number of rows.

We have opened an issue in the Apache Beam repo:
- https://github.com/apache/beam/issues/25041
Load/Save FAISS index using fsspec
Labels: name: enhancement
description: New feature or request

### Feature request

From what I understand `faiss` already support this [link](https://github.com/facebookresearch/faiss/wiki/Index-IO,-cloning-and-hyper-parameter-tuning#generic-io-support)

I would like to use a stream as input to `Dataset.load_faiss_index` and `Dataset.save_faiss_index`.

### Motivation

In my case, I'm saving faiss index in cloud storage and use `fsspec` to load them. It would be ideal if I could send the stream directly instead of copying the file locally (or mounting the bucket) and then load the index.

### Your contribution

I can submit the PR

Comments:
Hi! Sure, feel free to submit a PR. Maybe if we want to be consistent with the existing API, it would be cleaner to directly add support for `fsspec` paths in `Dataset.load_faiss_index`/`Dataset.save_faiss_index` in the same manner as it was done in `Dataset.load_from_disk`/`Dataset.save_to_disk`.

That's a great idea! I'll do that instead.
Support case-insensitive Hub dataset name in load_dataset
Labels: name: enhancement
description: New feature or request

### Feature request

The dataset name on the Hub is case-insensitive (see https://github.com/huggingface/moon-landing/pull/2399, internal issue), i.e., https://huggingface.co/datasets/GLUE redirects to https://huggingface.co/datasets/glue.

Ideally, we could load the glue dataset using the following:

```
from datasets import load_dataset
load_dataset('GLUE', 'cola')
```

It breaks because the loading script `GLUE.py` does not exist (`glue.py` should be selected instead).

Minor additional comment: in other cases without a loading script, we can load the dataset, but the automatically generated config name depends on the casing:
- `load_dataset('severo/danish-wit')` generates the config name `severo--danish-wit-e6fda5b070deb133`, while
- `load_dataset('severo/danish-WIT')` generates the config name `severo--danish-WIT-e6fda5b070deb133`

### Motivation

To follow the same UX on the Hub and in the datasets library.

### Your contribution

...

Comments:
Closing as case-insensitivity should be only for URL redirection on the Hub. In the APIs, we will only support the canonical name (https://github.com/huggingface/moon-landing/pull/2399#issuecomment-1382085611)