helmo/github-issues huggingface dataset:
TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset
State: closed | Author: vmuel (NONE)

### Describe the bug

Hi, 

Thanks for the amazing work on the library!

**Describe the bug**

I think I might have noticed a small bug in the filter method. 
Having loaded a dataset using `load_dataset`,  when I try to filter out empty entries with `batched=True`, I get a TypeError.

### Steps to reproduce the bug

```
train_dataset = train_dataset.filter(
            function=lambda example: example["image"] is not None, 
            batched=True,
            batch_size=10)
```
Error message:

```
File .../lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
...
-> 5666 indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
   5667 if indices_mapping is not None:
   5668     indices_array = pa.array(indices_array, type=pa.uint64())

TypeError: 'bool' object is not iterable
```

**Removing batched=True allows to bypass the issue.** 

### Expected behavior

According to the doc, "[batch_size corresponds to the] number of examples per batch provided to function if batched = True", so we shouldn't need to remove the batchd=True arg?

source: https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.Dataset.filter

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments: Hi! Instead of a single boolean, your filter function should return an iterable (of booleans) in the batched mode like so:
```python
train_dataset = train_dataset.filter(
            function=lambda batch: [image is not None for image in batch["image"]], 
            batched=True,
            batch_size=10)
```

PS: You can make this operation much faster by operating directly on the arrow data to skip the decoding part:
```python
train_dataset = train_dataset.with_format("arrow")
train_dataset = train_dataset.filter(
            function=lambda table: table["image"].is_valid().to_pylist(), 
            batched=True,
            batch_size=100)
train_dataset = train_dataset.with_format(None)
```

Thank a lot!

I hit the same issue and the error message isn't really clear on what's going wrong. It might be helpful to update the docs with a batched example.
Add a `reduce` method
State: closed | Author: zhangir-azerbayev (NONE)
Labels: name: enhancement

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend. 

Comments: Hi! Sure, feel free to open a PR, so we can see the API you have in mind.

I would like to give it a go! #self-assign

Closing as `Dataset.map` can be used instead (see https://github.com/huggingface/datasets/pull/5533#issuecomment-1440571658 and https://github.com/huggingface/datasets/pull/5533#issuecomment-1446403263)

Hello, is it possible for this issue/PR to be revisited? The problem with the alternatives presented (besides multiple map stages) is that they don't use the cache. A reduce operation is just as expensive as a map operation because it also goes over the entire dataset. It's equally worth caching.

Personally, I have a situation where I would need this and map is far from ideal. I'm working on updating a project of mine to use Huggingface Datasets, and I need to port the loop at https://github.com/colonelwatch/abstracts-search/blob/b90f31ee4cc6e394f829d3a6d9d0311ca390ada9/train.py#L112-L138. Please forgive the code style, here's what it does in English. I have a dataset of about 95 million embeddings, out of which 16384 is taken as a "query" set. For each embedding in the query set, I need to find the ten closest neighbors. These nearest neighbors are used to tune the parameters of a faiss index. The solution is to set up an "accumulator" comprising of the ten closest so far and their distances, then do a single scan over the 95 million (memmapped), then save the results of the "accumulator" for when I want to prototype another index.

The closest approximation to this is multiple map stages, but with such a large "accumulator" having the RAM to do a big batch size becomes critical. At a batch size of 1000, the intermediate accumulators would in theory be about 120 GB! That can be more if I want higher precision than float32. It would already be about the same size as the original embeddings. Using larger batch sizes puts strain on the RAM because I'd be dealing with batch_size x 16384 distances. The best I'd gotten with my RAM, single-threaded, was 65536, and for speed I had to use that thread to feed a GPU. It'd be better if I could use multiple threads to get high throughput instead, or even do all the work in CPU, but to fit the threads I'd need the batch size to be smaller.

All of this intermediate memory could be eliminated if there was a reduce operation.
Update audio installation doc page
State: closed | Author: polinaeterna (CONTRIBUTOR)
Labels: name: documentation

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327 
So we should update the doc page. But first investigate [this issue](5488). 

Comments: Totally agree, the docs should be in sync with our code.

Indeed to avoid confusing users, I think we should have updated the docs at the same time as this PR:
- #5167

@albertvillanova yeah sure I should have, but I forgot back then, sorry for that 😶

No, @polinaeterna, nothing to be sorry about.

My comment was for all of us datasets team, as a reminder: when making a PR, but also when reviewing some other's PR, we should not forget to update the corresponding docstring and doc pages. It is something we can improve if we help each other in reminding about it... :hugs: 

@polinaeterna I think we can close this issue now as we no longer use `torchaudio` for decoding.
Push_to_hub in a pull request
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

name: good first issue

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw 

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR

Comments: Assigned to myself and will get to it in the next week, but if someone finds this issue annoying and wants to submit a PR before I do, just ping me here and I'll reassign :). 

I would like to be assigned to this issue, @nateraw . #self-assign
Adding `sep` to TextConfig 
State: open | Author: omar-araboghli (NONE)

I have a local a `.txt` file that follows the `CONLL2003` format which I need to load using `load_script`. However, by using `sample_by='line'`, one can only split the dataset into lines without splitting each line into columns. Would it be reasonable to add a `sep` argument in combination with `sample_by='paragraph'` to parse a paragraph into an array for each column ? If so, I am happy to contribute!

## Environment
* `python 3.8.10`
* `datasets 2.9.0`

## Snippet of `train.txt`

```txt
Distribution	NN	O	O
and	NN	O	O
dynamics	NN	O	O
of	NN	O	O
electron	NN	O	B-RP
complexes	NN	O	I-RP
in	NN	O	O
cyanobacterial	NN	O	B-R
membranes	NN	O	I-R

The	NN	O	O
occurrence	NN	O	O
of	NN	O	O
prostaglandin	NN	O	B-R
F2α	NN	O	I-R
in	NN	O	O
Pharbitis	NN	O	B-R
seedlings	NN	O	I-R
grown	NN	O	O
under	NN	O	O
short	NN	O	B-P
days	NN	O	I-P
or	NN	O	I-P
days	NN	O	I-P

```

## Current Behaviour 
```python
# defining 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'] here would fail with `ValueError: Length of names (4) does not match length of arrays (1)`
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='line')
dataset['train']['tokens'][0]
>>> 'Distribution\tNN\tO\tO'
```

## Expected Behaviour / Suggestion
```python
# suppose we defined 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='paragraph', sep='\t')
dataset['train']['tokens'][0]
>>> ['Distribution', 'and', 'dynamics', ... ]

dataset['train']['ner_tags'][0]
>>> ['O', 'O', 'O', ... ]
```

Comments: Hi @omar-araboghli, thanks for your proposal.

Have you tried to use "csv" loader instead of "text"? That already has a `sep` argument.

Hi @albertvillanova, thanks for the quick response!

Indeed, I have been trying to use `csv` instead of `text`. However I am still not able to define range of rows as one sequence, that is achievable with passing `sample_by='paragraph'` to the `TextConfig`

For instance, the below code

```python
import datasets

dataset = datasets.load_dataset(
    path='csv',
    data_files={'train': TRAINING_SET_PATH},
    sep='\t',
    header=None,
    column_names=['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
)
```

leads to 

```python
dataset
>>> DatasetDict({
    train: Dataset({
        features: ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 62543
    })
})

dataset['train'][0]
>>> {'tokens': 'Distribution',
 'pos_tags': 'NN',
 'chunk_tags': 'O',
 'ner_tags': 'O'
}
```
Is there a way to deal with multiple csv rows as one dataset instance, where each column is a sequence of those rows ?
Reload features from Parquet metadata
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

name: good second issue

The idea would be to allow this :
```python 
ds.to_parquet("my_dataset/ds.parquet")
reloaded = load_dataset("my_dataset")
assert ds.features == reloaded.features 
```
And it should also work with Image and Audio types (right now they're reloaded as a dict type)

This can be implemented by storing and reading the feature types in the parquet metadata, as we do for arrow files.

Comments: I'd be happy to have a look, if nobody else has started working on this yet @lhoestq. 

It seems to me that for the `arrow` format features are currently attached as metadata [in `datasets.arrow_writer`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/arrow_writer.py#L412)  and retrieved from the metadata at `load_dataset` time using [`datasets.features.features.from_arrow_schema`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/features/features.py#L1602). 

This will need to be replicated for `parquet` via calls to [this api](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html) from `io.parquet.ParquetWriter` and `io.parquet.ParquetReader` [respectively](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/io/parquet.py#L104).

Any other important considerations?


Thanks @MFreidank ! That's correct :)

Reading the metadata to infer the features can be ideally done in the `parquet.py` file in `packaged_builder` when a parquet file is read. You can cast the arrow table to the schema you get from the features.arrow_schema

#self-assign
Unpin sqlalchemy once issue is fixed
State: closed | Author: albertvillanova (MEMBER)

Once the source issue is fixed:
- pandas-dev/pandas#51015

we should revert the pin introduced in:
- #5476

Comments: @albertvillanova It looks like that issue has been fixed so I made a PR to unpin sqlalchemy! 

The source issue:
- https://github.com/pandas-dev/pandas/issues/40686

has been fixed:
- https://github.com/pandas-dev/pandas/pull/48576

The fix was released yesterday (2023-04-03) only in `pandas-2.0.0`:
- https://github.com/pandas-dev/pandas/releases/tag/v2.0.0

but it will not be back-ported to `pandas-1`:
- https://github.com/pandas-dev/pandas/pull/48576#issuecomment-1466467159

Also note that `pandas-2.0.0` dropped support for Python 3.7:
- https://github.com/pandas-dev/pandas/issues/41678
- https://github.com/pandas-dev/pandas/pull/41989

Therefore, we cannot unpin `sqlalchemy` until we drop support for Python 3.7 (these Python users cannot use `pandas-2`).
Dataset scan time is much slower than using native arrow
State: closed | Author: jonny-cyberhaven (CONTRIBUTOR)

### Describe the bug

I'm basically running the same scanning experiment from the tutorials https://huggingface.co/course/chapter5/4?fw=pt except now I'm comparing to a native pyarrow version.

I'm finding that the native pyarrow approach is much faster (2 orders of magnitude). Is there something I'm missing that explains this phenomenon?

### Steps to reproduce the bug

https://colab.research.google.com/drive/11EtHDaGAf1DKCpvYnAPJUW-LFfAcDzHY?usp=sharing

### Expected behavior

I expect scan times to be on par with using pyarrow directly.

### Environment info

standard colab environment

Comments: Hi ! In your code you only iterate on the Arrow buffers - you don't actually load the data as python objects. For a fair comparison, you can modify your code using:
```diff
-     for _ in range(0, len(table), bsz):
-         _ = {k:table[k][_ : _ + bsz] for k in cols}
+     for _ in range(0, len(table), bsz):
+         _ = {k:table[k][_ : _ + bsz].to_pylist() for k in cols}
```

I re-ran your code and got a speed ratio of 1.00x and 1.02x

Ah I see, datasets is implicitly making this conversion. Thanks for pointing that out!

If it's not too much, I would also suggest updating some of your docs with the same `.to_pylist()` conversion in the code snippet that follows [here](https://huggingface.co/course/chapter5/4?fw=pt#:~:text=let%E2%80%99s%20run%20a%20little%20speed%20test%20by%20iterating%20over%20all%20the%20elements%20in%20the%20PubMed%20Abstracts%20dataset%3A).

This code snippet shows `datasets` code that reads the Arrow data as python objects already, there is no need to add to_pylist. Or were you thinking about something else ?
Column project operation on `datasets.Dataset`
State: closed | Author: daskol (CONTRIBUTOR)
Labels: name: duplicate

name: enhancement

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.

Comments: Hi ! This would be a nice addition indeed :) This sounds like a duplicate of https://github.com/huggingface/datasets/issues/5468

> Not sure. Some of my PRs are still open and some do not have any discussions.

Sorry to hear that, feel free to ping me on those PRs
Allow opposite of remove_columns on Dataset and DatasetDict
State: closed | Author: hollance (NONE)
Labels: name: enhancement

name: good first issue

### Feature request

In this blog post https://huggingface.co/blog/audio-datasets, I noticed the following code:

```python
COLUMNS_TO_KEEP = ["text", "audio"]
all_columns = gigaspeech["train"].column_names
columns_to_remove = set(all_columns) - set(COLUMNS_TO_KEEP)

gigaspeech = gigaspeech.remove_columns(columns_to_remove)
```

This kind of thing happens a lot when you don't need to keep all columns from the dataset. It would be more convenient (and less error prone) if you could just write:

```python
gigaspeech = gigaspeech.keep_columns(["text", "audio"])
```

Internally, `keep_columns` could still call `remove_columns`, but it expresses more clearly what the user's intent is.

### Motivation

Less code to write for the user of the dataset.

### Your contribution

-

Comments: Hi! I agree it would be nice to have a method like that. Instead of `keep_columns`, we can name it `select_columns` to be more aligned with PyArrow's naming convention (`pa.Table.select`).

Hi, I am a newbie to open source and would like to contribute. @mariosasko can I take up this issue ?

Hey, I also want to work on this issue I am a newbie to open source. 

This sounds related to https://github.com/huggingface/datasets/issues/5474

I'm fine with `select_columns`, or we could also override `select` to also accept a list of columns maybe ?

@lhoestq, I am planning to add a member function to the dataset class to perform the selection operation. Do you think its the right way to proceed? or there is a better option ?

Unless @mariosasko thinks otherwise, I think it can go in `Dataset.select()` :)
Though some parameters like keep_in_memory, indices_cache_file_name or writer_batch_size wouldn't when selecting columns, so we would need to update the docstring as well

If someone wants to give it a shot, feel free to comment `#self-assign` and it will assign the issue to you.

Feel free to ping us here if you have questions or if we can help :)

I would rather have this functionality as a separate method. IMO it's always better to be explicit than to have an API where a single method can do different/uncorrelated things (somewhat reminds me of Pandas, and there is probably a good reason why PyArrow is more rigid in this aspect).

In the end I also think it would be nice to have it as a separate method, this way we can also have it for `IterableDataset` (which can't have `select` for indices)
audiofolder creates empty dataset even though the dataset passed in follows the correct structure
State: closed | Author: jcho19 (NONE)

### Describe the bug

The structure of my dataset folder called "my_dataset" is : data  metadata.csv
The data folder consists of all mp3 files and metadata.csv consist of file locations like 'data/...mp3 and transcriptions. There's 400+ mp3 files and corresponding transcriptions for my dataset.

When I run the following:
ds = load_dataset("audiofolder", data_dir="my_dataset")


I get:
Using custom data configuration default-...
Downloading and preparing dataset audiofolder/default to /...
Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading data files: 0it [00:00, ?it/s]
Extracting data files: 0it [00:00, ?it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Dataset audiofolder downloaded and prepared to /.... Subsequent calls will reuse this data.
  0%|          | 0/1 [00:00<?, ?it/s]
DatasetDict({
    train: Dataset({
        features: ['audio', 'transcription'],
        num_rows: 1
    })
})





### Steps to reproduce the bug

Create a dataset folder called 'my_dataset' with a subfolder called 'data' that has mp3 files. Also, create metadata.csv that has file locations like 'data/...mp3' and their corresponding transcription.

Run:
ds = load_dataset("audiofolder", data_dir="my_dataset")

### Expected behavior

It should generate a dataset with numerous rows. 

### Environment info

Run on Jupyter notebook
NonMatchingChecksumError for hendrycks_test
State: closed | Author: sarahwie (NONE)

### Describe the bug

The checksum of the file has likely changed on the remote host.


### Steps to reproduce the bug

`dataset = nlp.load_dataset("hendrycks_test", "anatomy")`


### Expected behavior

no error thrown


### Environment info

- `datasets` version: 2.2.1
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Comments: Thanks for reporting, @sarahwie.

Please note this issue was already fixed in `datasets` 2.6.0 version:
- #5040

If you update your `datasets` version, you will be able to load the dataset:
```
pip install -U datasets
```

Oops, missed that I needed to upgrade. Thanks!
slice split while streaming
State: closed | Author: SvenDS9 (NONE)

### Describe the bug

When using the `load_dataset` function with streaming set to True, slicing splits is apparently not supported.

Did I miss this in the documentation?

### Steps to reproduce the bug

`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

causes ValueError: Bad split: train[:3]. Available splits: ['train', 'test'] in builder.py, line 1213, in as_streaming_dataset

### Expected behavior

The first 3 entries of the dataset as a stream

### Environment info

- `datasets` version: 2.8.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.9
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Comments: Hi! Yes, that's correct. When `streaming` is `True`, only split names can be specified as `split`, and for slicing, you have to use `.skip`/`.take` instead.

E.g. 
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

rewritten with `.skip`/`.take`:
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train").take(3)`




Thank you for your quick response!