josteece/github-issues huggingface dataset:
Add a `reduce` method
Opened by zhangir-azerbayev (closed)

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend. 

Comments:
Hi! Sure, feel free to open a PR, so we can see the API you have in mind.

I would like to give it a go! #self-assign

Closing as `Dataset.map` can be used instead (see https://github.com/huggingface/datasets/pull/5533#issuecomment-1440571658 and https://github.com/huggingface/datasets/pull/5533#issuecomment-1446403263)

Hello, is it possible for this issue/PR to be revisited? The problem with the alternatives presented (besides multiple map stages) is that they don't use the cache. A reduce operation is just as expensive as a map operation because it also goes over the entire dataset. It's equally worth caching.

Personally, I have a situation where I would need this and map is far from ideal. I'm working on updating a project of mine to use Huggingface Datasets, and I need to port the loop at https://github.com/colonelwatch/abstracts-search/blob/b90f31ee4cc6e394f829d3a6d9d0311ca390ada9/train.py#L112-L138. Please forgive the code style, here's what it does in English. I have a dataset of about 95 million embeddings, out of which 16384 is taken as a "query" set. For each embedding in the query set, I need to find the ten closest neighbors. These nearest neighbors are used to tune the parameters of a faiss index. The solution is to set up an "accumulator" comprising of the ten closest so far and their distances, then do a single scan over the 95 million (memmapped), then save the results of the "accumulator" for when I want to prototype another index.

The closest approximation to this is multiple map stages, but with such a large "accumulator" having the RAM to do a big batch size becomes critical. At a batch size of 1000, the intermediate accumulators would in theory be about 120 GB! That can be more if I want higher precision than float32. It would already be about the same size as the original embeddings. Using larger batch sizes puts strain on the RAM because I'd be dealing with batch_size x 16384 distances. The best I'd gotten with my RAM, single-threaded, was 65536, and for speed I had to use that thread to feed a GPU. It'd be better if I could use multiple threads to get high throughput instead, or even do all the work in CPU, but to fit the threads I'd need the batch size to be smaller.

All of this intermediate memory could be eliminated if there was a reduce operation.
Update audio installation doc page
Opened by polinaeterna (closed)

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327 
So we should update the doc page. But first investigate [this issue](5488). 

Comments:
Totally agree, the docs should be in sync with our code.

Indeed to avoid confusing users, I think we should have updated the docs at the same time as this PR:
- #5167

@albertvillanova yeah sure I should have, but I forgot back then, sorry for that 😶

No, @polinaeterna, nothing to be sorry about.

My comment was for all of us datasets team, as a reminder: when making a PR, but also when reviewing some other's PR, we should not forget to update the corresponding docstring and doc pages. It is something we can improve if we help each other in reminding about it... :hugs: 

@polinaeterna I think we can close this issue now as we no longer use `torchaudio` for decoding.
Push_to_hub in a pull request
Opened by lhoestq (closed)

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw 

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR

Comments:
Assigned to myself and will get to it in the next week, but if someone finds this issue annoying and wants to submit a PR before I do, just ping me here and I'll reassign :). 

I would like to be assigned to this issue, @nateraw . #self-assign
Adding `sep` to TextConfig 
Opened by omar-araboghli (open)

I have a local a `.txt` file that follows the `CONLL2003` format which I need to load using `load_script`. However, by using `sample_by='line'`, one can only split the dataset into lines without splitting each line into columns. Would it be reasonable to add a `sep` argument in combination with `sample_by='paragraph'` to parse a paragraph into an array for each column ? If so, I am happy to contribute!

## Environment
* `python 3.8.10`
* `datasets 2.9.0`

## Snippet of `train.txt`

```txt
Distribution	NN	O	O
and	NN	O	O
dynamics	NN	O	O
of	NN	O	O
electron	NN	O	B-RP
complexes	NN	O	I-RP
in	NN	O	O
cyanobacterial	NN	O	B-R
membranes	NN	O	I-R

The	NN	O	O
occurrence	NN	O	O
of	NN	O	O
prostaglandin	NN	O	B-R
F2α	NN	O	I-R
in	NN	O	O
Pharbitis	NN	O	B-R
seedlings	NN	O	I-R
grown	NN	O	O
under	NN	O	O
short	NN	O	B-P
days	NN	O	I-P
or	NN	O	I-P
days	NN	O	I-P

```

## Current Behaviour 
```python
# defining 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'] here would fail with `ValueError: Length of names (4) does not match length of arrays (1)`
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='line')
dataset['train']['tokens'][0]
>>> 'Distribution\tNN\tO\tO'
```

## Expected Behaviour / Suggestion
```python
# suppose we defined 4 features ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
dataset = datasets.load_dataset(path='text', features=features, data_files={'train': 'train.txt'}, sample_by='paragraph', sep='\t')
dataset['train']['tokens'][0]
>>> ['Distribution', 'and', 'dynamics', ... ]

dataset['train']['ner_tags'][0]
>>> ['O', 'O', 'O', ... ]
```

Comments:
Hi @omar-araboghli, thanks for your proposal.

Have you tried to use "csv" loader instead of "text"? That already has a `sep` argument.

Hi @albertvillanova, thanks for the quick response!

Indeed, I have been trying to use `csv` instead of `text`. However I am still not able to define range of rows as one sequence, that is achievable with passing `sample_by='paragraph'` to the `TextConfig`

For instance, the below code

```python
import datasets

dataset = datasets.load_dataset(
    path='csv',
    data_files={'train': TRAINING_SET_PATH},
    sep='\t',
    header=None,
    column_names=['tokens', 'pos_tags', 'chunk_tags', 'ner_tags']
)
```

leads to 

```python
dataset
>>> DatasetDict({
    train: Dataset({
        features: ['tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 62543
    })
})

dataset['train'][0]
>>> {'tokens': 'Distribution',
 'pos_tags': 'NN',
 'chunk_tags': 'O',
 'ner_tags': 'O'
}
```
Is there a way to deal with multiple csv rows as one dataset instance, where each column is a sequence of those rows ?
Reload features from Parquet metadata
Opened by lhoestq (closed)

The idea would be to allow this :
```python 
ds.to_parquet("my_dataset/ds.parquet")
reloaded = load_dataset("my_dataset")
assert ds.features == reloaded.features 
```
And it should also work with Image and Audio types (right now they're reloaded as a dict type)

This can be implemented by storing and reading the feature types in the parquet metadata, as we do for arrow files.

Comments:
I'd be happy to have a look, if nobody else has started working on this yet @lhoestq. 

It seems to me that for the `arrow` format features are currently attached as metadata [in `datasets.arrow_writer`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/arrow_writer.py#L412)  and retrieved from the metadata at `load_dataset` time using [`datasets.features.features.from_arrow_schema`](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/features/features.py#L1602). 

This will need to be replicated for `parquet` via calls to [this api](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html) from `io.parquet.ParquetWriter` and `io.parquet.ParquetReader` [respectively](https://github.com/huggingface/datasets/blob/5f810b7011a8a4ab077a1847c024d2d9e267b065/src/datasets/io/parquet.py#L104).

Any other important considerations?


Thanks @MFreidank ! That's correct :)

Reading the metadata to infer the features can be ideally done in the `parquet.py` file in `packaged_builder` when a parquet file is read. You can cast the arrow table to the schema you get from the features.arrow_schema

#self-assign
Unpin sqlalchemy once issue is fixed
Opened by albertvillanova (closed)

Once the source issue is fixed:
- pandas-dev/pandas#51015

we should revert the pin introduced in:
- #5476

Comments:
@albertvillanova It looks like that issue has been fixed so I made a PR to unpin sqlalchemy! 

The source issue:
- https://github.com/pandas-dev/pandas/issues/40686

has been fixed:
- https://github.com/pandas-dev/pandas/pull/48576

The fix was released yesterday (2023-04-03) only in `pandas-2.0.0`:
- https://github.com/pandas-dev/pandas/releases/tag/v2.0.0

but it will not be back-ported to `pandas-1`:
- https://github.com/pandas-dev/pandas/pull/48576#issuecomment-1466467159

Also note that `pandas-2.0.0` dropped support for Python 3.7:
- https://github.com/pandas-dev/pandas/issues/41678
- https://github.com/pandas-dev/pandas/pull/41989

Therefore, we cannot unpin `sqlalchemy` until we drop support for Python 3.7 (these Python users cannot use `pandas-2`).
Dataset scan time is much slower than using native arrow
Opened by jonny-cyberhaven (closed)

### Describe the bug

I'm basically running the same scanning experiment from the tutorials https://huggingface.co/course/chapter5/4?fw=pt except now I'm comparing to a native pyarrow version.

I'm finding that the native pyarrow approach is much faster (2 orders of magnitude). Is there something I'm missing that explains this phenomenon?

### Steps to reproduce the bug

https://colab.research.google.com/drive/11EtHDaGAf1DKCpvYnAPJUW-LFfAcDzHY?usp=sharing

### Expected behavior

I expect scan times to be on par with using pyarrow directly.

### Environment info

standard colab environment

Comments:
Hi ! In your code you only iterate on the Arrow buffers - you don't actually load the data as python objects. For a fair comparison, you can modify your code using:
```diff
-     for _ in range(0, len(table), bsz):
-         _ = {k:table[k][_ : _ + bsz] for k in cols}
+     for _ in range(0, len(table), bsz):
+         _ = {k:table[k][_ : _ + bsz].to_pylist() for k in cols}
```

I re-ran your code and got a speed ratio of 1.00x and 1.02x

Ah I see, datasets is implicitly making this conversion. Thanks for pointing that out!

If it's not too much, I would also suggest updating some of your docs with the same `.to_pylist()` conversion in the code snippet that follows [here](https://huggingface.co/course/chapter5/4?fw=pt#:~:text=let%E2%80%99s%20run%20a%20little%20speed%20test%20by%20iterating%20over%20all%20the%20elements%20in%20the%20PubMed%20Abstracts%20dataset%3A).

This code snippet shows `datasets` code that reads the Arrow data as python objects already, there is no need to add to_pylist. Or were you thinking about something else ?
Column project operation on `datasets.Dataset`
Opened by daskol (closed)

### Feature request

There is no operation to select a subset of columns of original dataset. Expected API follows.
```python
a = Dataset.from_dict({
    'int': [0, 1, 2]
    'char': ['a', 'b', 'c'],
    'none': [None] * 3,
})

b = a.project('int', 'char')  # usually, .select()

print(a.column_names)  # stdout: ['int', 'char', 'none']
print(b.column_names)  # stdout: ['int', 'char']
```
Method project can easily accept not only column names (as a `str)` but univariant function applied to corresponding column as an example. Or keyword arguments can be used in order to rename columns in advance (see `pandas`, `pyspark`, `pyarrow`, and SQL)..

### Motivation

Projection is a typical operation in every data processing library. And it is a basic block of a well-known data manipulation language like SQL. Without this operation `datasets.Dataset` interface is not complete.

### Your contribution

Not sure. Some of my PRs are still open and some do not have any discussions.

Comments:
Hi ! This would be a nice addition indeed :) This sounds like a duplicate of https://github.com/huggingface/datasets/issues/5468

> Not sure. Some of my PRs are still open and some do not have any discussions.

Sorry to hear that, feel free to ping me on those PRs
Allow opposite of remove_columns on Dataset and DatasetDict
Opened by hollance (closed)

### Feature request

In this blog post https://huggingface.co/blog/audio-datasets, I noticed the following code:

```python
COLUMNS_TO_KEEP = ["text", "audio"]
all_columns = gigaspeech["train"].column_names
columns_to_remove = set(all_columns) - set(COLUMNS_TO_KEEP)

gigaspeech = gigaspeech.remove_columns(columns_to_remove)
```

This kind of thing happens a lot when you don't need to keep all columns from the dataset. It would be more convenient (and less error prone) if you could just write:

```python
gigaspeech = gigaspeech.keep_columns(["text", "audio"])
```

Internally, `keep_columns` could still call `remove_columns`, but it expresses more clearly what the user's intent is.

### Motivation

Less code to write for the user of the dataset.

### Your contribution

-

Comments:
Hi! I agree it would be nice to have a method like that. Instead of `keep_columns`, we can name it `select_columns` to be more aligned with PyArrow's naming convention (`pa.Table.select`).

Hi, I am a newbie to open source and would like to contribute. @mariosasko can I take up this issue ?

Hey, I also want to work on this issue I am a newbie to open source. 

This sounds related to https://github.com/huggingface/datasets/issues/5474

I'm fine with `select_columns`, or we could also override `select` to also accept a list of columns maybe ?

@lhoestq, I am planning to add a member function to the dataset class to perform the selection operation. Do you think its the right way to proceed? or there is a better option ?

Unless @mariosasko thinks otherwise, I think it can go in `Dataset.select()` :)
Though some parameters like keep_in_memory, indices_cache_file_name or writer_batch_size wouldn't when selecting columns, so we would need to update the docstring as well

If someone wants to give it a shot, feel free to comment `#self-assign` and it will assign the issue to you.

Feel free to ping us here if you have questions or if we can help :)

I would rather have this functionality as a separate method. IMO it's always better to be explicit than to have an API where a single method can do different/uncorrelated things (somewhat reminds me of Pandas, and there is probably a good reason why PyArrow is more rigid in this aspect).

In the end I also think it would be nice to have it as a separate method, this way we can also have it for `IterableDataset` (which can't have `select` for indices)
audiofolder creates empty dataset even though the dataset passed in follows the correct structure
Opened by jcho19 (closed)

### Describe the bug

The structure of my dataset folder called "my_dataset" is : data  metadata.csv
The data folder consists of all mp3 files and metadata.csv consist of file locations like 'data/...mp3 and transcriptions. There's 400+ mp3 files and corresponding transcriptions for my dataset.

When I run the following:
ds = load_dataset("audiofolder", data_dir="my_dataset")


I get:
Using custom data configuration default-...
Downloading and preparing dataset audiofolder/default to /...
Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading data files: 0it [00:00, ?it/s]
Extracting data files: 0it [00:00, ?it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Dataset audiofolder downloaded and prepared to /.... Subsequent calls will reuse this data.
  0%|          | 0/1 [00:00<?, ?it/s]
DatasetDict({
    train: Dataset({
        features: ['audio', 'transcription'],
        num_rows: 1
    })
})





### Steps to reproduce the bug

Create a dataset folder called 'my_dataset' with a subfolder called 'data' that has mp3 files. Also, create metadata.csv that has file locations like 'data/...mp3' and their corresponding transcription.

Run:
ds = load_dataset("audiofolder", data_dir="my_dataset")

### Expected behavior

It should generate a dataset with numerous rows. 

### Environment info

Run on Jupyter notebook

Comments:
NonMatchingChecksumError for hendrycks_test
Opened by sarahwie (closed)

### Describe the bug

The checksum of the file has likely changed on the remote host.


### Steps to reproduce the bug

`dataset = nlp.load_dataset("hendrycks_test", "anatomy")`


### Expected behavior

no error thrown


### Environment info

- `datasets` version: 2.2.1
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Comments:
Thanks for reporting, @sarahwie.

Please note this issue was already fixed in `datasets` 2.6.0 version:
- #5040

If you update your `datasets` version, you will be able to load the dataset:
```
pip install -U datasets
```

Oops, missed that I needed to upgrade. Thanks!
slice split while streaming
Opened by SvenDS9 (closed)

### Describe the bug

When using the `load_dataset` function with streaming set to True, slicing splits is apparently not supported.

Did I miss this in the documentation?

### Steps to reproduce the bug

`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

causes ValueError: Bad split: train[:3]. Available splits: ['train', 'test'] in builder.py, line 1213, in as_streaming_dataset

### Expected behavior

The first 3 entries of the dataset as a stream

### Environment info

- `datasets` version: 2.8.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.9
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Comments:
Hi! Yes, that's correct. When `streaming` is `True`, only split names can be specified as `split`, and for slicing, you have to use `.skip`/`.take` instead.

E.g. 
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train[:3]")`

rewritten with `.skip`/`.take`:
`load_dataset("lhoestq/demo1",revision=None, streaming=True, split="train").take(3)`




Thank you for your quick response!
ImageFolder BadZipFile: Bad offset for central directory
Opened by hmartiro (closed)

### Describe the bug

I'm getting the following exception:
```
lib/python3.10/zipfile.py:1353 in _RealGetContents        │
│                                                                                                  │
│   1350 │   │   # self.start_dir:  Position of start of central directory                         │
│   1351 │   │   self.start_dir = offset_cd + concat                                               │
│   1352 │   │   if self.start_dir < 0:                                                            │
│ ❱ 1353 │   │   │   raise BadZipFile("Bad offset for central directory")                          │
│   1354 │   │   fp.seek(self.start_dir, 0)                                                        │
│   1355 │   │   data = fp.read(size_cd)                                                           │
│   1356 │   │   fp = io.BytesIO(data)                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
BadZipFile: Bad offset for central directory                                                                            
Extracting data files:  35%|█████████████████▊                                 | 38572/110812 [00:10<00:20, 3576.26it/s]
```



### Steps to reproduce the bug

```
load_dataset(
    args.dataset_name,
    args.dataset_config_name,
    cache_dir=args.cache_dir,
),
```

### Expected behavior

loads the dataset

### Environment info

datasets==2.8.0
Python 3.10.8
Linux 129-146-3-202 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Comments:
Hi ! Could you share the full stack trace ? Which dataset did you try to load ?

it may be related to https://github.com/huggingface/datasets/pull/5640

The `BadZipFile` error means the ZIP file is corrupted, so I'm closing this issue as it's not directly related to `datasets`.

For others that find this issue following a `BadZipFile` error, I had the same problem because I had a file in a folder dataset `my-image.target` and the datasets library was incorrectly determining that the (PNG) file was a zip archive. When it tried to extract the file, this error occurred. 

Updating to `datasets==2.12.0` fixed the problem for me.
test v0.12.0.rc0
Opened by Wauplin (closed)

DO NOT MERGE.
Only to test the CI.

cc @lhoestq @albertvillanova 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@Wauplin I was testing it in a dedicated branch without opening a PR: https://github.com/huggingface/datasets/commits/test-hfh-0.12.0rc0

Oops, sorry @albertvillanova. I thought for next time I'll start the CIs before pinging everyone.
I'm closing this one.

@Wauplin in your Slack message, you asked people from every major dependent library to check that our CI work. That is why I am checking it... :)

Also, I think for this purpose it is better to test it in a dedicated branch, rather than opening and closing a PR.

Yes, yes I know. Completely my fault on this one
CI tests are broken: AttributeError: 'mappingproxy' object has no attribute 'target'
Opened by albertvillanova (closed)

CI tests are broken, raising `AttributeError: 'mappingproxy' object has no attribute 'target'`. See: https://github.com/huggingface/datasets/actions/runs/3966497597/jobs/6797384185
```
...
ERROR tests/test_streaming_download_manager.py::TestxPath::test_xpath_rglob[mock://top_level-date=2019-10-0[1-4]/*-expected_paths4] - AttributeError: 'mappingproxy' object has no attribute 'target'
===== 2076 passed, 19 skipped, 15 warnings, 47 errors in 115.54s (0:01:55) =====
```

Comments: