SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
I understand the argument that it is bad to have a breaking change. How to deal with the introduction of breaking changes is a topic of its own and not sure how you want to deal with that (or is the policy this is never allowed, and there must be a `load_dataset_v2` or so if you really want to introduce a breaking change?).

Regardless of whether it is a breaking change, however, I don't see the other arguments.

> but **really** bad for Tensorflow and Flax where we **currently cannot** even use `datasets` to load `.mp3` files

I don't exactly understand this. Why not?

Why does the HF dataset on-the-fly decoding mechanism not work? Why is it anyway specific to PyTorch or TensorFlow? Isn't this independent?

But even if you just provide the raw bytes to TF, on TF you could just use sth like `tfio.audio.decode_mp3` or `tf.audio.decode_ogg` or `tfio.audio.decode_flac`?

> There are the following reasons why it is paramount to have access to the original audio file in my opinion  ...

I don't really understand the arguments (despite that it maybe breaks existing code). You anyway have the original audio files but it is just embedded in the dataset? I don't really know about any library which cannot also load the audio from memory (i.e. from the dataset).

Btw, on librosa being slow for decoding audio files, I saw that as well, so we have this comment RETURNN:

> Don't use librosa.load which internally uses audioread which would use Gstreamer as a backend which has multiple issues:
> https://github.com/beetbox/audioread/issues/62
> https://github.com/beetbox/audioread/issues/63
> Instead, use PySoundFile (soundfile), which is also faster. See here for discussions:
> https://github.com/beetbox/audioread/issues/64
> https://github.com/librosa/librosa/issues/681

Resampling is also a separate aspect, which is also less straightforward and with different compromises between speed and quality. So there the different tradeoffs and different implementations can make a difference.

However, I don't see how this is related to the question whether there should be the raw bytes inside the dataset or as separate local files.
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
> The problem with decoding on the fly is that we currently rely on `torchaudio` for this now which relies on `torch` which is not necessarily something people would like to install when using `tensorflow` or `flax`. Therefore we cannot just rely on people using the decoding on the fly method. We just didn't find a library that is ML framework independent and fast enough for all formats. `torchaudio` is currently in our opinion by far the best here.

But how is this relevant for this issue here? I thought this issue here is about having the (correct) path in the dataset or having raw bytes in the dataset.

How did TF users use it at all then? Or they just do not use on-the-fly decoding? I did not even notice this problem (maybe because I had `torchaudio` installed). But what do they use instead?

But as I outlined before, they could just use `tfio.audio.decode_flac` and co, where it would be more natural if you already provide the raw bytes.

> Looking at https://pytorch.org/audio/stable/torchaudio.html?highlight=load#torchaudio.load one would not see directly how to load raw bytes

I was not really familiar with `torchaudio`. It seems that they really don't provide an easy/direct API to operate on raw bytes. Which is very strange and unfortunate because as far as I can see, all the underlying backend libraries (e.g. soundfile) easily allow that. So I would say that this is the fault of `torchaudio` then. But despite, if you anyway use `torchaudio` with `soundfile` backend, why not just use `soundfile` directly. It's very simple to use and crossplatform.

But ok, now we are just discussing how to handle the on-the-fly decoding. I still think this is a separate issue and having raw bytes in the dataset instead of local files should just be fine as well.


> It is a bit scary IMO to see raw bytes for users.

I think nobody who writes code is scared by seeing the raw bytes content of a binary file. :)


> I guess it's still very much possible for the user to do this:
> 
> ```python
> def save_as_bytes:
>       batch["bytes"] = read_in_bytes_from_file(batch["file"])\
>       os.remove(batch["file"])
> 
> ds = ds.map(save_as_bytes)
> 
> ds.save_to_disk(...)
> ```

In https://github.com/huggingface/datasets/pull/4184#issuecomment-1105191639, you said/proposed that this `map` is not needed anymore and `save_to_disk` could do it automatically (maybe via some option)?

> Guess the question is more a bit about what should be the default case?

Yea this is up to you. I'm happy as long as we can get it the way we want easily and this is a well supported use case. :)
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
> In https://github.com/huggingface/datasets/pull/4184#issuecomment-1105191639, you said/proposed that this map is not needed anymore and save_to_disk could do it automatically (maybe via some option)?

Yes! Should be super easy now see discussion here: https://github.com/rwth-i6/i6_core/issues/257#issuecomment-1105494468

Thanks for the super useful input :-)
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
Despite the comments that this has been fixed, I am finding the exact same problem is occurring again (with datasets version 2.3.2)
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
> Despite the comments that this has been fixed, I am finding the exact same problem is occurring again (with datasets version 2.3.2)

It appears downgrading to torchaudio 0.11.0 fixed this problem.
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
@patrickvonplaten @lhoestq @polinaeterna I was unable to load audio from Common Voice using 🤗 with the current version of torchaudio, but downgrading to torchaudio 0.11.0 fixed it. This is probably more of a torch problem than a Hugging Face problem.
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
@DCNemesis that's interesting, could you please share the error message if you still can access it?
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
@polinaeterna I believe it is the same exact error as above. It occurs on other .mp3 sources as well, but the problem is with torchaudio > 0.11.0. I've created a short colab notebook that reproduces the error, and the fix here: https://colab.research.google.com/drive/18wsuwdHwBPN3JkcnhEtk8MUYqF9swuWZ?usp=sharing
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
Hi @DCNemesis,

Your issue was slightly different from the original one in this issue page. Yours seems related to a change in the backend used by `torchaudio` (`ffmpeg` instead of `sox`). Refer to the issue page here:
- #4776

Normally, it should be circumvented with the patch made by @polinaeterna in:
- #4923
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
I think the original issue reported here was already fixed by:
- #3736

Otherwise, feel free to reopen.
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
Thanks @lhoestq for finding the reason of incorrect resampling. This issue affects all languages which have sound files with different sampling rates such as Turkish and Luganda.
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
@cahya-wirawan - do you know how many languages have different sampling rates in Common Voice? I'm quite surprised to see this for multiple languages actually
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
@cahya-wirawan, I can reproduce the problem for Common Voice 7 for Turkish. Here a script you can use:


```python
#!/usr/bin/env python3
from datasets import load_dataset
import torchaudio
from io import BytesIO
from datasets import Audio
from collections import Counter
import sys

ds_name = str(sys.argv[1])
lang = str(sys.argv[2])

ds = load_dataset(ds_name, lang, split="train", use_auth_token=True)
ds = ds.cast_column("audio", Audio(decode=False))

all_sampling_rates = []


def print_sampling_rate(x):
    x, sr = torchaudio.load(BytesIO(x["audio"]["bytes"]), format="mp3")
    all_sampling_rates.append(sr)

ds.map(print_sampling_rate)


print(Counter(all_sampling_rates))
```

can be run with:

```bash
python run.py mozilla-foundation/common_voice_7_0 tr
```

For CV 6.1 all samples seem to have the same audio
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
It actually shows that many more samples are in 32kHz format than it 48kHz which is unexpected. Thanks a lot for flagging! Will contact Common Voice about this as well
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
I only checked the CV 7.0 for Turkish, Luganda and Indonesian, they have audio files with difference sampling rates, and all of them are affected by this issue. Percentage of incorrect resampling as follow, Turkish: 9.1%, Luganda: 88.2% and Indonesian: 64.1%.
I checked it using the original CV files. I check the original sampling rates and the length of audio array of each files and compare it with the length of audio array (and the sampling rate which is always 48kHz) from mozilla-foundation/common_voice_7_0 datasets. if the length of audio array from dataset is not equal to 48kHz/original sampling rate * length of audio array of the original audio file then it is affected,
Issue: [Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Comment:
Ok wow, thanks a lot for checking this - you've found a pretty big bug :sweat_smile: It seems like **a lot** more datasets are actually affected than I original thought. We'll try to solve this as soon as possible and make an announcement tomorrow.
Issue: push_to_hub but preview not working

Comment:
Hi @thomas-happify, please note that the preview may take some time before rendering the data.

I've seen it is already working.

I close this issue. Please feel free to reopen it if the problem arises again.
Issue: Dataset viewer issue for *P3*

Comment:
The error is now:

```
Status code:   400
Exception:     Status400Error
Message:       this dataset is not supported for now.
```

We've disabled the dataset viewer for several big datasets like this one. We hope being able to reenable it soon.
Issue: checksum error subjqa dataset

Comment:
Hi @RensDimmendaal, 

I'm sorry but I can't reproduce your bug:
```python
In [1]: from datasets import load_dataset
   ...: ds = load_dataset("subjqa", "electronics")
Downloading builder script: 9.15kB [00:00, 4.10MB/s]                                                                                                                                                       
Downloading metadata: 17.7kB [00:00, 8.51MB/s]                                                                                                                                                             
Downloading and preparing dataset subjqa/electronics (download: 10.86 MiB, generated: 3.01 MiB, post-processed: Unknown size, total: 13.86 MiB) to .../.cache/huggingface/datasets/subjqa/electronics/1.1.0/e5588f9298ff2d70686a00cc377e4bdccf4e32287459e3c6baf2dc5ab57fe7fd...
Downloading data: 11.4MB [00:03, 3.50MB/s]
Dataset subjqa downloaded and prepared to .../.cache/huggingface/datasets/subjqa/electronics/1.1.0/e5588f9298ff2d70686a00cc377e4bdccf4e32287459e3c6baf2dc5ab57fe7fd. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 605.09it/s]

In [2]: ds
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 1295
    })
    test: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 358
    })
    validation: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 255
    })
})
```

Could you please try again and see if the problem persists?

If that is the case, you can circumvent the issue by passing `ignore_verifications`:
```python
ds = load_dataset("subjqa", "electronics", ignore_verifications=True)
Issue: checksum error subjqa dataset

Comment:
Thanks checking!

You're totally right. I don't know what's changed, but I'm glad it's working now!
Issue: Pubmed dataset not reachable

Comment:
Hey @albertvillanova, sorry to reopen this... I can confirm that on `master` branch the dataset is downloadable now but it is still broken in streaming mode:

```python
 >>> import datasets
 >>> pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
 >>> next(iter(pubmed_train))
```
```
 No such file or directory: 'gzip://pubmed22n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n0001.xml.gz'
```
Issue: Pubmed dataset not reachable

Comment:
Hi @abhi-mosaic, would you mind opening another issue for this new problem?

First issue (already solved) was a ConnectionError due to the yearly update release of PubMed: we fixed it by updating the URLs from year 2021 to year 2022.

However this is another problem: to make pubmed streamable. Please note that NOT all our datastes are streamable: we are making streamable more and more of them... but this is an on-going process...

Thanks.
Issue: Pubmed dataset not reachable

Comment:
@albertvillanova 
When I tried below codes, I got the similar error

```

dataset=load_dataset("pubmed",split="train")

Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0601.xml.gz
```
Issue: Add a GROUP BY operator

Comment:
Hi ! At the moment you can use `to_pandas()` to get a pandas DataFrame that supports `group_by` operations (make sure your dataset fits in memory though)

We use Arrow as a back-end for `datasets` and it doesn't have native group by (see https://github.com/apache/arrow/issues/2189) unfortunately.

I just drafted what it could look like to have `group_by` in `datasets`:
```python
from datasets import concatenate_datasets

def group_by(d, col, join): 
    """from: https://github.com/huggingface/datasets/issues/3644"""
    # Get the indices of each group
    groups = {key: [] for key in d.unique(col)} 
    def create_groups_indices(key, i): 
        groups[key].append(i) 
    d.map(create_groups_indices, with_indices=True, input_columns=col) 
    # Get one dataset object per group
    groups = {key: d.select(indices) for key, indices in groups.items()} 
    # Apply join function
    groups = {
        key: dataset_group.map(join, batched=True, batch_size=len(dataset_group), remove_columns=d.column_names)
        for key, dataset_group in groups.items()
    } 
    # Return concatenation of all the joined groups
    return concatenate_datasets(groups.values())
```

example of usage:
```python

def join(batch): 
    # take the batch of all the examples of a group, and return a batch with one aggregated example
    # (we could aggregate examples into several rows instead of one, if you want)
    return {"total": [batch["i"]]} 

d = Dataset.from_dict({
    "i": [i for i in range(50)],
    "group_key": [i % 4 for i in range(50)],
})
print(group_by(d, "group_key", join))
#                                                total
# 0  [0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48]
# 1  [1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, 49]
# 2     [2, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46]
# 3     [3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47]
```

Let me know if that helps !

cc @albertvillanova @mariosasko for visibility
Issue: Add a GROUP BY operator

Comment:
@lhoestq As of PyArrow 7.0.0, `pa.Table` has the [`group_by` method](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.group_by), so we should also consider using that function for grouping.
Issue: Add a GROUP BY operator

Comment:
Hi, I have a similar issue as OP but the suggested solutions do not work for my case. Basically, I process documents through a model to extract the last_hidden_state, using the "map" method on a Dataset object, but would like to average the result over a categorical column at the end (i.e. groupby this column).
- A to_pandas() saturates the memory, although it gives me the desired result through a .groupby().apply(np.mean, axis=0) on a smaller use-case,
- The solution posted on Feb 4 is much too slow,
- datasets_sql seems to not like the fact that I'm averaging np.arrays.
So I'm kinda out of "non brute force" options... Any help appreciated
Issue: Add a GROUP BY operator

Comment:
> Hi, I have a similar issue as OP but the suggested solutions do not work for my case. Basically, I process documents through a model to extract the last_hidden_state, using the "map" method on a Dataset object, but would like to average the result over a categorical column at the end (i.e. groupby this column).
 
If you haven't yet, you could explore using [Polars](https://www.pola.rs/) for this. It's a new DataFrame library written in Rust with Python bindings. It is Pandas like it in many ways ,but does have some biggish differences in syntax/approach so it's definitely not a drop-in replacement. 

Polar's also uses Arrow as a backend but also supports out-of-memory operations; in this case, it's probably easiest to write out your dataset to parquet and then use the polar's `scan_parquet` method (this will lazily read from the parquet file). The thing you get back from that is a `LazyDataFrame` i.e. nothing is loaded into memory until you specify a query and call a  `collect` method. 

Example below of doing a groupby on a dataset which definitely wouldn't fit into memory on my machine:

```
from datasets import load_dataset
import polars as pl

ds = load_dataset("blbooks")
ds['train'].to_parquet("test.parquet")
df = pl.scan_parquet("test.parquet")
df.groupby('date').agg([pl.count()]).collect()
```

>datasets_sql seems to not like the fact that I'm averaging np.arrays.

I am not certain how Polars will handle this either. It does have NumPy support (https://pola-rs.github.io/polars-book/user-guide/howcani/interop/numpy.html) but I assume Polars will need to have at least enough memory in each group you want to average over so you may still end up needing more memory depending on the size of your dataset/groups.
Issue: Add a GROUP BY operator

Comment:
Hi @davanstrien , thanks a lot, I didn't know about this library and the answer works! I need to try it on the full dataset now, but I'm hopeful. Here's what my code looks like:
```
list_size = 768
df.groupby("date").agg(
    pl.concat_list(
        [
            pl.col("hidden_state")
            .arr.slice(n, 1)
            .arr.first()
            .mean()
            for n in range(0, list_size)
        ]
    ).collect()
```

For some reasons, the following code was giving me a "mean() got unexpected argument 'axis'":
```
df2 = df.groupby('date').agg(
    pl.col("hidden_state").map(np.mean).alias("average_hidden_state")
).collect()

```

EDIT: The solution works on my large dataset, the memory does not crash and the time is reasonable, thanks a lot again!
Issue: Add a GROUP BY operator

Comment:
I find this functionality missing in my workflow as well and the workarounds with SQL and Polars unsatisfying. Since PyArrow has exposed this functionality, I hope this soon makes it into a release. (:
Issue: AutoTokenizer hash value got change after datasets.map

Comment:
This issue was original reported at https://github.com/huggingface/transformers/issues/14931 and It seems like this issue also occur with other AutoClass like AutoFeatureExtractor.
Issue: AutoTokenizer hash value got change after datasets.map

Comment:
Thanks for moving the issue here !

I wasn't able to reproduce the issue on my env (the hashes stay the same):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 6.0.1
```
However I was able to reproduce it on Google Colab (the hashes end up different):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
```
I'll investigate why it doesn't work properly on Google Colab :)
Issue: AutoTokenizer hash value got change after datasets.map

Comment:
I found the issue: the tokenizer has something inside it that changes.

Before the call, `tokenizer._tokenizer.truncation` is None, and after the call it changes to this for some reason:
```
{'max_length': 512, 'strategy': 'longest_first', 'stride': 0}
```

Does anybody know why calling the tokenizer would change its state this way ? cc @Narsil @SaulLu maybe ?
Issue: AutoTokenizer hash value got change after datasets.map

Comment:
`tokenizer.encode(..)` does not accept argument like max_length, strategy or stride.

In `tokenizers` you have to modify the tokenizer state by setting various `TruncationParams` (and/or `PaddingParams`).
However, since this is modifying the state, you need to mutably borrow the tokenizer (a rust concept). The key principle is that there can ever be only 1 mutable borrow at a time during the span of the tokenizer lifecycle.

Because of this, if `transformers` blindly set  `TruncationParams` and `PaddingParams` on every call, it would cause the tokenizer to crash (or make the various threads accessing it hang, which is not necessarily better).

In order to avoid that, we decided to handle it this way : https://github.com/huggingface/transformers/pull/12550 . 

Which should explain the state of the tokenizer being modified (hence its hash).

Now for a temporary solution, simply encoding once with the tokenizer should give it it's proper hash (since by default the tokenizer doesn't have this state, looks at the first encoding call, and creates it).

We could try and set these 2 dicts at initialization time, but it wouldn't work if a user modified the tokenizer state later
```python
tokenizer = AutoTokenizer.from_pretrained(..)
tokenizer.truncation_side = "left"
# Now we have a difference between `tokenizer._tokenizer.truncation` and `tokenizer.truncation_side`
```
If we wanted to fix it correctly it would mean mapping every assignation to it's proper location on `tokenizer.{padding/truncation}`

I think it's important to note that we cannot guarantee a tokenizer' hash remains the same if *any* of those parameters are modified through the `.map` function.

Edit: Another option would be to override the default __hash__ function, but I don't know if there's  a sound implementation that could fit.