SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english

Comment:
Thanks for reporting. It's a known issue that should be fixed soon. Meanwhile, I had to manually trigger the dataset viewer. It's OK now.
Note that the extreme aspect ratio of the images generates another issue, that we're inspecting.
Issue: Add Multiface dataset

Comment:
Hey @nandwalritik! Thanks for offering to help!

This dataset might be somewhat complex and I'm concerned about it being 65 TB, which would be quite expensive to host. @lhoestq @mariosasko I would love your input if you think it's worth adding this dataset.
Issue: Add Multiface dataset

Comment:
Thanks for proposing this interesting dataset, @osanseviero.

Please note that the data files are already hosted in a third-party server: e.g. the index of data files for entity "6795937" is at https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/index.html 
- audio.tar: https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/audio.tar
- ...

Therefore, in principle, we don't need to host them on our Hub: it would be enough to just implement a loading script in the corresponding Hub dataset repo, e.g. "facebook/multiface"...
Issue: pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

Comment:
Thanks for reporting @conceptofmind.

Could you please give details about your environment? 
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```
Issue: pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

Comment:
Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico
Issue: pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

Comment:
I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code
Issue: pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

Comment:
Hi @loubnabnl,

Yes, the issue is solved in the discussion thread.

I will close this issue.

Thank you again for all of your help.

Enrico
Issue: RuntimeError when using torchaudio 0.12.0 to load MP3 audio file

Comment:
Requiring torchaudio<0.12.0 isn't really a viable solution because that implies torch<0.12.0 which means no sm_86 CUDA support which means no RTX 3090 support in PyTorch.

But in my case, the error only occurs if `_fallback_load` resolves to `_fail_load` inside torchaudio 0.12.0 which is only the case if FFMPEG initialization failed:  https://github.com/pytorch/audio/blob/b1f510fa5681e92ee82bdc6b2d1ed896799fc32c/torchaudio/backend/sox_io_backend.py#L36-L47

That means the proper solution for torchaudio>=0.12.0 is to check `torchaudio._extension._FFMPEG_INITIALIZED` and if it is False, then we need to remind the user to install a dynamically linked ffmpeg 4.1.8 and then maybe call `torchaudio._extension._init_ffmpeg()` to force a user-visible exception showing the missing ffmpeg dynamic library name.

On my system, installing 

- libavcodec.so.58 
- libavdevice.so.58  
- libavfilter.so.7  
- libavformat.so.58  
- libavutil.so.56  
- libswresample.so.3  
- libswscale.so.5

from ffmpeg 4.1.8 made HF datasets 2.3.2 work just fine with  torchaudio 0.12.1+cu116:

```python3
import sox, torchaudio, datasets
print('torchaudio', torchaudio.__version__)
print('datasets', datasets.__version__)
torchaudio._extension._init_ffmpeg()
print(torchaudio._extension._FFMPEG_INITIALIZED)
waveform, sample_rate = torchaudio.load('/workspace/.cache/huggingface/datasets/downloads/extracted/8e5aa88585efa2a4c74c6664b576550d32b7ff9c3d1d17cc04f44f11338c3dc6/cv-corpus-8.0-2022-01-19/en/clips/common_voice_en_100038.mp3', format='mp3')
print(waveform.shape)
```

```
torchaudio 0.12.1+cu116
datasets 2.3.2
True
torch.Size([1, 369792])
```
Issue: Streaming not supported in Theivaprakasham/wildreceipt

Comment:
Thanks for reporting @NitishkKarra.

The root source of the issue is that streaming mode is not supported out-of-the-box for that dataset, because it contains a TAR file.

We have opened a discussion in the corresponding Hub dataset page, pointing out this issue: https://huggingface.co/datasets/Theivaprakasham/wildreceipt/discussions/1

I'm closing this issue here, so this discussion is transferred there instead.
Issue: AssertionError when using label_cols in to_tf_dataset 

Comment:
Hi @lehrig, this is caused by the data collator renaming "label" to "labels". If you set `label_cols=["labels"]` in the call it will work correctly. However, I agree that the cause of the bug is not obvious, so I'll see if I can make a PR to clarify things when the collator renames columns.
Issue: AssertionError when using label_cols in to_tf_dataset 

Comment:
Thanks - and wow, that appears like a strange side-effect of the data collator. Is that really needed?

Why not make it more explicit? For example, extend `DefaultDataCollator` with an optional property `label_col_name` to be used as label column; only when it is not provided default to `labels` (and document that this happens) for backwards-compatibility?
Issue: AssertionError when using label_cols in to_tf_dataset 

Comment:
Haha, I honestly have no idea why our data collators rename `"label"` (the standard label column name in our datasets) to `"labels"` (the standard label column name input to our models). It's been a pain point when I design TF data pipelines, though, because I don't want to hardcode things like that - especially in `datasets`, because the renaming is something that happens purely at the `transformers` end. I don't think I could make the change in the data collators themselves at this point, because it would break backward compatibility for everything in PyTorch as well as TF.

In the most recent version of `transformers` we added a [prepare_tf_dataset](https://huggingface.co/docs/transformers/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset) method to our models which takes care of these details for you, and even chooses appropriate columns and labels for the model you're using. In future we might make that the officially recommended way to convert HF datasets to `tf.data.Dataset`.
Issue: AssertionError when using label_cols in to_tf_dataset 

Comment:
Interesting, that'd be great especially for clarity. https://huggingface.co/docs/datasets/use_with_tensorflow#data-loading already improved clarity, yet, all those options will still confuse people. Looking forward to those advances in the hope there'll be only 1 way in the future ;)

Anyways, I am happy for the time being with the work-around you provided. Thank you!
Issue: Dataset Viewer issue for openclimatefix/goes-mrms

Comment:
Thanks for reporting, @cheaterHy.

The cause of this issue is a misalignment between the names of the repo (`goes-mrms`, with hyphen) and its Python loading scrip file (`goes_mrms.py`, with underscore).

I've opened an Issue discussion in their repo: https://huggingface.co/datasets/openclimatefix/goes-mrms/discussions/1
Issue: parallel searching in multi-gpu setting using faiss

Comment:
And I don't see any speed up when increasing the number of GPUs while calling `get_nearest_examples_batch`.
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Hi ! Yes search_batch uses FAISS search which happens in parallel across the GPUs

> And I don't see any speed up when increasing the number of GPUs while calling get_nearest_examples_batch.

That's unexpected, can you share the code you're running ?
Issue: parallel searching in multi-gpu setting using faiss

Comment:
here is the code snippet

```python

# add faiss index
source_dataset = load_dataset(source_path)
queries = load_dataset(query_path)
gpu = [0,1,2,3]
source_dataset.add_faiss_index(
                "embedding",
                device=gpu,
            )


# batch query
batch_size = 32
for i in tqdm(range(0, len(queries), batch_size)):
    if i + batch_size >= len(queries):
        batched_queries = queries[i:]
    else:
        batched_queries = queries[i:i+batch_size]

    batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
    scores, candidates = source_dataset.get_nearest_examples_batch(
        "embedding",
        batched_query_embeddings,
        k=5
    )
```
Issue: parallel searching in multi-gpu setting using faiss

Comment:
The code looks all good to me, do you see all the GPUs being utilized ? What version of faiss are you using ?
Issue: parallel searching in multi-gpu setting using faiss

Comment:
I can see the memory usage of all the GPUs.
My version of `faiss-gpu` is `1.7.2`
Issue: parallel searching in multi-gpu setting using faiss

Comment:
It looks all good to me then ^^ though you said you didn't experienced speed improvements by adding more GPUs ? What size is your source dataset and what time differences did you experience ?
Issue: parallel searching in multi-gpu setting using faiss

Comment:
query set: 1e6
source dataset: 1e6
embedding size: 768
index: Flat
topk: 20
GPU: V100

The time taken to traverse the query set once is about 1.5h, which is almost not influenced by the value of query batch size or the number of GPUs according to my experiments.
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Hmmm the number of GPUs should divide the time, something is going wrong. Can you check that adding more GPU does divide the memory used per GPU ? Maybe it can be worth looking at similar issues in the FAISS repository or create a noew issue over there to understand what's going on
Issue: parallel searching in multi-gpu setting using faiss

Comment:
> Can you check that adding more GPU does divide the memory used per GPU 

The memory used per GPU is unchanged while adding more GPU. Is this unexpected?

I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆
Issue: parallel searching in multi-gpu setting using faiss

Comment:
> I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆

Oh indeed that's possible, I wasn't sure. Anyway you can check that calling get_nearest_examples_batch simply calls search under the hood: 

https://github.com/huggingface/datasets/blob/f90f71fbbb33889fe75a3ffc101cdf16a88a3453/src/datasets/search.py#L375
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Here is a runnable script. 
Multi-GPU searching still does not work in my experiments.


```python
import os
from tqdm import tqdm
import numpy as np
import datasets
from datasets import Dataset

class DPRSelector:

    def __init__(self, source, target, index_name, gpu=None):
        self.source = source
        self.target = target
        self.index_name = index_name

        cache_path = 'embedding.faiss'

        if not os.path.exists(cache_path):
            self.source.add_faiss_index(
                column="embedding",
                index_name=index_name,
                device=gpu,
            )
            self.source.save_faiss_index(index_name, cache_path)
        else:
            self.source.load_faiss_index(
                index_name,
                cache_path,
                device=gpu
            )
        print('index builded!')

    def build_dataset(self, top_k, batch_size):
        print('start search')

        for i in tqdm(range(0, len(self.target), batch_size)):
            if i + batch_size >= len(self.target):
                batched_queries = self.target[i:]
            else:
                batched_queries = self.target[i:i+batch_size]


            batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
            search_res = self.source.get_nearest_examples_batch(
                self.index_name,
                batched_query_embeddings,
                k=top_k
            )
      
        print('finish search')


def get_pseudo_dataset():
    pseudo_dict = {"embedding": np.zeros((1000000, 768), dtype=np.float32)}
    print('generate pseudo data')

    dataset = Dataset.from_dict(pseudo_dict)
    def list_to_array(data):
        return {"embedding": [np.array(vector, dtype=np.float32) for vector in data["embedding"]]} 
    dataset.set_transform(list_to_array, columns='embedding', output_all_columns=True)

    print('build dataset')
    return dataset



if __name__=="__main__":

    np.random.seed(42)


    source_dataset = get_pseudo_dataset()
    target_dataset = get_pseudo_dataset()

    gpu = [0,1,2,3,4,5,6,7]
    selector = DPRSelector(source_dataset, target_dataset, "embedding",  gpu=gpu)

    selector.build_dataset(top_k=20, batch_size=32)
```
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Maybe @albertvillanova you can take a look ? I won't be available in the following days
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Hi @xwwwwww I'm investigating it, but I'm not an expert in Faiss. In principle, it is weird that your code does not work properly because it seems right...
Issue: parallel searching in multi-gpu setting using faiss

Comment:
> Have you tried passing `gpu=-1` and check if there is a speedup?

yes, there is a speed up using GPU compared with CPU.
Issue: parallel searching in multi-gpu setting using faiss

Comment:
When passing `device=-1`, ALL existing GPUs are used (multi GPU): this is the maximum speedup you can get. To know the number of total GPUs:
```
import faiss

ngpus = faiss.get_num_gpus()
print(ngpus)
```

When passing a list of integers to `device`, then only that number of GPUs are used (multi GPU as well)
- the speedup should be proportional (more or less) to the ratio of the number of elements passed to `device` over `ngpus`
- if this is not the case, then there is an issue in the implementation of this use case (however, I have reviewed the code and in principle I can't find any evident bug)

When passing a positive integer to `device`, then only a single GPU is used.
- this time should be more or less proportional to the time when passing `device=-1` over `ngpus`
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Thanks for your help!
Have you run the code and replicated the same experimental results (i.e., no speedup while increasing the number of GPUs)?
Issue: parallel searching in multi-gpu setting using faiss

Comment:
I can confirm `add_faiss_index` calls `index = faiss.index_cpu_to_gpus_list(index, gpus=list(device))`.

Could this be an issue with your environment ? Could you try running with 1 and 8 GPUs with a code similar to[ this one from the FAISS examples](https://github.com/facebookresearch/faiss/blob/main/tutorial/python/5-Multiple-GPUs.py) but using `gpu_index = faiss.index_cpu_to_gpus_list(cpu_index, gpus=list(device))`, and see if the speed changes ?
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Hi, I test the FAISS example and the speed indeed changes.  I set `nb=1000000`, `nq=1000000` and `d=64`

| num GPUS | time cost |
| -------- | --------- |
| 1        | 28.53     |
| 5        | 7.16      |
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Ok the benchmark is great, not sure why it doesn't speed up the index in your case though. You can try running the benchmark with the same settings as your actual dataset
```
query set: 1e6
source dataset: 1e6
embedding size: 768
index: Flat
topk: 20
GPU: V100
```

Note that you can still pass a FAISS index you built yourself to a dataset using https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index_from_external_arrays
Issue: parallel searching in multi-gpu setting using faiss

Comment:
> Here is a runnable script. Multi-GPU searching still does not work in my experiments.
> 
> ```python
> import os
> from tqdm import tqdm
> import numpy as np
> import datasets
> from datasets import Dataset
> 
> class DPRSelector:
> 
>     def __init__(self, source, target, index_name, gpu=None):
>         self.source = source
>         self.target = target
>         self.index_name = index_name
> 
>         cache_path = 'embedding.faiss'
> 
>         if not os.path.exists(cache_path):
>             self.source.add_faiss_index(
>                 column="embedding",
>                 index_name=index_name,
>                 device=gpu,
>             )
>             self.source.save_faiss_index(index_name, cache_path)
>         else:
>             self.source.load_faiss_index(
>                 index_name,
>                 cache_path,
>                 device=gpu
>             )
>         print('index builded!')
> 
>     def build_dataset(self, top_k, batch_size):
>         print('start search')
> 
>         for i in tqdm(range(0, len(self.target), batch_size)):
>             if i + batch_size >= len(self.target):
>                 batched_queries = self.target[i:]
>             else:
>                 batched_queries = self.target[i:i+batch_size]
> 
> 
>             batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
>             search_res = self.source.get_nearest_examples_batch(
>                 self.index_name,
>                 batched_query_embeddings,
>                 k=top_k
>             )
>       
>         print('finish search')
> 
> 
> def get_pseudo_dataset():
>     pseudo_dict = {"embedding": np.zeros((1000000, 768), dtype=np.float32)}
>     print('generate pseudo data')
> 
>     dataset = Dataset.from_dict(pseudo_dict)
>     def list_to_array(data):
>         return {"embedding": [np.array(vector, dtype=np.float32) for vector in data["embedding"]]} 
>     dataset.set_transform(list_to_array, columns='embedding', output_all_columns=True)
> 
>     print('build dataset')
>     return dataset
> 
> 
> 
> if __name__=="__main__":
> 
>     np.random.seed(42)
> 
> 
>     source_dataset = get_pseudo_dataset()
>     target_dataset = get_pseudo_dataset()
> 
>     gpu = [0,1,2,3,4,5,6,7]
>     selector = DPRSelector(source_dataset, target_dataset, "embedding",  gpu=gpu)
> 
>     selector.build_dataset(top_k=20, batch_size=32)
> ```

By the way,  have you run this toy example and replicated my experiment results? I think it is a more direct way to figure this out :)
Issue: parallel searching in multi-gpu setting using faiss

Comment:
Hi,

I have a similar question and would like to know if there's any progress in this issue. 

`dataset.add_faiss_index(column="embedding")`, this takes around 5minutes to add the index.

`dataset.add_faiss_index(column="embedding", device=-1)`, this ran for more than 10minutes and still didn't complete execution. 

Now, I don't understand why that's the case as I expected for GPU the indexing should be faster
Issue: Issue with offline mode

Comment:
Hi @SaulLu, thanks for reporting.

I think offline mode is not supported for datasets containing only data files (without any loading script). I'm having a look into this...
Issue: Issue with offline mode

Comment:
Thanks for your feedback! 

To give you a little more info, if you don't set the offline mode flag, the script will load the cache. I first noticed this behavior with the `evaluate` library, and while trying to understand the downloading flow I realized that I had a similar error with datasets.
Issue: Issue with offline mode

Comment:
#5331 will be helpful to fix this, as it updates the cache directory template to be aligned with the other datasets
Issue: Dataset Viewer issue for Toygar/turkish-offensive-language-detection

Comment:
I refreshed the dataset viewer manually, it's fixed now. Sorry for the inconvenience.
<img width="1557" alt="Capture d’écran 2022-07-28 à 09 17 39" src="https://user-images.githubusercontent.com/1676121/181514666-92d7f8e1-ddc1-4769-84f3-f1edfdb902e8.png">
Issue: Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size

Comment:
I've built a minimal example that shows this bug without `n_proc`. It seems like it's a problem any way of using **tokenizers, `overflow_to_sample_mapping`, and Dataset.map, with a small batch size**:

```
import datasets
import transformers
pretrained = 'deepset/tinyroberta-squad2'
tokenizer = transformers.AutoTokenizer.from_pretrained(pretrained)

questions = ['Can you tell me why?', 'What time is it?']
contexts = ['This is context zero', 'Another paragraph goes here']   

def tok(questions, contexts):
    return tokenizer(text=questions,
            text_pair=contexts,
            truncation='only_second',
            return_overflowing_tokens=True,
            )
print(tok(questions, contexts)['overflow_to_sample_mapping'])
assert tok(questions, contexts)['overflow_to_sample_mapping'] == [0, 1] # PASSES

def tok2(d):
    return tok(d['question'], d['context'])

def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=1)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # FAILS produces [0,0]
```

Note that even if the batch size would be larger, there will be instances where we will not have a lot of data, and end up using small batches.  This can occur e.g. if `n_proc` causes batches to be underfill.  I imagine it can also occur in other ways, e.g. the final leftover batch at the end.
Issue: Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size

Comment:
A larger batch size does _not_ have this behavior:

```
def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=2)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # PASSES
```
Issue: DatasetInfo issue when testing multiple configs: mixed task_templates

Comment:
I've narrowed down the issue to the `dataset_module_factory` which already creates a `dataset_infos.json` file down in the `.cache/modules/dataset_modules/..` folder. That JSON file already contains the wrong task_templates for `unfiltered`.
Issue: DatasetInfo issue when testing multiple configs: mixed task_templates

Comment:
Ugh. Found the issue: apparently `datasets` was reusing the already existing `dataset_infos.json` that is inside `datasets/datasets/hebban-reviews`! Is this desired behavior?

Perhaps when `--save_infos` and `--all_configs` are given, an existing `dataset_infos.json` file should first be deleted before continuing with the test? Because that would assume that the user wants to create a new infos file for all configs anyway.
Issue: DatasetInfo issue when testing multiple configs: mixed task_templates

Comment:
Hi! I think this is a reasonable solution. Would you be interested in submitting a PR?
Issue: Easily create loading script for benchmark comprising multiple huggingface datasets

Comment:
Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467
Issue: Dataset Viewer issue for yanekyuk/wikikey

Comment:
The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets