jamesLeeeeeee/datasets-github-issues huggingface dataset:
For data-only datasets, streaming and non-streaming don't behave the same
Author: severo | Pull request: no | State: open

See https://huggingface.co/datasets/huggingface/transformers-metadata: it only contains two JSON files.

In streaming mode, the files are concatenated, and thus the rows might be dictionaries with different keys:

```python
import datasets as ds
iterable_dataset = ds.load_dataset("huggingface/transformers-metadata", split="train", streaming=True);
rows = list(iterable_dataset.take(100))
rows[0]
# {'model_type': 'albert', 'pytorch': True, 'tensorflow': True, 'flax': True, 'processor': 'AutoTokenizer'}
rows[99]
# {'model_class': 'BartModel', 'pipeline_tag': 'feature-extraction', 'auto_class': 'AutoModel'}
```

In normal mode, an exception is thrown:

```python
import datasets as ds
dataset = ds.load_dataset("huggingface/transformers-metadata", split="train");
```

```
ValueError: Couldn't cast
model_class: string
pipeline_tag: string
auto_class: string
to
{'model_type': Value(dtype='string', id=None), 'pytorch': Value(dtype='bool', id=None), 'tensorflow': Value(dtype='bool', id=None), 'flax': Value(dtype='bool', id=None), 'processor': Value(dtype='string', id=None)}
because column names don't match
```
Local paths in common voice
Author: lhoestq | Pull request: yes | State: closed

Continuation of https://github.com/huggingface/datasets/pull/3664:

- pass the `streaming` parameter to _split_generator
- update @anton-l's code to use this parameter for `common_voice`
- add a comment to explain why we use `download_and_extract` in non-streaming and `iter_archive` in streaming

Now the `common_voice` dataset has a local path back in `ds["path"]`, and this field is `None` in streaming mode.

cc @patrickvonplaten @anton-l @albertvillanova 

Fix #3663.
Performance of `datasets` at scale
Author: lvwerra | Pull request: no | State: open

# Performance of `datasets` at 1TB scale

## What is this?
During the processing of a large dataset I monitored the performance of the `datasets` library to see if there are any bottlenecks. The insights of this analysis could guide the decision making to improve the performance of the library.

## Dataset
The dataset is a 1.1TB extract from GitHub with 120M code files and is stored as 5000 `.json.gz` files. The goal of the preprocessing is to remove duplicates and filter files based on their stats. While the calculating of the hashes for deduplication and stats for filtering can be parallelized the filtering itself is run with a single process. After processing the files are pushed to the hub.

## Machine
The experiment was run on a `m1` machine on GCP with 96 CPU cores and 1.3TB RAM.

## Performance breakdown
- Loading the data **3.5h** (_30sec_ from cache)
    - **1h57min** single core loading (not sure what is going on here, corresponds to second progress bar)
    - **1h10min** multi core json reading
    - **20min** remaining time before and after the two main processes mentioned above
- Process the data **2h** (_20min_ from cache)
    - **20min** Getting reading for processing 
    - **40min** Hashing and files stats (96 workers)
    - **58min** Deduplication filtering (single worker)
- Save parquet files **5h**
    -  Saving 1000 parquet files (16 workers)
- Push to hub **37min**
    - **34min** git add
    - **3min** git push (several hours with `Repository.git_push()`)

## Conclusion
It appears that loading and saving the data is the main bottleneck at that scale (**8.5h**) whereas processing (**2h**) and pushing the data to the hub (**0.5h**) is relatively fast. To optimize the performance at this scale it would make sense to consider such an end-to-end example and target the bottlenecks which seem to be loading from and saving to disk. The processing itself seems to run relatively fast.

## Notes
- map operation on a 1TB dataset with 96 workers requires >1TB RAM
- map operation does not maintain 100% CPU utilization with 96 workers
- sometimes when the script crashes all the data files have a corresponding `*.lock` file in the data folder (or multiple e.g. `*.lock.lock` when it happened a several times). This causes the cache **not** to be triggered (which is significant at that scale) - i guess because there are new data files
- parallelizing `to_parquet` decreased the saving time from 17h to 5h, however adding more workers at this point had almost no effect. not sure if this is:
    a) a bug in my parallelization logic, 
    b) i/o limit to load data form disk to memory or 
    c) i/o limit to write from memory to disk.
- Using `Repository.git_push()` was much slower than using command line `git-lfs` - 10-20MB/s vs. 300MB/s! The `Dataset.push_to_hub()` function is even slower as it only uploads one file at a time with only a few MB/s, whereas `Repository.git_push()` pushes files in parallel (each at a similar speed).

cc @lhoestq @julien-c @LysandreJik @SBrandeis
Bugs in NewsQA dataset
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
NewsQA dataset has the following bugs:
- the field `validated_answers` is an exact copy of the field `answers` but with the addition of `'count': [0]` to each dict
- the field `badQuestion` does not appear in `answers` nor `validated_answers`

## Steps to reproduce the bug
By inspecting the dataset script we can see that:
- the parsing of `validated_answers` is a copy-paste of the one for `answers`
- the `badQuestion` field is ignored in the parsing of both `answers` and `validated_answers`
Checksum Error when loading multi-news dataset
Author: byw2 | Pull request: no | State: closed

## Describe the bug
When using the load_dataset function from datasets module to load the Multi-News dataset, does not load the dataset but throws Checksum Error instead.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("multi_news")
```


## Expected results
Should download and load Multi-News dataset.

## Actual results
Throws the following error and cannot load data successfully:

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C']
```

Could this issue please be looked at? Thanks!
Wrong number of examples when loading a text dataset
Author: kg-nlp | Pull request: no | State: closed

## Describe the bug
when I use load_dataset to read a txt file  I find  that the number of the samples is incorrect

## Steps to reproduce the bug
```
fr = open('train.txt','r',encoding='utf-8').readlines()
print(len(fr))  # 1199637

datasets = load_dataset('text', data_files={'train': ['train.txt']}, streaming=False)
print(len(datasets['train']))  # 1199649
```
I also use command line operation to verify it
```
$ wc -l train.txt 
1199637 train.txt
```

## Expected results
please fix that issue 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.3
- Platform:windows&linux
- Python version:3.7
- PyArrow version:6.0.1
VoxPopuli
Author: VictorSanh | Pull request: no | State: closed

## Adding a Dataset
- **Name:** VoxPopuli
- **Description:** A Large-Scale Multilingual Speech Corpus
- **Paper:** https://arxiv.org/pdf/2101.00390.pdf
- **Data:** https://github.com/facebookresearch/voxpopuli
- **Motivation:** one of the largest (if not the largest) multilingual speech corpus: 400K hours of multilingual unlabeled speech + 17k hours of labeled speech

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

👀 @kahne @Molugan
Patch all module attributes in its namespace
Author: albertvillanova | Pull request: yes | State: closed

When patching module attributes, only those defined in its `__all__` variable were considered by default (only falling back to `__dict__` if `__all__` was None).

However those are only a subset of all the module attributes in its namespace (`__dict__` variable).

This PR fixes the problem of modules that have non-None `__all__` variable, but try to access an attribute present in `__dict__` (and not in `__all__`).

For example, `pandas` has attribute `__version__` only present in `__dict__`.
- Before version 1.4, pandas `__all__` was None, thus all attributes in `__dict__` were patched
- From version 1.4, pandas `__all__` is not None, thus attributes in `__dict__` not present in `__all__` are ignored

Fix #3724.

CC: @severo @lvwerra
Bug while streaming CSV dataset with pandas 1.4
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
If we upgrade to pandas `1.4`, the patching of the pandas module is no longer working
```
AttributeError: '_PatchedModuleObj' object has no attribute '__version__'
```

## Steps to reproduce the bug
```
pip install pandas==1.4
```
```python
from datasets import load_dataset
ds = load_dataset("lvwerra/red-wine", split="train", streaming=True)
item = next(iter(ds))
item
```
Multi-GPU support for `FaissIndex`
Author: rentruewang | Pull request: yes | State: closed

Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716
Builder Configuration Update Required on Common Voice Dataset
Author: aasem | Pull request: no | State: open

Missing language in Common Voice dataset

**Link:** https://huggingface.co/datasets/common_voice

I tried to call the Urdu dataset using `load_dataset("common_voice", "ur", split="train+validation")` but couldn't due to builder configuration not found. I checked the source file here for the languages support:

https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py

and Urdu isn't included there. I assume a quick update will fix the issue as Urdu speech is now available at the Common Voice dataset.

Am I the one who added this dataset? No
wrong condition in `Features ClassLabel encode_example`
Author: Tudyx | Pull request: no | State: closed

## Describe the bug

The `encode_example` function in *features.py* seems to have a wrong condition.

```python
if not -1 <= example_data < self.num_classes:
    raise ValueError(f"Class label {example_data:d} greater than configured num_classes {self.num_classes}")
```

## Expected results

The `not - 1` condition change the result of the condition. For instance, if  `example_data`  equals 4 and ` self.num_classes` equals 4 too, `example_data < self.num_classes` will give `False` as expected  . But if i add the  `not - 1`  condition, `not -1 <= example_data < self.num_classes` will give `True` and raise an exception.

## Environment info

- `datasets` version: 1.18.3
- Python version: 3.8.10
- PyArrow version: 7.00
`FaissIndex` to support multiple GPU and `custom_index`
Author: rentruewang | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!
Fix bugs in msr_sqa dataset
Author: Timothyxxx | Pull request: yes | State: closed

The last version has many problems,

1) Errors in table load-in. Split by a single comma instead of using pandas is wrong.
2) id reduplicated in _generate_examples function.
3) Missing information of history questions which make it hard to use.

I fix it refer to https://github.com/HKUNLP/UnifiedSKG. And we test it to perform normally.
tatoeba_mt: File not found error and key error
Author: jorgtied | Pull request: no | State: closed

## Dataset viewer issue for 'tatoeba_mt'

**Link:** https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt

My data loader script does not seem to work.
The files are part of the local repository but cannot be found. An example where it should work is the subset for "afr-eng".

Another problem is that I do not have validation data for all subsets and I don't know how to properly check whether validation exists in the configuration before I try to download it. An example is the subset for "afr-deu".

Am I the one who added this dataset ? Yes
Loading JSON gets stuck with many workers/threads
Author: lvwerra | Pull request: no | State: open

## Describe the bug
Loading a JSON dataset with `load_dataset` can get stuck when running on a machine with many CPUs. This is especially an issue when loading a large dataset on a large machine. 


## Steps to reproduce the bug
I originally created the following script to reproduce the issue:
```python
from datasets import load_dataset
from multiprocessing import Process
from tqdm import tqdm
import datasets
from transformers import set_seed

def run_tasks_in_parallel(tasks, ds_list):
    for _ in tqdm(range(1000)):
        print('new batch')
        running_tasks = [Process(target=task, args=(ds, i)) for i, (task, ds) in enumerate(zip(tasks, ds_list))]
        for running_task in running_tasks:
            running_task.start()
        for running_task in running_tasks:
            running_task.join()

def get_dataset():
    dataset_name = 'transformersbook/codeparrot'
    ds = load_dataset(dataset_name+'-train', split="train", streaming=True)
    ds = ds.shuffle(buffer_size=1000, seed=1)
    return iter(ds)

def get_next_element(ds, process_id,  N=10000):
    for _ in range(N):
        _ = next(ds)['content']
    print(f'process {process_id} done')
    return

set_seed(1)
datasets.utils.logging.set_verbosity_debug()

n_processes = 8
tasks = [get_next_element for _ in range(n_processes)]
args = [get_dataset() for _ in range(n_processes)]
run_tasks_in_parallel(tasks, args)
```

Today I noticed that it can happen when running it on a single process on a machine with many cores without streaming. So just `load_dataset("transformersbook/codeparrot-train")` alone might cause the issue after waiting long enough or trying many times. It's a slightly random process which makes it especially hard to track down. When I encountered it today it had already processed 17GB of data (the size of the cache folder when it got stuck) before getting stuck.

Here's my current understanding of the error. As far as I can tell it happens in the following block: https://github.com/huggingface/datasets/blob/be701e9e89ab38022612c7263edc015bc7feaff9/src/datasets/packaged_modules/json/json.py#L119-L139

When the try on line 121 fails and the `block_size` is increased it can happen that it can't read the JSON again and gets stuck indefinitely. A hint that points in that direction is that increasing the `chunksize` argument decreases the chance of getting stuck and vice versa. Maybe it is an issue with a lock on the file that is not properly released.

## Expected results
Read a JSON before the end of the universe.

## Actual results
Read a JSON not before the end of the universe.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.10
- PyArrow version: 7.0.0

@lhoestq we dicsussed this a while ago. @albertvillanova we discussed this today :)
`.select`: unexpected behavior with `indices`
Author: gabegma | Pull request: no | State: closed

## Describe the bug
The `.select` method will not throw when sending `indices` bigger than the dataset length; `indices` will be wrapped instead. This behavior is not documented anywhere, and is not intuitive. 

## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"text": ["d", "e", "f"], "label": [4, 5, 6]})
res1 = ds.select([1, 2, 3])['text']
res2 = ds.select([1000])['text']
```

## Expected results
Both results should throw an `Error`.

## Actual results
`res1` will give `['e', 'f', 'd']`
`res2` will give `['e']`

## Environment info
Bug found from this environment:
- `datasets` version: 1.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.7
- PyArrow version: 6.0.1

It was also replicated on `master`.
Unable to load dataset 'big_patent'
Author: ankitk2109 | Pull request: no | State: closed

## Describe the bug
Unable to load the "big_patent" dataset

## Steps to reproduce the bug
```python
load_dataset('big_patent', 'd', 'validation')
```

## Expected results
Download big_patents' validation split from the 'd' subset

## Getting an error saying:
{FileNotFoundError}Local file ..\huggingface\datasets\downloads\6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5\bigPatentData\train.tar.gz doesn't exist

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.18.3
- Platform: Windows
- Python version:3.8
- PyArrow version:7.0.0
ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'
Author: zhangyifei1 | Pull request: no | State: closed

hi :
I want to use the seqeval indicator because of direct load_ When metric ('seqeval '), it will prompt that the network connection fails. So I downloaded the seqeval Py to load locally. Loading code: metric = load_ metric(path='mymetric/seqeval/seqeval.py')

But tips:

Traceback (most recent call last):
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 604, in <module>
    main()
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 481, in main
    metric = load_metric(path='mymetric/seqeval/seqeval.py')
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 610, in load_metric
    dataset=False,
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 450, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'


**What should I do? Please help me, thank you**
Unable to load a dataset
Author: PaulchauvinAI | Pull request: no | State: closed

## Describe the bug
Unable to load a dataset from Huggingface that I have just saved.


## Steps to reproduce the bug
On Google colab
`! pip install datasets `
`from datasets import load_dataset`
`my_path = "wiki_dataset"`
`dataset = load_dataset('wikipedia', "20200501.fr")`
`dataset.save_to_disk(my_path)`
`dataset = load_dataset(my_path)`


## Expected results
Loading the dataset

## Actual results
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: null
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: string
to
{'builder_name': Value(dtype='string', id=None), 'citation': Value(dtype='string', id=None), 'config_name': Value(dtype='string', id=None), 'dataset_size': Value(dtype='int64', id=None), 'description': Value(dtype='string', id=None), 'download_checksums': {}, 'download_size': Value(dtype='int64', id=None), 'features': {'title': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}, 'text': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'post_processed': Value(dtype='null', id=None), 'post_processing_size': Value(dtype='null', id=None), 'size_in_bytes': Value(dtype='int64', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='string', id=None)}}, 'supervised_keys': Value(dtype='null', id=None), 'task_templates': Value(dtype='null', id=None), 'version': {'version_str': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'major': Value(dtype='int64', id=None), 'minor': Value(dtype='int64', id=None), 'patch': Value(dtype='int64', id=None)}}
because column names don't match

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1
Fix ClassLabel to/from dict when passed names_file
Author: albertvillanova | Pull request: yes | State: closed

Currently, `names_file` is a field of the data class `ClassLabel`, thus appearing when transforming it to dict (when saving infos). Afterwards, when trying to read it from infos, it conflicts with the other field `names`.

This PR, removes `names_file` as a field of the data class `ClassLabel`.
- it is only used at instantiation to generate the `labels` field

Fix #3631.
Fix streaming for servers not supporting HTTP range requests
Author: albertvillanova | Pull request: yes | State: closed

Some servers do not support HTTP range requests, whereas this is required to stream some file formats (like ZIP).

~~This PR implements a workaround for those cases, by download the files locally in a temporary directory (cleaned up by the OS once the process is finished).~~

This PR raises custom error explaining that streaming is not possible because data host server does not support HTTP range requests.

Fix #3677.