konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Loading Data from HDF files

**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.


I'm currently working on bringing [Ecoset](https://www.pnas.org/doi/10.1073/pnas.2011417118) to huggingface datasets and I would second this request...

I would also like this support or something similar. Geospatial datasets come in netcdf which is derived from hdf5, or zarr. I've gotten zarr stores to work with datasets and streaming, but it takes awhile to convert the data to zarr if it's not stored in that natively. 

@mariosasko , I would like to contribute on this "good second issue" . Is there anything in the works for this Issue or can I go ahead ? 


Hi @VijayKalmath! As far as I know, nobody is working on it, so feel free to take over. Also, before you start, I suggest you comment `#self-assign` on this issue to assign it to yourself.

#self-assign

Hey @mariosasko can you assign this issue to me !!

So basically, we just need to load HDF5 files to Parquet?

e.g. Like this? https://stackoverflow.com/questions/46157709/converting-hdf5-to-parquet-without-loading-into-memory

#self-assign
Hi! I'd love to take a shot at implementing this feature to allow loading HDF5 datasets via Hugging Face Datasets. Will keep you updated here!

Hey @mariosasko,
I've opened a PR to address this: #7625
It adds a new h5folder loader to support .h5 files via load_dataset("h5folder", data_dir=...).
Feedback welcome!
concatenate_datasets removes ClassLabel typing.

## Describe the bug

When concatenating two datasets, we lose typing of ClassLabel columns.

I can work on this if this is a legitimate bug,

## Steps to reproduce the bug
```python
import datasets
from datasets import Dataset, ClassLabel, Value, concatenate_datasets

DS_LEN = 100
my_dataset = Dataset.from_dict(
    {
        "sentence": [f"{chr(i % 10)}" for i in range(DS_LEN)],
        "label": [i % 2 for i in range(DS_LEN)]
    }
)
my_predictions = Dataset.from_dict(
    {
        "pred": [(i + 1) % 2 for i in range(DS_LEN)]
    }
)

my_dataset = my_dataset.cast(datasets.Features({"sentence": Value("string"), "label": ClassLabel(2, names=["POS", "NEG"])}))
print("Original")
print(my_dataset)
print(my_dataset.features)


concat_ds = concatenate_datasets([my_dataset, my_predictions], axis=1)
print("Concatenated")
print(concat_ds)
print(concat_ds.features)


```

## Expected results
The features of `concat_ds` should contain ClassLabel.

## Actual results

On master, I get: 
```
{'sentence': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'pred': Value(dtype='int64', id=None)}
```

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1



Something like this would fix it I think: https://github.com/huggingface/datasets/compare/master...Dref360:HF-3111/concatenate_types?expand=1
Stream TAR-based dataset using iter_archive

I converted all the dataset based on TAR archive to use iter_archive instead, so that they can be streamable.

It means that around 80 datasets become streamable :)

I'm creating a new branch `stream-tar-audio` just for the audio datasets since they need https://github.com/huggingface/datasets/pull/3129 to be merged first

The CI fails are only related to missing sections or tags in the dataset cards - which is unrelated to this PR
download_mode=`force_redownload` does not work on removed datasets

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1
Missing Zenodo 1.13.3 release

After `datasets` 1.13.3 release, this does not appear in Zenodo releases: https://zenodo.org/record/5570305

TODO:
- [x] Contact Zenodo support
- [x] Check it is fixed

Zenodo has fixed on their side the 1.13.3 release: https://zenodo.org/record/5589150
`ModuleNotFoundError: No module named 'fsspec.exceptions'`

## Describe the bug
I keep runnig into a fsspec ModuleNotFound error

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_infos
2021-10-15 15:25:37.863206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-15 15:25:37.863252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 56, in <module>
    from .utils.streaming_download_manager import StreamingDownloadManager
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 11, in <module>
    from fsspec.exceptions import FSTimeoutError
ModuleNotFoundError: No module named 'fsspec.exceptions'
```

Yet, I do have `fsspec`:
```bash
hf@victor-scale:~/dev/promptsource$ pip show fsspec
Name: fsspec
Version: 2021.5.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /home/hf/dev/promptsource/.venv/lib/python3.7/site-packages
Requires: 
Required-by: datasets
```

With the same version of fsspec and `datasets==1.9.0`, I don't see this problem....

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I can't even run `datasets-cli env` actually.., but here's my env:
- `datasets` version: 1.13.3
- Platform: Ubuntu 18.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


Thanks for reporting, @VictorSanh.

I'm fixing it.
`cast_column` makes audio decoding fail

## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


cc @anton-l @albertvillanova 

Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.
Support loading a dataset from SQLite files

As requested by @julien-c, we could eventually support loading a dataset from SQLite files, like it is the case for JSON/CSV files.

for reference Kaggle has a good number of open source datasets stored in sqlite

Alternatively a tutorial or tool on how to convert from sqlite to parquet would be cool too

Hello, could we leverage [`pandas.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) for this? 

This would be basically the same as [`CSVBuilder`](https://github.com/huggingface/datasets/blob/7380140accf522a4363bb56c0b77a4190f49bed6/src/datasets/packaged_modules/csv/csv.py#L127)
, but uses `pandas.read_sql(..., chunksize=1)` instead of `pandas.read_csv(..., iterator=True)`

I'm happy to work on this :) 

self-assign
Fix JNLBA dataset

As mentioned in #3089, I've added more tags and also updated the link for dataset which was earlier using a Google Drive link. 
I'm having problem with generating dummy data as `datasets-cli dummy_data  ./datasets/jnlpba --auto_generate --match_text_files "*.iob2"` is giving `datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
` error. I'll try to add dummy data manually.

Fix #3089.

@albertvillanova all tests are passing now. Either you or @lhoestq can review it!
Fixes to `to_tf_dataset`


Hi ! Can you give some details about why you need these changes ?

Hey, sorry, I should have explained! I've been getting a lot of `VisibleDeprecationWarning` from Numpy, due to an issue in the formatter, see #3084 . This is a temporary workaround (since I'm using these methods in the upcoming course) until I can fix that issue, because I couldn't see an obvious fix for the Numpy formatter. If you can see a quick way to fix that, though, that might be even better!
VisibleDeprecationWarning when using `set_format("numpy")`

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```

I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)
Datasets with Audio feature raise error when loaded from cache due to _resampler parameter

## Describe the bug
As reported by @patrickvonplaten, when loaded from the cache, datasets containing the Audio feature raise TypeError.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: __init__() got an unexpected keyword argument '_resampler'
```
[Audio datasets] Adapting all audio datasets

This PR adds the new `Audio(...)` features - see: https://github.com/huggingface/datasets/pull/2324 to the most important audio datasets:

- Librispeech
- Timit
- Common Voice
- AMI
- ... (others I'm forgetting now)

The PR is curently blocked because the following leads to a problem:

```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

As soon as it's unblocked, I'll adapt the other audio datasets as well. 

@lhoestq - are there other important speech datasets that I'm forgetting here? 

Think PR is good to go otherwise

@lhoestq @albertvillanova - how can we make an exception for the AMI README so that the test doesn't fail? The dataset card definitely should have a data preprocessing section

Hi @patrickvonplaten ,

the data preprocessing section is not defined as a valid section in the readme validation file. After this line:
https://github.com/huggingface/datasets/blob/568db594d51110da9e23d224abded2a976b3c8c7/src/datasets/utils/resources/readme_structure.yaml#L20
feel free to insert (correctly indented of course):
```python
- name: "Dataset Preprocessing"
  allow_empty: true
  allow_empty_text: true
  subsections: null
```
and then the tests should pass.

Thanks a lot @albertvillanova - I've added the feature to all audio datasets and corrected the task of `covost2`
Error related to timeout keyword argument

## Describe the bug
As reported by @patrickvonplaten, a TypeError is raised when trying to load a dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: dataset_info() got an unexpected keyword argument 'timeout'
```
Error when loading a metric

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```
add XCSR dataset

Hi,

I wanted to add the [XCSR ](https://inklab.usc.edu//XCSR/xcsr_datasets) dataset to huggingface! :) 

I followed the instructions of adding new dataset to huggingface and have all the required files ready now! It would be super helpful if you can take a look and review them. Thanks in advance for your time and help. Look forward to hearing from you and can't wait to add XCSR to huggingface :D 

> Hi ! Thanks for adding this dataset :)
> 
> Do you know how the translations were done ? Maybe we can mention that in the dataset card.
> 
> The rest looks all good to me :) good job with the dataset script and the dataset card !
> 
> Just one thing: we try to have dummy_data.zip files that are as small as possible, however here each zip file is 70KB+. It think we can make them even smaller if we remove unnecessary files in them. In particular in the `ar` dummy data zip file, we don't need the data for all languages, but rather only the `ar` files. Could you try to remove the unnecessary files in the dummy data zip files ?

Hi! 

Thank you so much for reviewing this PR. I've updated the README to briefly mention the translations and added a link to the paper, where a detailed description of the translation procedure can be found in the appendix.

For the dummy_data.zip files, is it possible to keep all the current files? I tried to remove some of the files, but the removal led to a failure in the local testing. We also think it may be better to keep the current dummy_data.zip files because all the data are useful actually. Thanks a lot!!

Hi @lhoestq, just a gentle ping on this PR. :D
Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?



Hi @zixiliuUSC, 

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```
feat: increase streaming retry config

Increase streaming config parameters:
* retry interval set to 5 seconds
* max retries set to 20 (so 1mn 40s)

@lhoestq I had 2 runs for more than 2 days each, continuously streaming (they were failing before with 3 retries at 1 sec interval).

They are running on TPU's (so great internet connection) and only had connection errors a few times each (3 & 4). Each time it worked after only 1 retry.
The reason for a higher number of retries is for local connections. It would allow for almost 2mn of a wifi/ethernet disconnection. In practice this should not happen very often.

Let me know if you think it's too much.
add story_cloze


Thanks for pushing this dataset :)

According to the CI, the file `cloze_test_val__spring2016 - cloze_test_ALL_val.csv` is missing in the dummy data zip file (the zip files seem empty). Feel free to add this file with 4-5 lines and it should be good

And you can fix the YAML tags with
```yaml
pretty_name: Story Cloze Test
```
and filling the other tags task_categories and task_ids

If the dataset doesn exist on paperswithcode, you can just leave
```yaml
paperswithcode_id: null
```

@lhoestq can't fix the last test fails.

> Thanks @zaidalyafeai, the failing test is due to an issue in the master branch, that has already been fixed.
> 
> You can include the fix:
> 
> ```
> git checkout add_story_cloze
> git fetch upstream master
> git merge upstream/master
> ```

Thanks @albertvillanova, passed all the tests now. 

Thanks Albert, I fixed the suggested comments. This dataset has no train splits, it is only used for evaluation.
Make `interleave_datasets` more robust

**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration. 
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module, 
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets. 

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq 

Hi @lhoestq  Any response on this issue?

Hi ! Sorry for the late response

I agree `interleave_datasets` would benefit a lot from having more flexibility. If I understand correctly it would be nice to be able to define stopping strategies like `stop="first_exhausted"` (default) or `stop="all_exhausted"`. If you'd like to contribute this feature I'd be happy to give you some pointers :)

Also one can already set the max number of iterations per dataset by doing `dataset.take(n)` on the dataset that should only have `n` samples.

Regarding the `iter_cnt` counter, I think this requires a bit more thoughts, since we might have to be able to backpropagate the the counter if `map` or other transforms have been applied after `interleave_datasets`. 

@sbmaruf I just notice that (1)`interleave_datasets` only samples indices once and reuse for all epochs, and (2) it's limited by the smallest dataset. Do you figure out an alternative way to achieve the same purpose?
Windows CI is unable to test streaming properly because of SSL issues

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```

I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI
Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?) 

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik

@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer
Error in per class precision computation 

## Describe the bug
When trying to get the per class precision values by providing `average=None`, following error is thrown `ValueError: can only convert an array of size 1 to a Python scalar`

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
precision_metric = load_metric("precision")
predictions = [0, 2, 1, 0, 0, 1]
references = [0, 1, 2, 0, 1, 2]
results = precision_metric.compute(predictions=predictions, references=references, average=None)
```

## Expected results
`  {'precision': array([0.66666667, 0.        , 0.        ])}`
as per https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py

## Actual results
```
   output = self._compute(predictions=predictions, references=references, **kwargs)
  File "~/.cache/huggingface/modules/datasets_modules/metrics/precision/94709a71c6fe37171ef49d3466fec24dee9a79846c9f176dff66a649e9811690/precision.py", line 110, in _compute
    sample_weight=sample_weight,
ValueError: can only convert an array of size 1 to a Python scalar
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.6.9
- PyArrow version: 5.0.0


Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```
load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type

## Describe the bug
When loading `the_pile_openwebtext2`, we get the error `pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type`

## Steps to reproduce the bug
```python
import datasets
ds = datasets.load_dataset('the_pile_openwebtext2')
```

## Expected results
Should download the dataset, convert it to an arrow file, and return a working Dataset object.

## Actual results
The download works, but conversion to the arrow file fails as follows:

```
>>> ds = datasets.load_dataset('the_pile_openwebtext2')
Downloading and preparing dataset openwebtext2/plain_text (download: 27.33 GiB, generated: 63.86 GiB
, post-processed: Unknown size, total: 91.19 GiB) to /home/davidbau/.cache/huggingface/datasets/open
webtext2/plain_text/1.0.0/c48ec73ba3483bac673463f48f67e9a4fd8cb49a9d6ec4fb957f0b424b97cf25...
Traceback (most recent call last):
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/builder.py", line 1133,
 in _prepare_split
    writer.write(example, key)
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
366, in write
    self.write_examples_on_file()
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
311, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
115, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
```

- Platform: Ubuntu 20.04
- Python version: python 3.9
- PyArrow version: 3.0.0


I encountered the same bug using different datasets.
any suggestions?

+1, can reproduce here!

I get the same error
Platform: Windows 10
Python: python 3.8.8
PyArrow: 5.0

I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).

Addressed in https://huggingface.co/datasets/the_pile_openwebtext2/discussions/4
load_dataset cannot download the data and hangs on forever if cache dir specified

## Describe the bug
After updating datasets, a code that ran just fine for ages began to fail. Specifying _datasets.load_dataset_'s _cache_dir_ optional argument on Windows 10 machine results in data download to hang on forever. Same call without cache_dir works just fine. Surprisingly exact same code just runs perfectly fine on Linux docker instance running in cloud.

Unfortunately I updated Windows also at the same time and I can't remember which version of datasets was running in my conda environment prior to the update otherwise I would have tried both to check this out.  :(

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train',cache_dir=cache_dir) 
```
Note that exact same code without specifying _cache_dir_ argument works perfectly fine.
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train') 
```

## Expected results
Downloads the dataset and cache is handled in the _cache_dir_ directory
## Actual results
Data download keeps hanging on forever, **NO TRACEBACK**!

## Environment info
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0


Issue was environment inconsistency, updating packages did the trick

`conda install -c huggingface -c conda-forge datasets`

> Collecting package metadata (current_repodata.json): done
> Solving environment: |
> The environment is inconsistent, please check the package plan carefully
> The following packages are causing the inconsistency:
> 
>   - conda-forge/noarch::datasets==1.12.1=pyhd8ed1ab_1
>   - conda-forge/win-64::multiprocess==0.70.12.2=py38h294d835_0
> done
> 
> Package Plan
> 
>   environment location: C:\xxx\anaconda3\envs\UnBias-94-1
> 
>   added / updated specs:
>     - datasets
> 
> 
> The following NEW packages will be INSTALLED:
> 
>   dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
> 
> The following packages will be UPDATED:
> 
>   ca-certificates    pkgs/main::ca-certificates-2021.9.30-~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
>   certifi            pkgs/main::certifi-2021.5.30-py38haa9~ --> conda-forge::certifi-2021.10.8-py38haa244fe_0
> 
> The following packages will be SUPERSEDED by a higher-priority channel:
>
Non-Matching Checksum Error with crd3 dataset

## Describe the bug
When I try loading the crd3 dataset (https://huggingface.co/datasets/crd3), an error is thrown.

## Steps to reproduce the bug
```python
dataset = load_dataset('crd3', split='train')
```

## Expected results
I expect no error to be thrown.


## Actual results
A non-matching checksum error is thrown.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/RevanthRameshkumar/CRD3/archive/master.zip']
```

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0



I got the same error for another dataset (`multi_woz_v22`):

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json']
```

I'm seeing the same issue as @RylanSchaeffer:
Python 3.7.11, macOs 11.4
datasets==1.14.0

fails on:
```python
dataset = datasets.load_dataset("multi_woz_v22")
```
Identify which shard data belongs to

**Is your feature request related to a problem? Please describe.**

I'm training on a large dataset made of multiple sub-datasets.
During training I can observe some jumps in loss which may correspond to different shards.

![image](https://user-images.githubusercontent.com/715491/136668758-521263aa-a9b2-4ad2-8d22-060b6bf86a1c.png)

My suspicion is that either:
* some of the sub-datasets are harder for the model than others
* some of the sub-datasets are not formatted properly

I'd like to identify which shards correspond to those jumps.

**Describe the solution you'd like**

It would be nice to have a key associated to each data sample or data batch containing details on where the data comes from (shard idx + item idx within the shard).
This should be supported both in local and streaming mode.

**Describe alternatives you've considered**
A fix would be for me to add myself details (shard id, sample id) as part of each data sample.
The inconvenient is that it requires users to process/reupload every dataset when they need this feature.

Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.
Add PASS dataset

## Adding a Dataset
- **Name:** PASS
- **Description:** An ImageNet replacement for self-supervised pretraining without humans
- **Data:** https://www.robots.ox.ac.uk/~vgg/research/pass/ https://github.com/yukimasano/PASS

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Improving elasticsearch integration

- adding murmurhash signature to sample in index
- adding optional credentials for remote elasticsearch server
- enabling sample update in index
- upgrade the elasticsearch 7.10.1 python client
- adding ElasticsearchBulider to instantiate a dataset from an index and a filtering query

@lhoestq @albertvillanova Iwas trying to fix the failing tests in circleCI but is there a test elasticsearch instance somewhere? If not, can I launch a docker container to have one?
Load private data files + use glob on ZIP archives for json/csv/etc. module inference

As mentioned in https://github.com/huggingface/datasets/issues/3032 loading data files from private repository isn't working correctly because of the data files resolved.

#2986 did a refactor of the data files resolver. I added authentication to it.

I also improved it to glob inside ZIP archives to look for json/csv/etc. files and infer which dataset builder (json/csv/etc.) to use.

Fix https://github.com/huggingface/datasets/issues/3032

Note that #2986 needs to get merged first

I have an error on windows:
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```
at the `fsspec` call in `xglob`:
```python
fs, *_ = fsspec.get_fs_token_paths(urlpath, storage_options=storage_options)
```

Looks like the windows CI has an SSL issue... 

I can reproduce it on my windows machine. On linux it works fine though

I'm just skipping the windows test for now

The Windows CI failure seems unrelated to this PR
```python
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3
```
[save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

## Describe the bug

When only keeping a dummy size of a dataset (say the first 100 samples), and then saving it  to disk to upload it in the following to the hub for easy demo/use - not just the small dataset is saved but the whole dataset with an indices file. The problem with this is that the dataset is still very big.

## Steps to reproduce the bug

E.g. run the following:

```python
from datasets import load_dataset, save_to_disk

nlp = load_dataset("glue", "mnli", split="train")
nlp.save_to_disk("full")

nlp = nlp.select(range(100))
nlp.save_to_disk("dummy")
```

Now one can see that both `"dummy"` and `"full"` have the same size. This shouldn't be the case IMO.

## Expected results

IMO `"dummy"` should be much smaller so that one can easily play around with the dataset on the hub.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 5.0.0



Hi,

the `save_to_disk` docstring explains that `flatten_indices` has to be called on a dataset before saving it to save only the shard/slice of the dataset.

That works! Thansk!

Might be worth doing that automatically actually in case the `save_to_disk` is called on a dataset that has an indices mapping :-)

I agree with @patrickvonplaten: this issue is reported recurrently, so better if we implement the `.flatten_indices()` automatically?

That would be great indeed - I don't really see a use case where one would not like to call `.flatten_indices()` before calling `save_to_disk`

+1 on this !
Protect master branch to force contributions via Pull Requests

In order to have a clearer Git history in the master branch, I propose to protect it so that all contributions must be done through a Pull Request and no direct commits to master are allowed.
- The Pull Request allows to give context, discuss any potential issues and improve the quality of the contribution
- The Pull Request will eventually be squashed and merged into master with a single commit that links to the Pull Request page (with all the context/discussions)

Note that we already implemented a protection in the master branch to avoid *merge* commits and ensure a linear history. This proposal goes one step further by avoiding all kind of direct commits and forcing contributions **only** through Pull Requests.

Please note that we can temporarily deactivate this protection if we need to make a direct commit, e.g. at each new version release.

The only way GitHub allows this kind or protection is by requiring a minimal number (at least one) of approvals of the Pull Request. The inconvenient is that the PR creator cannot approve their own PR: another person must approve it before it can be merged into master. To circumvent this, we could eventually disable this protection in the master branch when an urgent commit is needed (e.g. for a hotfix) and there is no other person available at that time to approve the PR.


It would be nice to protect the master from direct commits, but still having a way to merge our own PRs when no review is required (for example when updating a dataset_infos.json file, or minor bug fixes - things that happen quite often actually).
Do you know if there's a way ?

you can if you're an admin of the repo

This is done. Now the master branch is protected:
- [x] Require a pull request before merging: all commits must be made to a non-protected branch and submitted via a pull request
  - Required number of approvals before merging: 1 
- [x] Require linear history: prevent merge commits from being pushed
- [x] These requirements are not enforced for administrators
- [x] Additionally, the master branch is also protected against deletion and force pushes

CC: @lhoestq @julien-c @thomwolf
Errors loading dataset using fs = a gcsfs.GCSFileSystem

## Describe the bug

Cannot load dataset using a `gcsfs.GCSFileSystem`.  I'm not sure if this should be a bug in `gcsfs` or here...

Basically what seems to be happening is that since datasets saves datasets as folders and folders aren't "real objects" in gcs, gcsfs raises a 404 error.  There are workarounds if you use gcsfs directly to download the file, but as is I can't get `load_from_disk` to work.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# load some dataset
dataset = load_dataset("squad", split="train")

# save it to gcs
import gcsfs
fs = gcsfs.GCSFileSystem(project="my-gs-project")
dataset.save_to_disk("gs://my-bucket/squad", fs=fs)

# try to load it from gcs
from datasets import load_from_disk
dataset2 = load_from_disk("my-bucket/squad", fs=fs)
```


## Expected results
`dataset2` would be a copy of `dataset` but loaded from my bucket.

## Actual results
Long traceback but essentially it's a 404 error from gcsfs saying the object `my-bucket/squad` doesn't exist when this is called:
https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L977

This is because there is no actual object called `my-bucket/squad`, there are objects called `my-bucket/squad/dataset.arrow`, etc.

Note that *this* works fine, since it's explicitly saying "download all the objects with this prefix":
```python
            fs.download(src_dataset_path + "/*", dataset_path.as_posix(), recursive=True)
```
For example, I can do a workaround this way:
```python
import tempfile

with tempfile.TemporaryDirectory() as temppath:
    fs.download("gs://my-bucket/squad/*", temppath)
    dataset2 = load_from_disk(temppath)
```

It's unclear to me if it's `gcsfs`'s responsibility to say "hey that's folder not a file, I should try to get objects inside of it not the object itself", or if that's `datasets`'s responsibility... I'm leaning towards the latter since you're never loading a dataset from one file using this function/method, only a dataset folder?

Another minor thing that should maybe should be rolled into this bug...

https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L968

These fail if you pass in a `gs://` path, e.g.

```python
dataset2 = load_from_disk("gs://my-bucket/squad", fs=fs)
```
Because at this point, `dataset_info_path` is `gs:/my-bucket/squad/dataset_info.json`, gcsfs throws a:
```
Invalid bucket name: 'gs:'
```
error



## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: macOS Big Sur 11.6
- Python version: 3.7.12
- PyArrow version: 5.0.0
Error when loading private dataset with "data_files" arg

## Describe the bug
A clear and concise description of what the bug is.

Private datasets with no loading script can't be loaded using `data_files` parameter.

## Steps to reproduce the bug
```python
from datasets import load_dataset
data_files = {"train": "**/train/*/*.jsonl", "valid": "**/valid/*/*.jsonl"}
dataset = load_dataset('dalle-mini/encoded', data_files=data_files, use_auth_token=True, streaming=True)
```

Same error happens in non-streaming mode.

## Expected results
Files should be loaded (whether in streaming or not).

## Actual results
Error:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    539                 try:
--> 540                     local_path = cached_path(file_path, download_config=download_config)
    541                 except FileNotFoundError:

8 frames
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/dalle-mini/encoded/resolve/main/encoded.py

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/dalle-mini/encoded?full=true

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    547                     except Exception:
    548                         raise FileNotFoundError(
--> 549                             f"Couldn't find a directory or a {resource_type} named '{path}'. "
    550                             f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely on {hf_api.endpoint}/datasets"
    551                         )

FileNotFoundError: Couldn't find a directory or a dataset named 'dalle-mini/encoded'. It doesn't exist locally at /content/dalle-mini/encoded or remotely on https://huggingface.co/datasets
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0

@lhoestq 

We'll do a release tomorrow or on wednesday to make the fix available :)

Thanks for reproting !