noamaanMulla-03/datasets-issues huggingface dataset:
Preserve dtype for numpy/torch/tf/jax arrays

Fixes #625. This lets the user preserve the dtype of numpy array to pyarrow array which was getting lost due to conversion of numpy array -> list -> pyarrow array. 


Hi @lhoestq, 
It turns out that pyarrow `ListArray` are not recognized as list-like when we get output from `numpy_to_pyarrow_listarray`. This might cause tests to fail. If possible can we convert that `ListArray` output to list inorder for tests to pass? Under the hood it'll maintain the dtype as that of numpy array passed during input only

Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch` https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039 and `test_map_tf`https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056 
they're expecting `float64`. Shouldn't that be `float32` now?

It's normal: pytorch and tensorflow use `float32` by default, unlike numpy which uses `float64`.

I think that we should always keep the precision of the original tensor (torch/tf/numpy).
It means that as it is in this PR it's fine (the precision is conserved when doing the torch/tf -> numpy conversion).

This is a breaking change but in my opinion the fact that we had Value("float64") for torch.float32 tensors was an issue already.

Let me know what you think. Cc @albertvillanova if you have an opinion on this

If we agree on doing this breaking change, we can just change the test. 

Hi @lhoestq, 
Merged master into this branch. Only changing the test is left for now (mentioned below) after which all tests should pass.

> Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch`
> 
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039
> 
> and `test_map_tf`
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
> 
> 
> they're expecting `float64`. Shouldn't that be `float32` now?



> they're expecting float64. Shouldn't that be float32 now?

Yes feel free to update those tests :)

It would be nice to have the same test for JAX as well

Added same test for for JAX too. Also, I saw that I missed changing `test_cast_to_python_objects_jax` like I did for TF and PyTorch. Finished that as well
Bump tqdm version

The recently released tqdm 4.62.1 includes a fix for PermissionError on Windows (submitted by me in https://github.com/tqdm/tqdm/pull/1207), which means we can remove expensive `gc.collect` calls by bumping tqdm to that version. This PR does exactly that and, additionally, fixes a `disable_tqdm` definition that would previously, if used, raise a PermissionError on Windows.
add stack exchange

stack exchange is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

I also change default `timeout` to 100 seconds instead of 10 seconds, otherwise I keep getting read time out when downloading source data of stack exchange and cc100 dataset.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


Hi ! Merging this one since it's all good :)

However I think it would also be better to actually rename it `the_pile_stack_exchange` to make things clearer and to avoid name collisions in the future. I would like to do the same for `books3` as well.

If you don't mind I'll open a PR to do the renaming


> If you don't mind I'll open a PR to do the renaming

@lhoestq That will be nice !!
add books3

books3 is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797 

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


> When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Thanks for the message, we'll definitely improve this

> Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675

Well currently no, but I think @lewtun was about to do it (though he's currently on vacations)

> > Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675
> 
> Well currently no, but I think @lewtun was about to do it (though he's currently on vacations)

yes i plan to start working on this next week #2185 

one question for @richarddwang - do you know if eleutherai happened to also release the "existing" datasets like enron emails and opensubtitles? 

in appendix c of their paper, they provide details on how they extracted these datasets, but it would be nice if we could just point to a url so we can be as close as possible to original implementation.

@lewtun 

> yes i plan to start working on this next week

Nice! Looking forward to it.

> one question for @richarddwang - do you know if eleutherai happened to also release the "existing" datasets like enron emails and opensubtitles?

Sadly, I don't know any existing dataset of enron emails, but I believe opensubtitles dataset is hosted at here. https://the-eye.eu/public/AI/pile_preliminary_components/
![image](https://user-images.githubusercontent.com/17963619/130061667-8c17985a-1c2f-432f-89f0-66a5288611b8.png)


thanks for the link @richarddwang! i think that corpus is actually the youtube subtitles one and my impression is that eleutherai have only uploaded the 14 new datasets they created. i've contacted one of the authors so hopefully they can share some additional info for us :)

btw it might take a while to put together all the corpora if i also need to preprocess them (e.g. the open subtitles / enron email etc), but i expect no longer than a few weeks.
HF_DATASETS_CACHE variable in Windows

I can't seem to use a custom Cache directory in Windows. I have tried:

set HF_DATASETS_CACHE = "C:\Datasets"
set HF_DATASETS_CACHE = "C:/Datasets"
set HF_DATASETS_CACHE = "C:\\Datasets"
set HF_DATASETS_CACHE = "r'C:\Datasets'"
set HF_DATASETS_CACHE = "\Datasets"
set HF_DATASETS_CACHE = "/Datasets"

In each instance I get the "[WinError 123] The filename, directory name, or volume label syntax is incorrect" error when attempting to load a dataset


Agh - I'm a muppet. No quote marks are needed.
set HF_DATASETS_CACHE = C:\Datasets
works as intended.
add openwebtext2

openwebtext2 is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


It seems we need to `pip install jsonlines` to pass the checks ?

Hi ! Do you really need `jsonlines` ? I think it simply uses `json.loads` under the hood.

Currently the test are failing because `jsonlines` is not part of the extra requirements `TESTS_REQUIRE` in setup.py

So either you can replace `jsonlines` with a simple for loop on the lines of the files and use `json.loads`, or you can add `TESTS_REQUIRE` to the test requirements (but in this case users will have to install it as well).

Thanks for your suggestion. I now know `io` and json lines format better and has changed `jsonlines` to just `readlines`.
Add url prefix convention for many compression formats

## Intro

When doing dataset streaming, the uncompression of compressed files is done on the fly using `fsspec`.

In particular, the download manager method `download_and_extract` doesn't return a path to the local download and extracted file, but instead a chained URL so that the uncompression can be done when the file is opened. A few examples of chained URLS:
- `gz://file.txt::https://foo.bar/file.txt.gz`
- `bz2://file.txt::https://foo.bar/file.txt.bz2`
- `zip://::https://foo.bar/archive.zip`
- `tar://::https://foo.bar/archive.tar.gz` (the TAR uncompression includes gz, bz2 etc. uncompression in `fsspec`)

This syntax is highly inspired by the `fsspec` URL chaining syntax from https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining

This url prefixing allows `open` to know what kind of uncompression to do in a dataset script when doing
```python
def _generate_examples(self, urlpath):
    with open(urlpath) as f:
        ....
```

## What it changes

This changes the previous behavior from https://github.com/huggingface/datasets/pull/2786 , in which `open` was trying to infer the compression automatically. Infering the compression made it impossible to know whether the user wanted `open` to return compressed data (as the default behavior of the buitin open), or the uncompressed data. By adding uncompression prefixes to the URL, `open` know directly if it has to uncompress or not, and also which protocol to use.

## Additional notes

This PR should close https://github.com/huggingface/datasets/issues/2813

It should also close this PR https://github.com/huggingface/datasets/pull/2811 since the oscar dataset script won't try to uncompress twice anymore

Note that I had to temporarily remove the support for passing tar and zip files to `data_files` for streaming to make it work, since it makes it ambiguous whether a zip file passed as `data_files` should be uncompressed or not. IMO we can make it work again by changing the syntax to make the glob explicit:
```python
load_dataset("json", data_files="zip://*.jsonl::https://foo.bar/archive.zip")
```
This is the exact same convention as fsspec and it removes all ambiguities

cc @albertvillanova @lewtun 


Thanks for the feedback :) I will also complete the documentation to explain this convention

I just added some documentation about how streaming works with chained URLs.

I will also add some docs about how to use chained URLs directly in `load_dataset` in #2662, since #2662 does change the documentation already and to avoid having to resolve conflicts.

Merging this one now, next step is resolve the conflicts in #2662 and update the docs for URL chaining :)

There is also the glob feature of zip files that I need to add, to be able to do this for example:
```python
load_dataset("json", data_files="zip://*::https://foo.bar/archive.zip")
```
Rename The Pile subsets

After discussing with @yjernite we think it's better to have the subsets of The Pile explicitly have "the_pile" in their names.

I'm doing the changes for the subsets that @richarddwang added:
- [x] books3 -> the_pile_books3 https://github.com/huggingface/datasets/pull/2801
- [x] stack_exchange -> the_pile_stack_exchange https://github.com/huggingface/datasets/pull/2803
- [x] openwebtext2 -> the_pile_openwebtext2 https://github.com/huggingface/datasets/pull/2802

For consistency we should also rename `bookcorpusopen` to `the_pile_bookcorpus` IMO, but let me know what you think.
(we can just add a deprecation message to `bookcorpusopen` for now and add `the_pile_bookcorpus`)


Sounds good. Should we also have a “the_pile” dataset with the subsets as configuration?

I think the main `the_pile` datasets will be the one that is the mix of all the subsets: https://the-eye.eu/public/AI/pile/

We can also add configurations for each subset, and even allow users to specify the subsets they want:
```python
from datasets import load_dataset

load_dataset("the_pile", subsets=["openwebtext2", "books3", "hn"])
```

We're alrady doing something similar for mC4, where users can specify the list of languages they want to load.
IndexError when accessing first element of a Dataset if first RecordBatch is empty

The computation of the offsets of the underlying Table of a Dataset has some issues if the first RecordBatch is empty.

```python
from datasets import Dataset
import pyarrow as pa

pa_table = pa.Table.from_pydict({"a": [1]})
pa_table2 = pa.Table.from_pydict({"a": []}, schema=pa_table.schema)
ds_table = pa.concat_tables([pa_table2, pa_table])

dataset = Dataset(ds_table)

print([len(b) for b in dataset.data._batches])
# [0, 1]

print(dataset.data._offsets)
# [0 0 1] (should be [0, 1])

dataset[0]
```
raises
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/table.py in _interpolation_search(arr, x)
     90         else:
     91             i, j = i, k
---> 92     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
     93 
     94 

IndexError: Invalid query '0' for size 1.
```

This can be fixed by ignoring empty batches when computing `table._batches` and `table._offsets`

cc @SaulLu
SubjQA wrong boolean values in entries

## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4


Labels: name: bug

Hi @arnaudstiegler, thanks for reporting. I'm investigating it.

@arnaudstiegler I have just checked that these mismatches are already present in the original dataset: https://github.com/megagonlabs/SubjQA

We are going to contact the dataset owners to report this.

I have:
- opened an issue in their repo: https://github.com/megagonlabs/SubjQA/issues/3
- written an email to all the paper authors

Please [see my response](https://github.com/megagonlabs/SubjQA/issues/3#issuecomment-905160010). There will be a fix in a couple of days.
Load datasets from the Hub without requiring a dataset script

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)


This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
Load Dataset from the Hub (NO DATASET SCRIPT)

## Load the data from any Dataset repository on the Hub

This PR adds support for loading datasets from any dataset repository on the hub, without requiring any dataset script.

As a user it's now possible to create a repo and upload some csv/json/text/parquet files, and then be able to load the data in one line. Here is an example with the `allenai/c4` repository that contains a lot of compressed json lines files:
```python
from datasets import load_dataset

data_files = {"train": "en/c4-train.*.json.gz"}
c4 = load_dataset("allenai/c4", data_files=data_files, split="train", streaming=True)

print(c4.n_shards)
# 1024
print(next(iter(c4)))
# {'text': 'Beginners BBQ Class Takin...'}
```

By default it loads all the files, but as shown in the example you can choose the ones you want with unix style patterns.

Of course it's still possible to use dataset scripts since they offer the most flexibility.

## Implementation details

It uses `huggingface_hub` to list the files in a dataset repository.

If you provide a path to a local directory instead of a repository name, it works the same way but it uses `glob`.

Depending on the data files available, or passed in the `data_files` parameter, one of the available builders will be used among the csv, json, text and parquet builders.

Because of this, it's not possible to load both csv and json files at once. In this case you have to load them separately and then concatenate the two datasets for example.

## TODO

- [x] tests
- [x] docs
- [x] when huggingface_hub gets a new release, update the CI and the setup.py

Close https://github.com/huggingface/datasets/issues/2629


This is ready for review now :)

I would love to have some feedback on the changes in load.py @albertvillanova. There are many changes so if you have questions let me know, especially on the `resolve_data_files` functions and on the changes in `prepare_module`.

And @thomwolf if you want to take a look at the documentation, feel free to share your suggestions :)

I took your comments into account thanks !
And I made `aiohttp` a required dependency :)

Just updated the documentation :)
[share_datasets.html](https://45532-250213286-gh.circle-artifacts.com/0/docs/_build/html/share_dataset.html)

Let me know if you have some comments

Merging this one :) 

We can try to integrate the changes in the docs to #2718 @stevhliu !

Baked this into the [docs](https://44335-250213286-gh.circle-artifacts.com/0/docs/_build/html/loading.html#hugging-face-hub) already, let me know if there is anything else I should add! :)
load_metric from local "glue.py" meet error 'NoneType' object is not callable

actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset(path='/home/glue.py', name=actual_task)
metric = load_metric(path='/home/glue.py', name=actual_task)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-7ab77a465d81> in <module>
      1 actual_task = "mnli" if task == "mnli-mm" else task
      2 dataset = load_dataset(path='/home/jcli/glue.py', name=actual_task)
----> 3 metric = load_metric(path='/home/jcli/glue.py', name=actual_task)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    508         keep_in_memory=keep_in_memory,
    509         experiment_id=experiment_id,
--> 510         **metric_init_kwargs,
    511     )
    512 

TypeError: 'NoneType' object is not callable

Please help


@Frankie123421 what was the resolution to this?

> @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

thank you!
Fix stream oscar

Previously, an additional `open` was added to oscar to make it stream-compatible: 587bbb94e891b22863b312b99696e32708c379f4.

This was argued that might be problematic: https://github.com/huggingface/datasets/pull/2786#discussion_r690045921

This PR:
- removes that additional `open`
- patches `gzip.open` with `xopen` + `compression="gzip"`


One additional note: if we can try to not change the code of oscar.py too often, I'm sure users that have it in their cache directory will be happy to not have to redownload it every time they update the library ;)

(since changing the code changes the cache directory of the dataset)

I don't think this is confusing for users because users don't even know we have patched `open`. The only thing users care is that if the pass `streaming=True`, they want to be able to load the dataset in streaming mode.

I don't see any other dataset where patching `open` with `fsspec.open`+`compression` is an "underlying issue". Are there other datasets where this is an issue?

The only dataset where this was an issue is in oscar and the issue is indeed due to the additional `open` you added inside `zip.open`.

Closing this one since https://github.com/huggingface/datasets/pull/2822 reverted the change of behavior of `open`
NonMatchingSplitsSizesError error when reading the IMDB dataset

Hi!

I am trying to load the `imdb` dataset with this line:

`dataset = nlp.load_dataset('imdb', data_dir='/A/PATH', cache_dir='/A/PATH')`

but I am getting the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/load.py", line 517, in load_dataset
    save_infos=save_infos,
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 363, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 421, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=5929447, num_examples=4537, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

Am I overlooking something? Thanks!


I just tried on my side and I didn't encounter your problem.
Apparently the script doesn't generate all the examples on your side.

Can you provide the version of `nlp` you're using ?
Can you try to clear your cache and re-run the code ?

I updated it, that was it, thanks!

Hello, I am facing the same problem... how do you clear the huggingface cache?

Hi ! The cache is at ~/.cache/huggingface
You can just delete this folder if needed :)
Update: GooAQ - add train/val/test splits

[GooAQ](https://github.com/allenai/gooaq) dataset was recently updated after splits were added for the same. This PR contains new updated GooAQ with train/val/test splits and updated README as well.


@albertvillanova my tests are failing here:
```
dataset_name = 'gooaq'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True, use_local_dummy_data=True)

tests/test_dataset_common.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_dataset_common.py:187: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```
When I try loading dataset on local machine it works fine. Any suggestions on how can I avoid this error?

Thanks for the help, @albertvillanova! All tests are passing now.