noamaanMulla-03/datasets-issues huggingface dataset:
`Proto_qa` hosting seems to be broken

## Describe the bug
The hosting (on Github) of the `proto_qa` dataset seems broken. I haven't investigated more yet, just flagging it for now. 

@zaidalyafeai if you want to dive into it, I think it's just a matter of changing the links in `proto_qa.py`

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("proto_qa")
```

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/hf/.cache/huggingface/modules/datasets_modules/datasets/proto_qa/445346efaad5c5f200ecda4aa7f0fb50ff1b55edde3003be424a2112c3e8102e/proto_qa.py", line 131, in _split_generators
    train_fpath = dl_manager.download(_URLs[self.config.name]["train"])
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 199, in download
    num_proc=download_config.num_proc,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/iesl/protoqa-data/master/data/train/protoqa_train.jsonl
```

Labels: name: bug

@VictorSanh , I think @mariosasko is already working on it.
[Question] How to move and reuse preprocessed dataset? 

Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :

copy path_to_cache_dir/datasets to new_cache_dir/datasets
set export HF_DATASETS_CACHE="new_cache_dir/"
but the program still re-preprocess the whole dataset without loading cache.

I also tried to torch.save(lm_datasets, fw), but the saved file is only 14M.

What is the proper way to do this?


@lhoestq @LysandreJik


Hi :) Can you share with us the code you used ?


EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?


Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

> Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~
Fix PermissionError on Windows when using tqdm >=4.50.0

See: https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111

```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
```

Labels: name: bug
Allow to use tqdm>=4.50.0

We used to have permission errors on windows whith the latest versions of tqdm (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/6365/workflows/24f7c960-3176-43a5-9652-7830a23a981e/jobs/39232))

They were due to open arrow files not properly closed by pyarrow.
Since https://github.com/huggingface/datasets/commit/42320a110d9d072703814e1f630a0d90d626a1e6 gc.collect is called each time we don't need an arrow file to make sure that the files are closed.

close https://github.com/huggingface/datasets/issues/2471

cc @lewtun
Implement layered building

As discussed with @stas00 and @lhoestq (see also here https://github.com/huggingface/datasets/issues/2481#issuecomment-859712190):

> My suggestion for this would be to have this enabled by default.
> 
> Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:
>
> 1. uncompress a handful of files via a generator enough to generate one arrow file
> 2. process arrow file 1
> 3. delete all the files that went in and aren't needed anymore.
>
> rinse and repeat.
> 
> 1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
> 2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
> 3. It would already include deleting temp files this issue is talking about
> 
> I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.

Labels: name: enhancement
Don't use pyarrow 4.0.0 since it segfaults when casting a sliced ListArray of integers

This test `tests/test_table.py::test_concatenation_table_cast` segfaults with the latest update of pyarrow 4.0.0.
Setting `pyarrow<4.0.0` for now. I'll open an issue on JIRA once I know more about the origin of the issue


@lhoestq note that the segfault also occurs on Linux.

Created the ticket at
https://issues.apache.org/jira/browse/ARROW-12568

@lhoestq the ticket you mentioned is now in state resolved. Pyarrow supports AArch64 after version 4.0.0. Because of this restriction `datasets` is not installing in AArch64 systems.
Issue in timit_asr database

## Describe the bug
I am trying to load the timit_asr dataset however only the first record is shown (duplicated over all the rows).
I am using the next code line
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))

The above code result with the same sentence duplicated ten times.
It also happens when I use the dataset viewer at Streamlit .

## Steps to reproduce the bug
from datasets import load_dataset
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))
data = dataset.to_pandas()

# Sample code to reproduce the bug
```

## Expected results
table with different row information

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.4.1 (also occur in the latest version)
- Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 1.15.3 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Labels: name: bug

This bug was fixed in #1995. Upgrading datasets to version 1.6 fixes the issue!

Indeed was a fixed bug.
Works on version 1.8
Thanks
PyTorch not available error on SageMaker GPU docker though it is installed 

I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines 

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`. 

Also as a first line in the data loading module I have 

```
import os
os.environ["USE_TF"] = "0" 
os.environ["USE_TORCH"] = "1" 
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks! 




cc @philschmid 

Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`

Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`. 
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same. 

Could paste the code you use the start your training job and the fine-tuning script you run? 

@sivakhno this should be now fixed in `datasets>=1.5.0`. 

@philschmid Recently released tensorflow-macos seems to be missing. 

I've created a PR to add this.
Allow latest pyarrow version once segfault bug is fixed

As pointed out by @symeneses (see https://github.com/huggingface/datasets/pull/2268#issuecomment-860048613), pyarrow has fixed the segfault bug present in version 4.0.0 (see https://issues.apache.org/jira/browse/ARROW-12568):
- it was fixed on 3 May 2021
- version 4.0.1 was released on 19 May 2021 with the bug fix

Labels: name: enhancement
Fix docs custom stable version

Currently docs default version is 1.5.0. This PR fixes this and sets the latest version instead.


I see that @lhoestq overlooked this PR with his commit 07e2b05. 😢 

I'm adding a script so that this issue does not happen again.


For the moment, the script only includes `update_custom_js`, but in a follow-up PR I will include all the required steps to make a package release.

I think we just need to clarify the release process in setup.py instead of adding a script that does the replacement

@lhoestq I really think we should implement a script that performs the release (instead of doing it manually as it is done now), as it is already the case in `transformers`. I will do it in a next PR.

For the moment, this PR includes one of the steps of the release script.
Update DatasetMetadata and ReadMe

This PR contains the changes discussed in #2395.

**Edit**:
In addition to those changes, I'll be updating the `ReadMe` as follows:

Currently, `Section` has separate parsing and validation error lists. In `.validate()`, we add these lists to the final lists and throw errors.

One way to make `ReadMe` consistent with `DatasetMetadata` and add a separate `.validate()` method is to throw separate parsing and validation errors. 

This way, we don't have to throw validation errors, but only parsing errors in `__init__ ()`. We can have an option in `__init__()` to suppress parsing errors so that an object is created for validation. Doing this will allow the user to get all the errors in one go.

In `test_dataset_cards` , we are already catching error messages and appending to a list. This can be done for `ReadMe()`  for parsing errors, and `ReadMe(...,suppress_errors=True); readme.validate()`  for validation, separately.

**Edit 2**:
The only parsing issue we have as of now is multiple headings at the same level with the same name. I assume this will happen very rarely, but it is still better to throw an error than silently pick one of them. It should be okay to separate it this way. 

Wdyt @lhoestq ?
adding masahaner dataset

Adding Masakhane dataset https://github.com/masakhane-io/masakhane-ner 

@lhoestq , can you please review


Thank you for the review. 

Thanks a lot for the corrections and comments. 

I have resolved point 2. The make style still throws some errors, please see below

black --line-length 119 --target-version py36 tests src benchmarks datasets/**/*.py metrics
/bin/sh: 1: black: not found
Makefile:13: recipe for target 'style' failed
make: *** [style] Error 127

Can you help to resolve this?

Thank you very much @lhoestq for the help.
Update version in xor_tydi_qa.py

Fix #2449

@lhoestq Should I revert to the old `dummy/1.0.0` or  delete it and keep only `dummy/1.1.0`?


Hi ! Thanks for updating the version

> Should I revert to the old dummy/1.0.0 or delete it and keep only dummy/1.1.0?

Feel free to delete the old dummy data files
Fix automatic generation of Zenodo DOI

After the last release of Datasets (1.8.0), the automatic generation of the Zenodo DOI failed: it appears in yellow as "Received", instead of in green as "Published".

I have contacted Zenodo support to fix this issue.

TODO:
- [x] Check with Zenodo to fix the issue
- [x] Check BibTeX entry is right

Labels: name: bug

I have received a reply from Zenodo support:
> We are currently investigating and fixing this issue related to GitHub releases. As soon as we have solved it we will reach back to you.

Other repo maintainers had the same problem with Zenodo. 

There is an open issue on their GitHub repo: zenodo/zenodo#2181

I have received the following request from Zenodo support:
> Could you send us the link to the repository as well as the release tag?

My reply:
> Sure, here it is:
> - Link to the repository: https://github.com/huggingface/datasets
> - Link to the repository at the release tag: https://github.com/huggingface/datasets/releases/tag/1.8.0
> - Release tag: 1.8.0

Zenodo issue has been fixed. The 1.8.0 release DOI can be found here: https://zenodo.org/record/4946100#.YMd6vKj7RPY
Missing original answers in kilt-TriviaQA

I previously opened an issue at https://github.com/facebookresearch/KILT/issues/42 but from the answer of @fabiopetroni it seems that the problem comes from HF-datasets

## Describe the bug
The `answer` field in kilt-TriviaQA, e.g. `kilt_tasks['train_triviaqa'][0]['output']['answer']` contains a list of alternative answer which are accepted for the question.  
However it'd be nice to know the original answer to the question (the only fields in `output` are `'answer', 'meta', 'provenance'`)

## How to fix
It can be fixed by retrieving the original answer from the original TriviaQA (e.g. `trivia_qa['train'][0]['answer']['value']`), perhaps at the same place as here where one retrieves the questions https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md#loading-the-kilt-knowledge-source-and-task-data

cc @yjernite who previously answered to an issue about KILT and TriviaQA :)


Labels: name: bug

That could be useful indeed! Feel free to open a PR on the dataset card if you already have some code that runs, otherwise we'll take care of it soon :) 

I can open a PR but there is 2 details to fix:
- the name for the corresponding key (e.g. `original_answer`)
- how to implement it: I’m not sure what happens when you map `lambda x: {'input': ...}` as it keeps the other keys (e.g. `output`) intact but here since we want to set a nested value (e.g. `x['output']['original_answer']`) I implemented it with a regular function (not lambda), see below

```py
def add_original_answer(x, trivia_qa, triviaqa_map):
    i = triviaqa_map[x['id']]
    x['output']['original_answer'] = trivia_qa['validation'][i]['answer']['value']
    return x
```
fix #2391 add original answers in kilt-TriviaQA

cc @yjernite is it ok like this?


LGTM, but I'm not sure what's going on with the Unix tests @lhoestq 

The CI error is unrelated to this PR, it's been fixed now on master.

Thanks @PaulLerner !

> #- [ ] - Hey![image](https://user-images.githubusercontent.com/71971234/121969638-00030e00-cd75-11eb-9512-25d32ac08051.jpeg)@fr[fr_fr**fr~~fr `fr```
FR
````~~**_]()

Oh that was unexpected. I didn't know pokemons were into NLP
Load_dataset for local CSV files

The method load_dataset fails to correctly load a dataset from csv. 

Moreover, I am working on a token-classification task ( POS tagging) , where each row in my CSV contains two columns each of them having a list of strings.
row example:
```tokens  |  labels
['I' , 'am', 'John']  |  ['PRON', 'AUX', 'PROPN' ] 
```
The method, loads each list as a string:  (i.g "['I' , 'am', 'John']").
To solve this issue, I copied the Datasets.Features, created Sequence types ( instead of Value)  and tried to cast the features type
```
new_features['tokens'] = Sequence(feature=Value(dtype='string', id=None))
new_features['labels'] = Sequence(feature=ClassLabel(num_classes=len(tag2idx), names=list(unique_tags)))
dataset = dataset.cast(new_features)
```
but I got the following error 
```
ArrowNotImplementedError: Unsupported cast from string to list using function cast_list
```
Moreover, I tried to set feature parameter in load_dataset method, to my new_features, but this fails as well.
How can this be solved ?

Labels: name: bug

Hi,

this is not a standard CSV file (requires additional preprocessing) so I wouldn't label this as s bug. You could parse the examples with the regex module or the string API to extract the data, but the following approach is probably the easiest (once you load the data):
```python
import ast
# load the dataset and copy the features
def process(ex):
    return {"tokens": ast.literal_eval(ex["tokens"]), "labels": ast.literal_eval(ex["labels"])}
dataset = dataset.map(process, features=new_features)
```


Hi,

Thanks for the reply.
I  have already used  ```ast.literal_eval``` to evaluate the string into list, but I was getting another error:
```
ArrowInvalid: Could not convert X with type str: tried to convert to int
```
Why this happens ? Should labels be mapped to their ids and use int instead of str ?

Yes, just map the labels to their ids.
Set download/extracted paths configurable

As discussed with @stas00 and @lhoestq, setting these paths configurable may allow to overcome disk space limitation on different partitions/drives.

TODO:
- [x] Set configurable extracted datasets path: #2487
- [x] Set configurable downloaded datasets path: #2488
- [ ] Set configurable "incomplete" datasets path?

Labels: name: enhancement

For example to be able to send uncompressed and temp  build files to another volume/partition, so that the user gets the minimal disk usage on their primary setup - and ends up with just the downloaded compressed data + arrow files, but outsourcing the huge files and building to another partition. e.g. on JZ there is a special partition for fast data, but it's also volatile, so only temp files should go there.

Think of it as `TMPDIR` so we need the equivalent for `datasets`.
Keep original features order

When loading a Dataset from a JSON file whose column names are not sorted alphabetically, we should get the same column name order, whether we pass features (in the same order as in the file) or not.

I found this issue while working on #2366.


The arrow writer was supposing that the columns were always in the sorted order. I just pushed a fix to reorder the arrays accordingly to the schema. It was failing for many datasets like squad

and obviously it broke everything

Feel free to revert my commit. I can investigate this in the coming days

@lhoestq I do not understand when you say:
> It was failing for many datasets like squad

All the tests were green after my last commit.

> All the tests were green after my last commit.

Yes but loading the actual squad dataset was failing :/
Use default cast for sliced list arrays if pyarrow >= 4

From pyarrow version 4, it is supported to cast sliced lists.

This PR uses default pyarrow cast in Datasets to cast sliced list arrays if pyarrow version is >= 4.

In relation with PR #2461 and #2490.

cc: @lhoestq, @abhi1thakur, @SBrandeis


I believe we don't use PyArrow >= 4.0.0 because of some segfault issues:
https://github.com/huggingface/datasets/blob/1206ffbcd42dda415f6bfb3d5040708f50413c93/setup.py#L78
Can you confirm @lhoestq ?

@SBrandeis pyarrow version 4.0.1 has fixed that issue: #2489 😉
Python Programming Puzzles

## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

👀 @TalSchuster

Thanks @VictorSanh!
There's also a [notebook](https://aka.ms/python_puzzles) and [demo](https://aka.ms/python_puzzles_study) available now to try out some of the puzzles
Json loader fails if user-specified features don't match the json data fields order

If you do
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then depending on the order of the features in the json data field it fails:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ValueError: Target schema's field names are not matching the table's field names: ['tokens', 'ner_tags'], ['ner_tags', 'tokens']
```

This is because one must first re-order the columns of the table to match the `self.config.schema` before calling cast.

One way to fix the `cast` would be to replace it with:
```python
# reorder the arrays if necessary + cast to schema
# we can't simply use .cast here because we may need to change the order of the columns
pa_table = pa.Table.from_arrays([pa_table[name] for name in schema.names], schema=schema)
```

Labels: name: bug