helmo/github-issues huggingface dataset:
[Question] Create Apache Arrow dataset from raw text file
State: closed | Author: mrm8488 (CONTRIBUTOR)

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu


Comments: We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the [pyarrow documentation](https://arrow.apache.org/docs/python/)

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in `/datasets`. Here is a link to the steps to [add a dataset](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)

Hello @mrm8488 and @lhoestq 

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :)

> Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:
- [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
- OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
- OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)


> > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?
> 
> Sure, to get a dataset in arrow format you can either:
> 
>     * [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
> 
>     * OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
> 
>     * OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

Links were broken. 

Updated links provided as below
- [load from local files (txt, json, csv)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-or-remote-files)
- [load from python data (dict, pandas)](https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data)
- [create your own dataset script](https://huggingface.co/docs/datasets/loading_datasets.html#using-a-custom-dataset-loading-script)
add squad Spanish v1 and v2
State: closed | Author: mariamabarham (CONTRIBUTOR)

This PR add the Spanish Squad versions 1 and 2 datasets. 
Fixes #164 

Comments: Nice ! :) 
Can we group them into one dataset with two versions, instead of having two datasets ?

Yes sure, I can use the version as config name

@lhoestq can you check? I grouped them

Awesome :) feel free to merge after fixing the test in the CI

@mariamabarham - feel free to merge when you're ready. I only checked the dummy files. I did not run the SLOW tests.
[Question] BERT-style multiple choice formatting
State: closed | Author: sarahwie (NONE)

Hello, I am wondering what the equivalent formatting of a dataset should be to allow for multiple-choice answering prediction, BERT-style. Previously, this was done by passing a list of `InputFeatures` to the dataloader instead of a list of `InputFeature`, where `InputFeatures` contained lists of length equal to the number of answer choices in the MCQ instead of single items. I'm a bit confused on what the output of my feature conversion function should be when using `dataset.map()` to ensure similar behavior.

Thanks!

Comments: Hi @sarahwie, can you details this a little more?

I'm not sure I understand what you refer to and what you mean when you say "Previously, this was done by passing a list of InputFeatures to the dataloader instead of a list of InputFeature"

I think I've resolved it. For others' reference: to convert from using the [`MultipleChoiceDataset` class](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/utils_multiple_choice.py#L82)/[`run_multiple_choice.py`](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/run_multiple_choice.py) script in Huggingface Transformers, I've done the following for hellaswag:

1. converted the `convert_examples_to_features()` function to only take one input and return a dictionary rather than a list:
```
def convert_examples_to_features(example, tokenizer, max_length):

    choices_inputs = defaultdict(list)
    for ending_idx, ending in enumerate(example['endings']['ending']):
        text_a = example['ctx']
        text_b = ending

        inputs = tokenizer.encode_plus(
            text_a,
            text_b,
            add_special_tokens=True,
            max_length=max_length,
            pad_to_max_length=True,
            return_overflowing_tokens=True,
        )
        if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
            logger.info(
                "Attention! you are cropping tokens (swag task is ok). "
                "If you are training ARC and RACE and you are poping question + options,"
                "you need to try to use a bigger max seq length!"
            )

        for key in inputs:
            choices_inputs[key].append(inputs[key])
            
        choices_inputs['label'] = int(example['label'])

    return choices_inputs
```
2. apply this directly (instance-wise) to dataset, convert dataset to torch tensors. Dataset is then ready to be passed to `Trainer` instance.

```
dataset['train'] = dataset['train'].map(lambda x: convert_examples_to_features(x, tokenizer, max_length), batched=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'label']
dataset['train'].set_format(type='torch', columns=columns)
```
When will the remaining math_dataset modules be added as dataset objects
State: closed | Author: tylerroost (NONE)

Currently only the algebra_linear_1d is supported. Is there a timeline for making the other modules supported. If no timeline is established, how can I help?

Comments: On a similar note it would be nice to differentiate between train-easy, train-medium, and train-hard

Hi @tylerroost, we don't have a timeline for this at the moment.
If you want to give it a look we would be happy to review a PR on it.
Also, the library is one week old so everything is quite barebones, in particular the doc.
You should expect some bumps on the road.

To get you started, you can check the datasets scripts in the `./datasets` folder on the repo and find the one on math_datasets that will need to be modified. Then you should check the original repository on the math_dataset to see where the other files to download are located and what is the expected format for the various parts of the dataset.

To get a general overview on how datasets scripts are written and used, you can read the nice tutorial on how to add a new dataset for TensorFlow Dataset [here](https://www.tensorflow.org/datasets/add_dataset), our API is not exactly identical but it can give you a high-level overview.

Thanks I'll give it a look
Weird-ish: Not creating unique caches for different phases
State: closed | Author: zphang (NONE)

Sample code:

```python
import nlp
dataset = nlp.load_dataset('boolq')

def func1(x):
    return x

def func2(x):
    return None

train_output = dataset["train"].map(func1)
valid_output = dataset["validation"].map(func1)
print()
print(len(train_output), len(valid_output))
# Output: 9427 9427
```

The map method in both cases seem to be pointing to the same cache, so the latter call based on the validation data will return the processed train data cache.

What's weird is that the following doesn't seem to be an issue:

```python
train_output = dataset["train"].map(func2)
valid_output = dataset["validation"].map(func2)
print()
print(len(train_output), len(valid_output))
# 9427 3270
```

Comments: Looks like a duplicate of #120.
This is already fixed on master. We'll do a new release on pypi soon

Good catch, it looks fixed.
[Commands] In-detail instructions to create dummy data folder
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

### Dummy data command 

This PR adds a new command `python nlp-cli dummy_data <path_to_dataset_folder>` that gives in-detail instructions on how to add the dummy data files. 

It would be great if you can try it out by moving the current dummy_data folder of any dataset in `./datasets` with `mv datasets/<dataset_script>/dummy_data datasets/<dataset_name>/dummy_data_copy` and running the command `python nlp-cli dummy_data ./datasets/<dataset_name>` to see if you like the instructions. 

### CONTRIBUTING.md
Also the CONTRIBUTING.md is made cleaner including a new section on "How to add a dataset". 

### Current PRs 
It would be nice if we can try out if this command helps current PRs, *e.g.* #169  to add a dataset. I comment on those PRs.

Comments: awesome !
Use IndexError instead of ValueError when index out of range
State: closed | Author: richarddwang (CONTRIBUTOR)

**`default __iter__ needs IndexError`**.

When I want to create a wrapper of arrow dataset to adapt to fastai,
I don't know how to initialize it, so I didn't use inheritance but use object composition.
I wrote sth like this.
```
clas HF_dataset():
  def __init__(self, arrow_dataset):
    self.dset = arrow_dataset
  def __getitem__(self, i):
    return self.my_get_item(self.dset)
```
But `for sample in my_dataset:` gave me `ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")` . This is because default `__iter__` will stop when it catched `IndexError`.

You can also see my [work](https://github.com/richardyy1188/Pretrain-MLM-and-finetune-on-GLUE-with-fastai/blob/master/GLUE_with_fastai.ipynb) that uses fastai2 to show/load batches from huggingface/nlp GLUE datasets

So I hope we can use `IndexError` instead to let other people who want to wrap it for any purpose won't be caught by this caveat.

BTW, I super appreciate your work, both transformers and nlp save my life. 💖💖💖💖💖💖💖
[Bug] labels of glue/ax are all -1 
State: closed | Author: richarddwang (CONTRIBUTOR)

```
ax = nlp.load_dataset('glue', 'ax')
for i in range(30): print(ax['test'][i]['label'], end=', ')
```
```
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 
```

Comments: This is the test set given by the Glue benchmark. The labels are not provided, and therefore set to -1.

Ah, yeah. Why it didn’t occur to me. 😂
Thank you for your comment.
[Feature request] separate split name and split instructions
State: closed | Author: yjernite (MEMBER)

Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?

Comments: If your dataset is a collection of sub-datasets, you should probably consider having one config per sub-dataset. For example for Glue, we have sst2, mnli etc.
If you want to have multiple train sets (for example one per stage). The easiest solution would be to name them `nlp.Split("train_stage1")`, `nlp.Split("train_stage2")`, etc. or something like that.

Thanks for the tip! I ended up setting up three different versions of the dataset with their own configs.

for the named splits, I was trying with `nlp.Split("train-stage1")`, which fails. Changing to `nlp.Split("train_stage1")` works :) I looked for examples of what works in the code comments, it may be worth adding some examples of valid/invalid names in there?
[Manual data] improve error message for manual data in general
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

`nlp.load("xsum")` now leads to the following error message:

![Screenshot from 2020-05-20 20-05-28](https://user-images.githubusercontent.com/23423619/82481825-3587ea00-9ad6-11ea-9ca2-5794252c6ac7.png)

I guess the manual download instructions for `xsum` can also be improved.
[Manual data dir] Error message: nlp.load_dataset('xsum') -> TypeError
State: closed | Author: sshleifer (CONTRIBUTOR)

v 0.1.0 from pip

```python
import nlp
xsum = nlp.load_dataset('xsum')
```

Issue is `dl_manager.manual_dir`is `None`

```python

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a32f066f3bd> in <module>
----> 1 xsum = nlp.load_dataset('xsum')

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    397         split_dict = SplitDict(dataset_name=self.name)
    398         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 399         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    400         # Checksums verification
    401         if verify_infos:

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/datasets/xsum/5c5fca23aaaa469b7a1c6f095cf12f90d7ab99bcc0d86f689a74fd62634a1472/xsum.py in _split_generators(self, dl_manager)
    102         with open(dl_path, "r") as json_file:
    103             split_ids = json.load(json_file)
--> 104         downloaded_path = os.path.join(dl_manager.manual_dir, "xsum-extracts-from-downloads")
    105         return [
    106             nlp.SplitGenerator(

~/miniconda3/envs/nb/lib/python3.7/posixpath.py in join(a, *p)
     78     will be discarded.  An empty last part will result in a path that
     79     ends with a separator."""
---> 80     a = os.fspath(a)
     81     sep = _get_sep(a)
     82     path = a

TypeError: expected str, bytes or os.PathLike object, not NoneType


```
Rm extracted test dirs
State: closed | Author: lhoestq (MEMBER)

All the dummy data used for tests were duplicated. For each dataset, we had one zip file but also its extracted directory. I removed all these directories

Furthermore instead of extracting next to the dummy_data.zip file, we extract in the temp `cached_dir` used for tests, so that all the extracted directories get removed after testing.

Finally there was a bug in the `mock_download_manager` that would let it create directories with invalid names, as in #172. I fixed that by encoding url arguments. I had to rename the dummy data for `scientific_papers` and `cnn_dailymail` (the aws tests don't pass for those 2 in this PR, but they will once aws will be synced, as the local ones do)

Let me know if it sounds good to you @patrickvonplaten . I'm still not entirely familiar with the mock downloader

Comments: Thanks for cleaning up the extracted dummy data folders! Instead of changing the file_utils we could also just put these folders under `.gitignore` (or maybe already done?).

Awesome! I guess you might have to add the changes for the MockDLManager now in a different file though because of my last PR - sorry!
Clone not working on Windows environment
State: closed | Author: codehunk628 (CONTRIBUTOR)

Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument



Comments: Should be fixed on master now :)

Thanks @lhoestq 👍  Now I can uninstall WSL and get back to work with windows.🙂
fix squad metric format
State: closed | Author: lhoestq (MEMBER)

The format of the squad metric was wrong.
This should fix #143 

I tested with
```python3
predictions = [
    {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
]
references = [
    {'answers': [{'text': 'Denver Broncos'}], 'id': '56be4db0acb8001400a502ec'}
]
```

Comments: One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.

(maybe it's not really possible in general though)

This is kinda related to one thing I had in mind which is that we may want to be able to dump our model predictions in a `Dataset` as well so that we don't keep them in memory (and we can export them in a nice format later as well when we will have a serialization formats).

Maybe this is overkill though, I haven't fully wraped my head around this.

I'm also perfectly fine with merging this PR in the current state and working on a larger scope later.

This is the format needed to run the official script directly. The format of the squad dataset is different from the input of the metric. 

> One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.
> 
> (maybe it's not really possible in general though)

Ok I see. I'll try to use the same format

Ok with this update I changed the format to fit the squad dataset format.
Now you can do:
```python
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```
Rename anli dataset
State: closed | Author: lhoestq (MEMBER)

What we have now as the `anli` dataset is actually the αNLI dataset from the ART challenge dataset. This name is confusing because `anli` is also the name of adversarial NLI (see [https://github.com/facebookresearch/anli](https://github.com/facebookresearch/anli)).

I renamed the current `anli` dataset by `art`.
Adding Qanta (Quizbowl) Dataset
State: closed | Author: EntilZha (CONTRIBUTOR)

This PR adds the qanta question answering datasets from [Quizbowl: The Case for Incremental Question Answering](https://arxiv.org/abs/1904.04792) and [Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples](https://www.aclweb.org/anthology/Q19-1029/) (adversarial fold)

This partially continues a discussion around fixing dummy data from https://github.com/huggingface/nlp/issues/161

I ran the following code to double check that it works and did some sanity checks on the output. The majority of the code itself is from our `allennlp` version of the dataset reader.

```python
import nlp
# Default is full question
data = nlp.load_dataset('./datasets/qanta') 
# Four configs
# Primarily useful for training
data = nlp.load_dataset('./datasets/qanta', 'mode=sentences,char_skip=25')  
# Primarily used in evaluation
data = nlp.load_dataset('./datasets/qanta', 'mode=first,char_skip=25')  
data = nlp.load_dataset('./datasets/qanta', 'mode=full,char_skip=25')  
# Primarily useful in evaluation and "live" play
data = nlp.load_dataset('./datasets/qanta', 'mode=runs,char_skip=25')  
```

Comments: Hi @EntilZha - sorry for waiting so long until taking action here. We created a new command and a new recipe of how to add dummy_data. Can you maybe rebase to `master` as explained in 7. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-contribute-to-nlp and check that your dummy data is correct following the instructions here: https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset ? 

If the tests described in 5. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset pass we can merge the PR :-) 

I updated to the most recent master and followed the steps, but still having the similar error where it can't find the correct file since the path to the directory is given, rather than the individual files within them. This still something wrong about how I'm inputting the data or how the tests are reading it?

It's the dummy_data structure. You actually have to call the dummy data file name `dummy_data` (not .json anything). So there should not be a `dummy_data` folder but for each config only a `dummy_data` which contains your json dummy data. Can you maybe try once more - if it doesn't work I do it for you :-). 

Would that work if there are multiple files? In my case, I'm including something similar to squad 1.0/2.0 where we have the main dataset + an additional challenge set in different files. Would I have the zip decompress to two files in that case?

This dataset was actually a special case. It helped us improve the  dummy data instructions :-), see #195 .Close this PR and merge #194.
[Tests] refactor tests
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

This PR separates AWS and Local tests to remove these ugly statements in the script:
```python
        if "/" not in dataset_name:
            logging.info("Skip {} because it is a canonical dataset")
            return
```

To run a `aws` test, one should now run the following command: 

```python 
pytest -s tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_wmt14
```

The same `local` test, can be run with:
```python 
pytest -s tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_wmt14
```

Comments: Nice !
Add a method to shuffle a dataset
State: closed | Author: thomwolf (MEMBER)
Labels: name: generic discussion

Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?

Comments: +1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)

+1 for shuffle in `Dataloader`. 
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.

As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher


We added the `.shuffle` method :)

Closing this one.