helmo/github-issues huggingface dataset:
Add MWSC
State: closed | Author: ghomasHudson (CONTRIBUTOR)

Adding the [Modified Winograd Schema Challenge](https://github.com/salesforce/decaNLP/blob/master/local_data/schema.txt) dataset which formed part of the [decaNLP](http://decanlp.com/) benchmark. Not sure how much use people would find for it it outside of the benchmark, but it is general purpose.

Code is heavily borrowed from the [decaNLP repo](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L773-L877).

There's a few (possibly overly opinionated) design choices I made:

- I used the train/test/dev split [buried in the decaNLP code](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L852-L855)
- I split out each example into the 2 alternatives. Originally the data uses the format:
    ```
    The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. 
    Who [feared/advocated] violence? 
    councilmen/demonstrators
    ```
    I split into the 2 variants:
    ```
    The city councilmen refused the demonstrators a permit because they feared violence. 
    Who feared violence? 
    councilmen/demonstrators
    
    The city councilmen refused the demonstrators a permit because they advocated violence. 
    Who advocated violence? 
    councilmen/demonstrators
    ```
    I can't see any use for having the options combined into a single example (splitting them is [the way decaNLP processes](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L846-L850)) them. You can't train on both versions with them combined, and splitting the examples later would be a pain to do. I think [winogrande.py](https://github.com/huggingface/nlp/blob/master/datasets/winogrande/winogrande.py) presents the data in this way?

- I've not used the decaNLP framing (appending the options to the question e.g. `Who feared violence? 
 -- councilmen or demonstrators?`) but left it more generic by adding the options as a new key: `"options":["councilmen","demonstrators"]` This should be an easy thing to change using `map` if needed by a specific application.

Dataset is working as-is but if anyone has any thoughts/preferences on the design decisions here I'm definitely open to different choices.

Comments: Looks good to me
[Feature request] Add `shard()` method to dataset
State: closed | Author: jarednielsen (CONTRIBUTOR)

Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.

Comments: Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?

Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.
add wikisql
State: closed | Author: ghomasHudson (CONTRIBUTOR)

Adding the [WikiSQL](https://github.com/salesforce/WikiSQL) dataset.

Interesting things to note:
- Have copied the function (`_convert_to_human_readable`) which converts the SQL query to a human-readable (string) format as this is what most people will want when actually using this dataset for NLP applications.
- `conds` was originally a tuple but is converted to a dictionary to support differing types.

Would be nice to add the logical_form metrics too at some point.

Comments: That's great work @ghomasHudson !
add pg19 dataset
State: closed | Author: lucidrains (CONTRIBUTOR)

https://github.com/huggingface/nlp/issues/274

Add functioning PG19 dataset with dummy data

`cos_e.py` was just auto-linted by `make style`

Comments: @lucidrains - Thanks a lot for making the PR - PG19 is a super important dataset! Thanks for making it. Many people are asking for PG-19, so it would be great to have that in the library as soon as possible @thomwolf .

@mariamabarham yup! around 11GB!

I'm looking forward to our first deep learning written novel already lol. It's definitely happening

Good to merge IMO.

Oh I just noticed but as we changed the urls to download the files, we have to update `dataset_infos.json`.
Could you re-rurn `nlp-cli test ./datasets/pg19 --save_infos` ?

@lhoestq on it!

should be good!

@lhoestq - I think it's good to merge no?

`dataset_infos.json` is still not up to date with the new urls (we can see that there are urls like `gs://deepmind-gutenberg/train/*` instead of `https://storage.googleapis.com/deepmind-gutenberg/train/*` in the json file)

Can you check that you re-ran the command to update the json file, and that you pushed the changes @lucidrains ?

@lhoestq ohhh, I made the change in this commit https://github.com/lucidrains/nlp/commit/f3e23d823ad9942031be80b7c4e4212c592cd90c , that's interesting that the pull request didn't pick it up. maybe it's because I did it on another machine, let me check and get back to you!

@lhoestq wrong branch 😅 thanks for catching! 

Awesome thanks 🎉
Importing downloaded package repository fails
State: closed | Author: yjernite (MEMBER)
Labels: name: metric bug

The `get_imports` function in `src/nlp/load.py` has a feature to download a package as a zip archive of the github repository and import functions from the unpacked directory. This is used for example in the `metrics/coval.py` file, and would be useful to add BLEURT (@ankparikh).

Currently however, the code seems to have trouble with imports within the package. For example:
```
import nlp
coval = nlp.load_metric('coval')
```
yields:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 432, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/yacine/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval.py", line 21, in <module>
    from .coval_backend.conll import reader  # From: https://github.com/ns-moosavi/coval
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval_backend/conll/reader.py", line 2, in <module>
    from conll import mention
ModuleNotFoundError: No module named 'conll'
```

Not sure what the fix would be there.
Problem while printing doc string when instantiating multiple metrics.
State: closed | Author: codehunk628 (CONTRIBUTOR)
Labels: name: metric bug

When I load more than one metric and try to print doc string of a particular metric,. It shows the doc strings of all imported metric one after the other which looks quite confusing and clumsy.
Attached [Colab](https://colab.research.google.com/drive/13H0ZgyQ2se0mqJ2yyew0bNEgJuHaJ8H3?usp=sharing) Notebook for problem clarification..
Question - Sign Language Datasets
State: closed | Author: AmitMY (CONTRIBUTOR)
Labels: name: enhancement

name: generic discussion

An emerging field in NLP is SLP - sign language processing.

I was wondering about adding datasets here, specifically because it's shaping up to be large and easily usable.
The metrics for sign language to text translation are the same.

So, what do you think about (me, or others) adding datasets here?


An example dataset would be [RWTH-PHOENIX-Weather 2014 T](https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/)
For every item in the dataset, the data object includes:
1. video_path - path to mp4 file
2. pose_path - a path to `.pose` file with human pose landmarks
3. openpose_path - a path to a `.json` file with human pose landmarks
4. gloss - string
5. text - string
6. video_metadata - height, width, frames, framerate


------

To make it a tad more complicated - what if sign language libraries add requirements to `nlp`? for example, sign language is commonly annotated using `ilex`, `eaf`, or `srt` files, which are all loadable as text, but there is no reason for the dataset to parse that file by itself, if libraries exist to do so.

Comments: Even more complicating - 

As I see it, datasets can have "addons".
For example, the WebNLG dataset is a dataset for data-to-text. However, a work of mine and other works enriched this dataset with text plans / underlying text structures. In that case, I see a need to load the dataset "WebNLG" with "plans" addon.

Same for sign language - if there is a dataset of videos, one addon can be to run OpenPose, another to run ARKit4 pose estimation, and another to run PoseNet, or even just a video embedding addon. (which are expensive to run individually for everyone who wants to use these data)

This is something I dabbled with my own implementation to a [research datasets library](https://github.com/AmitMY/meta-scholar/) and I love to get the discussion going on these topics.

This is a really cool idea !
The example for data objects you gave for the RWTH-PHOENIX-Weather 2014 T dataset can totally fit inside the library.

For your point about formats like `ilex`, `eaf`, or `srt`, it is possible to use any library in your dataset script.
However most user probably won't need these libraries, as most datasets don't need them, and therefore it's unlikely that we will have them in the minimum requirements to use `nlp` (we want to keep it as light-weight as possible). If a user wants to load your dataset and doesn't have the libraries you need, an error is raised asking the user to install them.

More generally, we plan to have something like a `requirements.txt` per dataset. This could also be a place for addons as you said. What do you think ?

Thanks, Quentin, I think a `requirements.txt` per dataset will be a good thing.
I will work on adding this dataset next week, and once we sort all of the kinks, I'll add more.
Setting cache_dir gives error on wikipedia download
State: closed | Author: hallvagi (NONE)

First of all thank you for a super handy library! I'd like to download large files to a specific drive so I set `cache_dir=my_path`. This works fine with e.g. imdb and squad. But on wikipedia I get an error:
```
nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=my_path)
```
```
OSError                                   Traceback (most recent call last)
<ipython-input-2-23551344d7bc> in <module>
      1 import nlp
----> 2 nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=path)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    385                     with utils.temporary_assignment(self, "_cache_dir", tmp_data_dir):
    386                         reader = ArrowReader(self._cache_dir, self.info)
--> 387                         reader.download_from_hf_gcs(self._cache_dir, self._relative_data_dir(with_version=True))
    388                         downloaded_info = DatasetInfo.from_directory(self._cache_dir)
    389                         self.info.update(downloaded_info)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/arrow_reader.py in download_from_hf_gcs(self, cache_dir, relative_data_dir)
    231             remote_dataset_info = os.path.join(remote_cache_dir, "dataset_info.json")
    232             downloaded_dataset_info = cached_path(remote_dataset_info)
--> 233             os.rename(downloaded_dataset_info, os.path.join(cache_dir, "dataset_info.json"))
    234             if self._info is not None:
    235                 self._info.update(self._info.from_directory(cache_dir))

OSError: [Errno 18] Invalid cross-device link: '/home/local/NTU/nn/.cache/huggingface/datasets/025fa4fd4f04aaafc9e939260fbc8f0bb190ce14c61310c8ae1ddd1dcb31f88c.9637f367b6711a79ca478be55fe6989b8aea4941b7ef7adc67b89ff403020947' -> '/data/nn/nlp/wikipedia/20200501.de/1.0.0.incomplete/dataset_info.json'
```

Comments: Whoops didn't mean to close this one.
I did some changes, could you try to run it from the master branch ?

Now it works, thanks!
Fix bertscore references
State: closed | Author: lhoestq (MEMBER)

I added some type checking for metrics. There was an issue where a metric could interpret a string a a list. A `ValueError` is raised if a string is given instead of a list.

Moreover I added support for both strings and lists of strings for `references` in `bertscore`, as it is the case in the original code.

Both ways work:
```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, [lg])
score = scorer.compute(lang="en")
```

```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

This should fix #295 and #238
Error in Demo for Specific Datasets
State: closed | Author: s-jse (NONE)
Labels: name: nlp-viewer

Selecting `natural_questions` or `newsroom` dataset in the online demo results in an error similar to the following.

![image](https://user-images.githubusercontent.com/60150701/85347842-ac861900-b4ae-11ea-98c4-a53a00934783.png)


Comments: Thanks for reporting these errors :)

I can actually see two issues here.

First, datasets like `natural_questions` require apache_beam to be processed. Right now the import is not at the right place so we have this error message. However, even the imports are fixed, the nlp viewer doesn't actually have the resources to process NQ right now so we'll have to wait until we have a version that we've already processed on our google storage (that's what we've done for wikipedia for example).

Second, datasets like `newsroom` require manual downloads as we're not allowed to redistribute the data ourselves (if I'm not wrong). An error message should be displayed saying that we're not allowed to show the dataset.

I can fix the first issue with the imports but for the second one I think we'll have to see with @srush to show a message for datasets that require manual downloads (it can be checked whether a dataset requires manual downloads if `dataset_builder_instance.manual_download_instructions is not None`).



I added apache-beam to the viewer. We can think about how to add newsroom. 

We don't plan to host the source files of newsroom ourselves for now.
You can still get the dataset if you follow the download instructions given by `dataset = load_dataset('newsroom')` though.
The viewer also shows the instructions now.

Closing this one. If you have other questions, feel free to re-open :)
snli -1 labels
State: closed | Author: jxmorris12 (CONTRIBUTOR)

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```


Comments: @jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset). 

Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?

Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong 
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them. 

thanks @mariamabarham  :)
Improve input warning for evaluation metrics
State: closed | Author: Tiiiger (NONE)

Hi, 

I am the author of `bert_score`. Recently, we received [ an issue ](https://github.com/Tiiiger/bert_score/issues/62) reporting a problem in using `bert_score` from the `nlp` package (also see #238 in this repo).  After looking into this, I realized that the problem arises from the format `nlp.Metric` takes input. 

Here is a minimal example:
```python
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

The problem in the above code is that `scorer.add()` expects a list of strings as input for the references. As a result, the `scorer` here would take a list of characters in `lg` to be the references. The correct implementation would be calling
```python
scorer.add(lp, [lg])
```

I just want to raise this issue to you to prevent future user errors of a similar kind. I assume some simple type checking can prevent this from happening?

Thanks!
Update metadata for x_stance dataset
State: closed | Author: jvamvas (CONTRIBUTOR)

Thank you for featuring the x_stance dataset in your library. This PR updates some metadata:
- Citation: Replace preprint with proceedings
- URL: Use a URL with long-term availability


Comments: Great! Thanks @jvamvas for these updates.


I have fixed a warning. The remaining test failure is due to an unrelated dataset.

We just fixed the other dataset on master. Could you rebase from master and push to rerun the CI ?
break statement not required
State: closed | Author: mayurnewase (NONE)


Comments: I guess,test failing due to connection error?

We just fixed the other dataset on master. Could you rebase from master and push to rerun the CI ?

If I'm not wrong this function returns None if no main class was found.
I think it makes things less clear not to have a return at the end of the function.
I guess we can have one return in the for loop instead of the break statement, AND one return at the end to explicitly return None.
What do you think ?
ConnectionError - Eli5 dataset download
State: closed | Author: JovanNj (NONE)

Hi, I have a problem with downloading Eli5 dataset. When typing `nlp.load_dataset('eli5')`, I get ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/eli5/LFQA_reddit/1.0.0/explain_like_im_five-train_eli5.arrow

I would appreciate if you could help me with this issue.

Comments: It should ne fixed now, thanks for reporting this one :)
It was an issue on our google storage.

Let me now if you're still facing this issue.

It works now, thanks for prompt help!
update xsum
State: closed | Author: mariamabarham (CONTRIBUTOR)

This PR makes the following update to the xsum dataset:

- Manual download is not required anymore

- dataset can be loaded as follow: `nlp.load_dataset('xsum')`


**Important** 
Instead of using on outdated url to download the data:  "https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json" 

a more up-to-date url stored here: https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz is used
, so that the user does not need to manually download the data anymore. 
There might be slight breaking changes here for xsum. 

Comments: Looks cool!
@mariamabarham can you add a detailed description here what exactly is changed and how the user can load xsum now?

And a rebase should solve the conflicts

This is a super useful PR :-) @sshleifer - maybe you can take a look at the updated version of xsum if you can use it for your use case. Now, one should be able to just load it with:

```python 
nlp.load_datasets("xsum", ....)  # no manual dir required anymore
```
Add ANLI dataset.
State: closed | Author: easonnie (CONTRIBUTOR)

I completed all the steps in https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset and push the code for ANLI. Please let me know if there are any errors.

Comments: Awesome!! Thanks @easonnie.
Let's wait for additional reviews maybe from @lhoestq @patrickvonplaten @jplu