mengta666/huggingface_github huggingface dataset:
Issue: snli -1 labels

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```


Comments:
@jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset). 

Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?

Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong 
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them. 

thanks @mariamabarham  :)
Issue: ConnectionError - Eli5 dataset download

Hi, I have a problem with downloading Eli5 dataset. When typing `nlp.load_dataset('eli5')`, I get ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/eli5/LFQA_reddit/1.0.0/explain_like_im_five-train_eli5.arrow

I would appreciate if you could help me with this issue.

Comments:
It should ne fixed now, thanks for reporting this one :)
It was an issue on our google storage.

Let me now if you're still facing this issue.

It works now, thanks for prompt help!
Issue: Private/sensitive data

Hi all,
Thanks for this fantastic library, it makes it very easy to do prototyping for NLP projects interchangeably between TF/Pytorch. 

Unfortunately, there is data that cannot easily be shared publicly as it may contain sensitive information. 
Is there support/a plan to support such data with NLP, e.g. by reading it from local sources?

Use case flow could look like this: use NLP to prototype an approach on similar, public data and apply the resulting prototype on sensitive/private data without the need to rethink data processing pipelines. 

Many thanks for your responses ahead of time and kind regards,
MFreidank

Comments:
Hi @MFreidank, you should already be able to load a dataset from local sources, indeed. (ping @lhoestq and @jplu)

We're also thinking about the ability to host private datasets on a hosted bucket with permission management, but that's further down the road.

Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.

Hi @julien-c @jplu,
Thanks for sharing this solution with me, it helps, this is what I was looking for. 
If not already there and only missed by me, this could be a great addition in the docs.

Closing my issue as resolved, thanks again.
Issue: Dataset Preprocessing Cache with .map() function not working as expected

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system. 

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file. 

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess. 
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!

Comments:
When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?

Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.

Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?

Hi @sarahwie 
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?

I can't reproduce it, so I'll close for now.
Issue: Empty samples in glue/qqp

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string. 
BTW, I have checked and these two are the only naughty ones in all splits of qqp.

Comments:
We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?

Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.
Issue: NonMatchingChecksumError when loading pubmed dataset

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38 

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.


Comments:
For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
Issue: PG-19

Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.

Comments:
Sounds good! Do you want to give it a try?

Ok, I'll see if I can figure it out tomorrow!

Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?

Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
Issue: c4 dataset is not viewable in nlpviewer demo

I get the following error when I try to view the c4 dataset in [nlpviewer](https://huggingface.co/nlp/viewer/)

```python
ModuleNotFoundError: No module named 'langdetect'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 54, in <module>
    configs = get_confs(option.id)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 48, in get_confs
    builder_cls = nlp.load.import_main_class(module_path, dataset=True)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4.py", line 29, in <module>
    from .c4_utils import (
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4_utils.py", line 29, in <module>
    import langdetect
```

Comments:
C4 is too large to be shown in the viewer
Issue: How can I load/find WMT en-romanian?

I believe it is from `wmt16`

When I run

```python
wmt = nlp.load_dataset('wmt16')
```
I get:
```python
AssertionError: The dataset wmt16 with config cs-en requires manual data. 
 Please follow the manual download instructions:   Some of the wmt configs here, require a manual download.
  Please look into wmt.py to see the exact path (and file name) that has to
  be downloaded.
  . 
 Manual data can be loaded with `nlp.load(wmt16, data_dir='<path/to/manual/data>')
```
There is no wmt.py,as the error message suggests, and wmt16.py doesn't have manual download instructions.

Any idea how to do this?

Thanks in advance!




Comments:
I will take a look :-)
Issue: Why is dataset after tokenization far more larger than the orginal one ?

I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?

Comments:
Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`

Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔

Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.

Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
Issue: [Feature request] Add a feature to dataset

Is there a straightforward way to add a field to the arrow_dataset, prior to performing map?

Comments:
Do you have an example of what you would like to do? (you can just add a field in the output of the unction you give to map and this will add this field in the output table)

Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).

This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```

Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```

Oh great. That should work. I missed that in the documentation- thanks :)
Issue: [Feature request] Be able to remove a specific sample of the dataset

As mentioned in #117, it's currently not possible to remove a sample of the dataset.

But it is a important use case : After applying some preprocessing, some samples might be empty for example. We should be able to remove these samples from the dataset, or at least mark them as `removed` so when iterating the dataset, we don't iterate these samples.

I think it should be a feature. What do you think ?

---

Any work-around in the meantime ?

Comments:
Oh yes you can now do that with the `dataset.filter()` method that was added in #214
Issue: NonMatchingSplitsSizesError error when reading the IMDB dataset

Hi!

I am trying to load the `imdb` dataset with this line:

`dataset = nlp.load_dataset('imdb', data_dir='/A/PATH', cache_dir='/A/PATH')`

but I am getting the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/load.py", line 517, in load_dataset
    save_infos=save_infos,
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 363, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 421, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=5929447, num_examples=4537, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

Am I overlooking something? Thanks!

Comments:
I just tried on my side and I didn't encounter your problem.
Apparently the script doesn't generate all the examples on your side.

Can you provide the version of `nlp` you're using ?
Can you try to clear your cache and re-run the code ?

I updated it, that was it, thanks!

Hello, I am facing the same problem... how do you clear the huggingface cache?

Hi ! The cache is at ~/.cache/huggingface
You can just delete this folder if needed :)
Issue: What is the best way to cache a dataset? 

For example if I want to use streamlit with a nlp dataset:

```
@st.cache
def load_data():
    return nlp.load_dataset('squad')
```
This code raises the error "uncachable object"

Right now I just fixed with a constant for my specific case:
```
    @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
```
But I was curious to know what is the best way in general



Comments:
Everything is already cached by default in 🤗nlp (in particular dataset
loading and all the “map()” operations) so I don’t think you need to do any
specific caching in streamlit.

Tell us if you feel like it’s not the case.

On Sat, 6 Jun 2020 at 13:02, Fabrizio Milo <notifications@github.com> wrote:

> For example if I want to use streamlit with a nlp dataset:
>
> @st.cache
> def load_data():
>     return nlp.load_dataset('squad')
>
> This code raises the error "uncachable object"
>
> Right now I just fixed with a constant for my specific case:
>
>     @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
>
> But I was curious to know what is the best way in general
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/246>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHKAKO7CWGX2QY55UXLRVIO3ZANCNFSM4NV333RQ>
> .
>


Closing this one. Feel free to re-open if you have other questions !
Issue: UnicodeDecodeError when downloading GLUE-MNLI

When I run
```python
dataset = nlp.load_dataset('glue', 'mnli')
```
I get an encoding error (could it be because I'm using Windows?) :
```python
# Lots of error log lines later...
~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\5256cc2368cf84497abef1f1a5f66648522d5854b225162148cb8fc78a5a91cc\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    529 
--> 530                 for n, row in enumerate(reader):
    531                     if is_cola_non_test:

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6744: character maps to <undefined>
```
Anyway this can be solved by specifying to decode in UTF when reading the csv file. I am proposing a PR if that's okay.

Comments:
It should be good now, thanks for noticing and fixing it ! I would say that it was because you are on windows but not 100% sure

On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly. 
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in [PEP 540](https://www.python.org/dev/peps/pep-0540/))

For me this fixed the problem described by the OP.
Issue: Deterministic dataset loading

When calling:
```python 
import nlp
dataset = nlp.load_dataset("trivia_qa", split="validation[:1%]")
```

the resulting dataset is not deterministic over different google colabs. 
After talking to @thomwolf, I suspect the reason to be the use of `glob.glob` in line:

https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/datasets/trivia_qa/trivia_qa.py#L180

which seems to return an ordering of files that depends on the filesystem:
https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered

I think we should go through all the dataset scripts and make sure to have deterministic behavior.

A simple solution for `glob.glob()` would be to just replace it with `sorted(glob.glob())` to have everything sorted by name. 

What do you think @lhoestq?

Comments:
Yes good point !

I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it

I'm pretty sure it would solve the problem too.

The only other dataset that is not deterministic right now is `blog_authorship_corpus` (see #215) but this is a problem related to string encodings.

I think we should do the same also for `os.list_dir`
Issue: [Metric] Bertscore : Warning : Empty candidate sentence; Setting recall to be 0.

When running BERT-Score, I'm meeting this warning :

> Warning: Empty candidate sentence; Setting recall to be 0.

Code :

```
import nlp
metric = nlp.load_metric("bertscore")
scores = metric.compute(["swag", "swags"], ["swags", "totally something different"], lang="en", device=0)
```

---

**What am I doing wrong / How can I hide this warning ?**

Comments:
This print statement comes from the official implementation of bert_score (see [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L343)). The warning shows up only if the attention mask outputs no candidate.
Right now we want to only use official code for metrics to have fair evaluations, so I'm not sure we can do anything about it. Maybe you can try to create an issue on their [repo](https://github.com/Tiiiger/bert_score) ?
Issue: Huggingface NLP, Uploading custom dataset

Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!

Comments:
What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
`nlp.Dataset.from_file` and `load_dataset(..., cache_dir=)`

To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`

Also if you want to upload your script, you should be able to use the `nlp-cli`.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.


Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```
Issue: Not able to access the XNLI dataset

When I try to access the XNLI dataset, I get the following error. The option of plain_text get selected automatically and then I get the following error.

```
FileNotFoundError: [Errno 2] No such file or directory: '/home/sasha/.cache/huggingface/datasets/xnli/plain_text/1.0.0/dataset_info.json'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 86, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 72, in get
    builder_instance = builder_cls(name=conf)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 610, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 152, in __init__
    self.info = DatasetInfo.from_directory(self._cache_dir)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/info.py", line 157, in from_directory
    with open(os.path.join(dataset_info_dir, DATASET_INFO_FILENAME), "r") as f:
```

Is it possible to see if the dataset_info.json is correctly placed?

Comments:
Added pull request to change the name of the file from dataset_infos.json to dataset_info.json

Thanks for reporting this bug !
As it seems to be just a cache problem, I closed your PR.
I think we might just need to clear and reload the `xnli` cache @srush ? 

Update: The dataset_info.json error is gone, but we have a new one instead:
```
ConnectionError: Couldn't reach https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip
```
I am not able to reproduce on my side unfortunately. Any idea @srush ?

xnli is now properly shown in the viewer.
Closing this one.
Issue: Should we still have to force to install apache_beam to download wikipedia ?

Hi, first thanks to @lhoestq 's revolutionary work, I successfully downloaded processed wikipedia according to the doc. 😍😍😍

But at the first try, it tell me to install `apache_beam` and `mwparserfromhell`, which I thought wouldn't be used according to #204 , it was kind of confusing me at that time.

Maybe we should not force users to install these ? Or we just add them to`nlp`'s dependency ?

Comments:
Thanks for your message 😊 
Indeed users shouldn't have to install those dependencies

Got it, feel free to close this issue when you think it’s resolved.

It should be good now :)
Issue: [Feature Request/Help] BLEURT model -> PyTorch

Hi, I am interested in porting google research's new BLEURT learned metric to PyTorch (because I wish to do something experimental with language generation and backpropping through BLEURT). I noticed that you guys don't have it yet so I am partly just asking if you plan to add it (@thomwolf said you want to do so on Twitter).

I had a go of just like manually using the checkpoint that they publish which includes the weights. It seems like the architecture is exactly aligned with the out-of-the-box BertModel in transformers just with a single linear layer on top of the CLS embedding. I loaded all the weights to the PyTorch model but I am not able to get the same numbers as the BLEURT package's python api. Here is my colab notebook where I tried  https://colab.research.google.com/drive/1Bfced531EvQP_CpFvxwxNl25Pj6ptylY?usp=sharing . If you have any pointers on what might be going wrong that would be much appreciated!

Thank you muchly!

Comments:
Is there any update on this? 

Thanks!

Hitting this error when using bleurt with PyTorch ...

```
UnrecognizedFlagError: Unknown command line flag 'f'
```
... and I'm assuming because it was built for TF specifically.  Is there a way to use this metric in PyTorch?

We currently provide a wrapper on the TensorFlow implementation: https://huggingface.co/metrics/bleurt

We have long term plans to better handle model-based metrics, but they probably won't be implemented right away

@adamwlev it would still be cool to add the BLEURT checkpoints to the transformers repo if you're interested, but that would best be discussed there :) 

closing for now

Hi there. We ran into the same problem this year (converting BLEURT to PyTorch) and thanks to @adamwlev found his colab notebook which didn't work but served as a good starting point. Finally, we **made it work** by doing just two simple conceptual fixes: 

1. Transposing 'kernel' layers instead of 'dense' ones when copying params from the original model;
2. Taking pooler_output as a cls_state in forward function of the BleurtModel class.

Plus few minor syntactical fixes for the outdated parts. The result is still not exactly the same, but is very close to the expected one (1.0483 vs 1.0474).

Find the fixed version here (fixes are commented): https://colab.research.google.com/drive/1KsCUkFW45d5_ROSv2aHtXgeBa2Z98r03?usp=sharing  


I created a new model based on `transformers` that can load every BLEURT checkpoints released so far. https://github.com/lucadiliello/bleurt-pytorch

@LoraIpsum Thanks for sharing your work here. However, I'm unable to reproduce the results. That's strange because you are. FYI, I am trying to convert a finetuned BLEURT to PyTorch. Any suggestions on how I can reproduce results?
Issue: Colab Notebook breaks when downloading the squad dataset

When I run the notebook in Colab
https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb
breaks when running this cell:
![image](https://user-images.githubusercontent.com/338917/83311709-ffd1b800-a1dd-11ea-8394-3a87df0d7f8b.png)


Comments:
The notebook forces version 0.1.0. If I use the latest, things work, I'll run the whole notebook and create a PR.

But in the meantime, this issue gets fixed by changing:
`!pip install nlp==0.1.0`
to
`!pip install nlp`

It still breaks very near the end

![image](https://user-images.githubusercontent.com/338917/83312264-aa96a600-a1df-11ea-987f-2f4a0474247e.png)


When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.
If you don't restart, then it breaks like in your first message 

Thanks for reporting the second one ! We'll update the notebook to fix this one :)

This trick from @thomwolf seems to be the most reliable solution to fix this colab notebook issue:

```python
# install nlp
!pip install -qq nlp==0.2.0

# Make sure that we have a recent version of pyarrow in the session before we continue - otherwise reboot Colab to activate it
import pyarrow
if int(pyarrow.__version__.split('.')[1]) < 16:
    import os
    os.kill(os.getpid(), 9)
```

The second part got fixed here: 2cbc656d6fc4b18ce57eac070baec05b31180d39

Thanks! I'm then closing this issue.
Issue: ❓ How to get ROUGE-2 with the ROUGE metric ?

I'm trying to use ROUGE metric, but I don't know how to get the ROUGE-2 metric.

---

I compute scores with :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add([lp], [lg])
score = rouge.compute()
```

then : _(print only the F-score for readability)_

```python
for k, s in score.items():
    print(k, s.mid.fmeasure)
```

It gives :

>rouge1 0.7915168355671788
rougeL 0.7915168355671788

---

**How can I get the ROUGE-2 score ?**

Also, it's seems weird that ROUGE-1 and ROUGE-L scores are the same. Did I made a mistake ?

@lhoestq 

Comments:
ROUGE-1 and ROUGE-L shouldn't return the same thing. This is weird

For the rouge2 metric you can do

```python
rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add(lp, lg)
score = rouge.compute(rouge_types=["rouge2"])
```

Note that I just did a PR to have both `.add` and `.add_batch` for metrics, that's why now this is `rouge.add(lp, lg)` and not `rouge.add([lp], [lg])`

Well I just tested with the official script and both rouge1 and rougeL return exactly the same thing for the input you gave, so this is actually fine ^^

I hope it helped :)
Issue: Remove test set from NLP viewer

While the new [NLP viewer](https://huggingface.co/nlp/viewer/) is a great tool, I think it would be best to outright remove the option of looking at the test sets. At the very least, a warning should be displayed to users before showing the test set. Newcomers to the field might not be aware of best practices, and small things like this can help increase awareness.

Comments:
~is the viewer also open source?~
[is a streamlit app!](https://docs.streamlit.io/en/latest/getting_started.html)

Appears that [two thirds of those polled on Twitter](https://twitter.com/srush_nlp/status/1265734497632477185) are in favor of _some_ mechanism for averting eyeballs from the test data.

We do no longer use datasets-viewer.
Issue: [Question] Combine 2 datasets which have the same columns

Hi,

I am using ``nlp`` to load personal datasets. I created summarization datasets in multi-languages based on wikinews. I have one dataset for english and one for german (french is getting to be ready as well). I want to keep these datasets independent because they need different pre-processing (add different task-specific prefixes for T5 : *summarize:* for english and *zusammenfassen:* for german)

My issue is that I want to train T5 on the combined english and german datasets to see if it improves results. So I would like to combine 2 datasets (which have the same columns) to make one and train T5 on it. I was wondering if there is a proper way to do it? I assume that it can be done by combining all examples of each dataset but maybe you have a better solution.

Hoping this is clear enough,

Thanks a lot 😊
Best

Comments:
We are thinking about ways to combine datasets for T5 in #217, feel free to share your thoughts about this.

Ok great! I will look at it. Thanks
Issue: Mistaken `_KWARGS_DESCRIPTION` for XNLI metric

Hi!

The [`_KWARGS_DESCRIPTION`](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/xnli/xnli.py#L45) for the XNLI metric uses `Args` and `Returns` text from [BLEU](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/bleu/bleu.py#L58) metric:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: list of translations to score.
        Each translation should be tokenized into a list of tokens.
    references: list of lists of references for each translation.
        Each reference should be tokenized into a list of tokens.
    max_order: Maximum n-gram order to use when computing BLEU score.
    smooth: Whether or not to apply Lin et al. 2004 smoothing.
Returns:
    'bleu': bleu score,
    'precisions': geometric mean of n-gram precisions,
    'brevity_penalty': brevity penalty,
    'length_ratio': ratio of lengths,
    'translation_length': translation_length,
    'reference_length': reference_length
"""
```

But it should be something like:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: Predicted labels.
    references: Ground truth labels.
Returns:
    'accuracy': accuracy
```

Comments:
Indeed, good catch ! thanks
Fixing it right now
Issue: Index outside of table length

The offset input box warns of numbers larger than a limit (like 2000) but then the errors start at a smaller value than that limit (like 1955).

> ValueError: Index (2000) outside of table length (2000).
> Traceback:
> File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
>     exec(code, module.__dict__)
> File "/home/sasha/nlp_viewer/run.py", line 116, in <module>
>     v = d[item][k]
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
>     output_all_columns=self._output_all_columns,
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 290, in _getitem
>     raise ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")

Comments:
Sounds like something related to the nlp viewer @srush 

Fixed.
Issue: Scientific Papers only downloading Pubmed

Hi!

I have been playing around with this module, and I am a bit confused about the `scientific_papers` dataset. I thought that it would download two separate datasets, arxiv and pubmed. But when I run the following:

```
dataset = nlp.load_dataset('scientific_papers', data_dir='.', cache_dir='.')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.05k/5.05k [00:00<00:00, 2.66MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.90k/4.90k [00:00<00:00, 2.42MB/s]
Downloading and preparing dataset scientific_papers/pubmed (download: 4.20 GiB, generated: 2.33 GiB, total: 6.53 GiB) to ./scientific_papers/pubmed/1.1.1...
Downloading: 3.62GB [00:40, 90.5MB/s]
Downloading: 880MB [00:08, 101MB/s]
Dataset scientific_papers downloaded and prepared to ./scientific_papers/pubmed/1.1.1. Subsequent calls will reuse this data.
```

only a pubmed folder is created. There doesn't seem to be something for arxiv. Are these two datasets merged? Or have I misunderstood something?

Thanks!

Comments:
Hi so there are indeed two configurations in the datasets as you can see [here](https://github.com/huggingface/nlp/blob/master/datasets/scientific_papers/scientific_papers.py#L81-L82).

You can load either one with:
```python
dataset = nlp.load_dataset('scientific_papers', 'pubmed')
dataset = nlp.load_dataset('scientific_papers', 'arxiv')
```

This issues is actually related to a similar user-experience issue with GLUE. When several configurations are available and the first configuration is loaded by default (see issue #152 and #130), it seems to be unexpected for users.

I think we should maybe raise a (very explicit) error when there are several configurations available and the user doesn't specify one.

What do you think @lhoestq @patrickvonplaten @mariamabarham ?

Yes, it looks like the right thing to do 

Now if you don't specify which part you want, it raises an error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['pubmed', 'arxiv']
Example of usage:
	`load_dataset('scientific_papers', 'pubmed')`
```
Issue: [Question] Create Apache Arrow dataset from raw text file

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu


Comments:
We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the [pyarrow documentation](https://arrow.apache.org/docs/python/)

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in `/datasets`. Here is a link to the steps to [add a dataset](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)

Hello @mrm8488 and @lhoestq 

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :)

> Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:
- [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
- OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
- OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)


> > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?
> 
> Sure, to get a dataset in arrow format you can either:
> 
>     * [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
> 
>     * OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
> 
>     * OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

Links were broken. 

Updated links provided as below
- [load from local files (txt, json, csv)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-or-remote-files)
- [load from python data (dict, pandas)](https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data)
- [create your own dataset script](https://huggingface.co/docs/datasets/loading_datasets.html#using-a-custom-dataset-loading-script)
Issue: [Question] BERT-style multiple choice formatting

Hello, I am wondering what the equivalent formatting of a dataset should be to allow for multiple-choice answering prediction, BERT-style. Previously, this was done by passing a list of `InputFeatures` to the dataloader instead of a list of `InputFeature`, where `InputFeatures` contained lists of length equal to the number of answer choices in the MCQ instead of single items. I'm a bit confused on what the output of my feature conversion function should be when using `dataset.map()` to ensure similar behavior.

Thanks!

Comments:
Hi @sarahwie, can you details this a little more?

I'm not sure I understand what you refer to and what you mean when you say "Previously, this was done by passing a list of InputFeatures to the dataloader instead of a list of InputFeature"

I think I've resolved it. For others' reference: to convert from using the [`MultipleChoiceDataset` class](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/utils_multiple_choice.py#L82)/[`run_multiple_choice.py`](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/run_multiple_choice.py) script in Huggingface Transformers, I've done the following for hellaswag:

1. converted the `convert_examples_to_features()` function to only take one input and return a dictionary rather than a list:
```
def convert_examples_to_features(example, tokenizer, max_length):

    choices_inputs = defaultdict(list)
    for ending_idx, ending in enumerate(example['endings']['ending']):
        text_a = example['ctx']
        text_b = ending

        inputs = tokenizer.encode_plus(
            text_a,
            text_b,
            add_special_tokens=True,
            max_length=max_length,
            pad_to_max_length=True,
            return_overflowing_tokens=True,
        )
        if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
            logger.info(
                "Attention! you are cropping tokens (swag task is ok). "
                "If you are training ARC and RACE and you are poping question + options,"
                "you need to try to use a bigger max seq length!"
            )

        for key in inputs:
            choices_inputs[key].append(inputs[key])
            
        choices_inputs['label'] = int(example['label'])

    return choices_inputs
```
2. apply this directly (instance-wise) to dataset, convert dataset to torch tensors. Dataset is then ready to be passed to `Trainer` instance.

```
dataset['train'] = dataset['train'].map(lambda x: convert_examples_to_features(x, tokenizer, max_length), batched=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'label']
dataset['train'].set_format(type='torch', columns=columns)
```