helmo/github-issues huggingface dataset:
Not able to access the XNLI dataset
State: closed | Author: aswin-giridhar (NONE)
Labels: name: nlp-viewer

When I try to access the XNLI dataset, I get the following error. The option of plain_text get selected automatically and then I get the following error.

```
FileNotFoundError: [Errno 2] No such file or directory: '/home/sasha/.cache/huggingface/datasets/xnli/plain_text/1.0.0/dataset_info.json'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 86, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 72, in get
    builder_instance = builder_cls(name=conf)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 610, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 152, in __init__
    self.info = DatasetInfo.from_directory(self._cache_dir)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/info.py", line 157, in from_directory
    with open(os.path.join(dataset_info_dir, DATASET_INFO_FILENAME), "r") as f:
```

Is it possible to see if the dataset_info.json is correctly placed?

Comments: Added pull request to change the name of the file from dataset_infos.json to dataset_info.json

Thanks for reporting this bug !
As it seems to be just a cache problem, I closed your PR.
I think we might just need to clear and reload the `xnli` cache @srush ? 

Update: The dataset_info.json error is gone, but we have a new one instead:
```
ConnectionError: Couldn't reach https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip
```
I am not able to reproduce on my side unfortunately. Any idea @srush ?

xnli is now properly shown in the viewer.
Closing this one.
Should we still have to force to install apache_beam to download wikipedia ?
State: closed | Author: richarddwang (CONTRIBUTOR)

Hi, first thanks to @lhoestq 's revolutionary work, I successfully downloaded processed wikipedia according to the doc. 😍😍😍

But at the first try, it tell me to install `apache_beam` and `mwparserfromhell`, which I thought wouldn't be used according to #204 , it was kind of confusing me at that time.

Maybe we should not force users to install these ? Or we just add them to`nlp`'s dependency ?

Comments: Thanks for your message 😊 
Indeed users shouldn't have to install those dependencies

Got it, feel free to close this issue when you think it’s resolved.

It should be good now :)
[Feature Request/Help] BLEURT model -> PyTorch
State: closed | Author: adamwlev (NONE)
Labels: name: enhancement

Hi, I am interested in porting google research's new BLEURT learned metric to PyTorch (because I wish to do something experimental with language generation and backpropping through BLEURT). I noticed that you guys don't have it yet so I am partly just asking if you plan to add it (@thomwolf said you want to do so on Twitter).

I had a go of just like manually using the checkpoint that they publish which includes the weights. It seems like the architecture is exactly aligned with the out-of-the-box BertModel in transformers just with a single linear layer on top of the CLS embedding. I loaded all the weights to the PyTorch model but I am not able to get the same numbers as the BLEURT package's python api. Here is my colab notebook where I tried  https://colab.research.google.com/drive/1Bfced531EvQP_CpFvxwxNl25Pj6ptylY?usp=sharing . If you have any pointers on what might be going wrong that would be much appreciated!

Thank you muchly!

Comments: Is there any update on this? 

Thanks!

Hitting this error when using bleurt with PyTorch ...

```
UnrecognizedFlagError: Unknown command line flag 'f'
```
... and I'm assuming because it was built for TF specifically.  Is there a way to use this metric in PyTorch?

We currently provide a wrapper on the TensorFlow implementation: https://huggingface.co/metrics/bleurt

We have long term plans to better handle model-based metrics, but they probably won't be implemented right away

@adamwlev it would still be cool to add the BLEURT checkpoints to the transformers repo if you're interested, but that would best be discussed there :) 

closing for now

Hi there. We ran into the same problem this year (converting BLEURT to PyTorch) and thanks to @adamwlev found his colab notebook which didn't work but served as a good starting point. Finally, we **made it work** by doing just two simple conceptual fixes: 

1. Transposing 'kernel' layers instead of 'dense' ones when copying params from the original model;
2. Taking pooler_output as a cls_state in forward function of the BleurtModel class.

Plus few minor syntactical fixes for the outdated parts. The result is still not exactly the same, but is very close to the expected one (1.0483 vs 1.0474).

Find the fixed version here (fixes are commented): https://colab.research.google.com/drive/1KsCUkFW45d5_ROSv2aHtXgeBa2Z98r03?usp=sharing  


I created a new model based on `transformers` that can load every BLEURT checkpoints released so far. https://github.com/lucadiliello/bleurt-pytorch

@LoraIpsum Thanks for sharing your work here. However, I'm unable to reproduce the results. That's strange because you are. FYI, I am trying to convert a finetuned BLEURT to PyTorch. Any suggestions on how I can reproduce results?
Colab Notebook breaks when downloading the squad dataset
State: closed | Author: carlos-aguayo (NONE)

When I run the notebook in Colab
https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb
breaks when running this cell:
![image](https://user-images.githubusercontent.com/338917/83311709-ffd1b800-a1dd-11ea-8394-3a87df0d7f8b.png)


Comments: The notebook forces version 0.1.0. If I use the latest, things work, I'll run the whole notebook and create a PR.

But in the meantime, this issue gets fixed by changing:
`!pip install nlp==0.1.0`
to
`!pip install nlp`

It still breaks very near the end

![image](https://user-images.githubusercontent.com/338917/83312264-aa96a600-a1df-11ea-987f-2f4a0474247e.png)


When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.
If you don't restart, then it breaks like in your first message 

Thanks for reporting the second one ! We'll update the notebook to fix this one :)

This trick from @thomwolf seems to be the most reliable solution to fix this colab notebook issue:

```python
# install nlp
!pip install -qq nlp==0.2.0

# Make sure that we have a recent version of pyarrow in the session before we continue - otherwise reboot Colab to activate it
import pyarrow
if int(pyarrow.__version__.split('.')[1]) < 16:
    import os
    os.kill(os.getpid(), 9)
```

The second part got fixed here: 2cbc656d6fc4b18ce57eac070baec05b31180d39

Thanks! I'm then closing this issue.
Fix tests/test_dataset_common.py
State: closed | Author: tayciryahmed (CONTRIBUTOR)

When I run the command `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_arcd` while working on #220. I get the error ` unexpected keyword argument "'download_and_prepare_kwargs'"` at the level of  `load_dataset`. Indeed, this [function](https://github.com/huggingface/nlp/blob/master/src/nlp/load.py#L441) no longer has the argument `download_and_prepare_kwargs` but rather `download_config`. So here I change the tests accordingly. 

Comments: Thanks ! Good catch :)

To fix the CI you can do:
1 - rebase from master
2 - then run `make style` as specified in [CONTRIBUTING.md](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md) ?
❓ How to get ROUGE-2 with the ROUGE metric ?
State: closed | Author: astariul (NONE)

I'm trying to use ROUGE metric, but I don't know how to get the ROUGE-2 metric.

---

I compute scores with :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add([lp], [lg])
score = rouge.compute()
```

then : _(print only the F-score for readability)_

```python
for k, s in score.items():
    print(k, s.mid.fmeasure)
```

It gives :

>rouge1 0.7915168355671788
rougeL 0.7915168355671788

---

**How can I get the ROUGE-2 score ?**

Also, it's seems weird that ROUGE-1 and ROUGE-L scores are the same. Did I made a mistake ?

@lhoestq 

Comments: ROUGE-1 and ROUGE-L shouldn't return the same thing. This is weird

For the rouge2 metric you can do

```python
rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add(lp, lg)
score = rouge.compute(rouge_types=["rouge2"])
```

Note that I just did a PR to have both `.add` and `.add_batch` for metrics, that's why now this is `rouge.add(lp, lg)` and not `rouge.add([lp], [lg])`

Well I just tested with the official script and both rouge1 and rougeL return exactly the same thing for the input you gave, so this is actually fine ^^

I hope it helped :)
better message if missing beam options
State: closed | Author: lhoestq (MEMBER)

WDYT @yjernite ?
For example:
```python
dataset = nlp.load_dataset('wikipedia', '20200501.aa')
```
Raises:
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20200501.aa', beam_runner='DirectRunner')`
```
Add a Google Drive exception for small files
State: closed | Author: airKlizz (CONTRIBUTOR)

I tried to use the ``nlp`` library to load personnal datasets. I mainly copy-paste the code for ``multi-news`` dataset because my files are stored on Google Drive. 

One of my dataset is small (< 25Mo) so it can be verified by Drive without asking the authorization to the user. This makes the download starts directly. 

Currently the ``nlp`` raises a error: ``ConnectionError: Couldn't reach https://drive.google.com/uc?export=download&id=1DGnbUY9zwiThTdgUvVTSAvSVHoloCgun`` while the url is working. So I just add a new exception as you have already done for  ``firebasestorage.googleapis.com`` : 

```
elif (response.status_code == 400 and "firebasestorage.googleapis.com" in url) or (response.status_code == 405 and "drive.google.com" in url)
```

I make an example of the error that you can run on [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ae_JJ9uvUt-9GBh0uGZhjbF5aXkl-BPv?usp=sharing)

I avoid the error by adding an exception but there is maybe a proper way to do it.

Many thanks :hugs:
Best,

Comments: Can you run the style formatting tools to pass the code quality test?

You can find all the details in CONTRIBUTING.md: https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-contribute-to-nlp

Nice ! 

``make style`` done! Thanks for the approvals.
Remove test set from NLP viewer
State: closed | Author: chrisdonahue (NONE)
Labels: name: nlp-viewer

While the new [NLP viewer](https://huggingface.co/nlp/viewer/) is a great tool, I think it would be best to outright remove the option of looking at the test sets. At the very least, a warning should be displayed to users before showing the test set. Newcomers to the field might not be aware of best practices, and small things like this can help increase awareness.

Comments: ~is the viewer also open source?~
[is a streamlit app!](https://docs.streamlit.io/en/latest/getting_started.html)

Appears that [two thirds of those polled on Twitter](https://twitter.com/srush_nlp/status/1265734497632477185) are in favor of _some_ mechanism for averting eyeballs from the test data.

We do no longer use datasets-viewer.
[Question] Combine 2 datasets which have the same columns
State: closed | Author: airKlizz (CONTRIBUTOR)

Hi,

I am using ``nlp`` to load personal datasets. I created summarization datasets in multi-languages based on wikinews. I have one dataset for english and one for german (french is getting to be ready as well). I want to keep these datasets independent because they need different pre-processing (add different task-specific prefixes for T5 : *summarize:* for english and *zusammenfassen:* for german)

My issue is that I want to train T5 on the combined english and german datasets to see if it improves results. So I would like to combine 2 datasets (which have the same columns) to make one and train T5 on it. I was wondering if there is a proper way to do it? I assume that it can be done by combining all examples of each dataset but maybe you have a better solution.

Hoping this is clear enough,

Thanks a lot 😊
Best

Comments: We are thinking about ways to combine datasets for T5 in #217, feel free to share your thoughts about this.

Ok great! I will look at it. Thanks
Better arrow dataset iter
State: closed | Author: lhoestq (MEMBER)

I tried to play around with `tf.data.Dataset.from_generator` and I found out that the `__iter__` that we have for `nlp.arrow_dataset.Dataset` ignores the format that has been set (torch or tensorflow).
With these changes I should be able to come up with a `tf.data.Dataset` that uses lazy loading, as asked in #193.
Add Dataflow support + Wikipedia + Wiki40b
State: closed | Author: lhoestq (MEMBER)

# Add Dataflow support + Wikipedia + Wiki40b

## Support datasets processing with Apache Beam

Some datasets are too big to be processed on a single machine, for example: wikipedia, wiki40b, etc. Apache Beam allows to process datasets on many execution engines like Dataflow, Spark, Flink, etc.

To process such datasets with Beam, I added a command to run beam pipelines `nlp-cli run_beam path/to/dataset/script`. Then I used it to process the english + french wikipedia, and the english of wiki40b.
The processed arrow files are on GCS and are the result of a Dataflow job.

I added a markdown documentation file in `docs` that explains how to use it properly.

## Load already processed datasets

Now that we have those datasets already processed, I made it possible to load datasets that are already processed. You can do `load_dataset('wikipedia', '20200501.en')` and it will download the processed files from the Hugging Face GCS directly into the user's cache and be ready to use !

The Wikipedia dataset was already asked in #187 and this PR should soon allow to add Natural Questions as asked in #129 

## Other changes in the code

To make things work, I had to do a few adjustments:
- add a `ship_files_with_pipeline` method to the `DownloadManager`. This is because beam pipelines can be run in the cloud and therefore need to have access to your downloaded data. I used it in the wikipedia script:
    ```python
    if not pipeline.is_local():
            downloaded_files = dl_manager.ship_files_with_pipeline(downloaded_files, pipeline)
    ```
- add parquet to arrow conversion. This is because the output of beam pipelines are parquet files so we need to convert them to arrow and have the arrow files on GCS
- add a test script with a dummy beam dataset
- minor adjustments to allow read/write operations on remote files using `apache_beam.io.filesystems.FileSystems` if we want (it can be connected to gcp, s3, hdfs, etc...)
Raise an error if no config name for datasets like glue
State: closed | Author: lhoestq (MEMBER)

Some datasets like glue (see #130) and scientific_papers (see #197) have many configs.
For example for glue there are cola, sst2, mrpc etc.

Currently if a user does `load_dataset('glue')`, then Cola is loaded by default and it can be confusing. Instead, we should raise an error to let the user know that he has to pick one of the available configs (as proposed in #152). For example for glue, the message looks like:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```

The error is raised if the config name is missing and if there are >=2 possible configs.
Mistaken `_KWARGS_DESCRIPTION` for XNLI metric
State: closed | Author: phiyodr (NONE)

Hi!

The [`_KWARGS_DESCRIPTION`](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/xnli/xnli.py#L45) for the XNLI metric uses `Args` and `Returns` text from [BLEU](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/bleu/bleu.py#L58) metric:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: list of translations to score.
        Each translation should be tokenized into a list of tokens.
    references: list of lists of references for each translation.
        Each reference should be tokenized into a list of tokens.
    max_order: Maximum n-gram order to use when computing BLEU score.
    smooth: Whether or not to apply Lin et al. 2004 smoothing.
Returns:
    'bleu': bleu score,
    'precisions': geometric mean of n-gram precisions,
    'brevity_penalty': brevity penalty,
    'length_ratio': ratio of lengths,
    'translation_length': translation_length,
    'reference_length': reference_length
"""
```

But it should be something like:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: Predicted labels.
    references: Ground truth labels.
Returns:
    'accuracy': accuracy
```

Comments: Indeed, good catch ! thanks
Fixing it right now
[ArrowWriter] Set schema at first write example
State: closed | Author: lhoestq (MEMBER)

Right now if the schema was not specified when instantiating `ArrowWriter`, then it could be set with the first `write_table` for example (it calls `self._build_writer()` to do so).

I noticed that it was not done if the first example is added via `.write`, so I added it for coherence.

Comments: Good point!

I guess we could add this to `write_batch` as well (before using `self._schema` in the first line of this method)?
Index outside of table length
State: closed | Author: casajarm (NONE)

The offset input box warns of numbers larger than a limit (like 2000) but then the errors start at a smaller value than that limit (like 1955).

> ValueError: Index (2000) outside of table length (2000).
> Traceback:
> File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
>     exec(code, module.__dict__)
> File "/home/sasha/nlp_viewer/run.py", line 116, in <module>
>     v = d[item][k]
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
>     output_all_columns=self._output_all_columns,
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 290, in _getitem
>     raise ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")

Comments: Sounds like something related to the nlp viewer @srush 

Fixed.
Scientific Papers only downloading Pubmed
State: closed | Author: antmarakis (NONE)

Hi!

I have been playing around with this module, and I am a bit confused about the `scientific_papers` dataset. I thought that it would download two separate datasets, arxiv and pubmed. But when I run the following:

```
dataset = nlp.load_dataset('scientific_papers', data_dir='.', cache_dir='.')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.05k/5.05k [00:00<00:00, 2.66MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.90k/4.90k [00:00<00:00, 2.42MB/s]
Downloading and preparing dataset scientific_papers/pubmed (download: 4.20 GiB, generated: 2.33 GiB, total: 6.53 GiB) to ./scientific_papers/pubmed/1.1.1...
Downloading: 3.62GB [00:40, 90.5MB/s]
Downloading: 880MB [00:08, 101MB/s]
Dataset scientific_papers downloaded and prepared to ./scientific_papers/pubmed/1.1.1. Subsequent calls will reuse this data.
```

only a pubmed folder is created. There doesn't seem to be something for arxiv. Are these two datasets merged? Or have I misunderstood something?

Thanks!

Comments: Hi so there are indeed two configurations in the datasets as you can see [here](https://github.com/huggingface/nlp/blob/master/datasets/scientific_papers/scientific_papers.py#L81-L82).

You can load either one with:
```python
dataset = nlp.load_dataset('scientific_papers', 'pubmed')
dataset = nlp.load_dataset('scientific_papers', 'arxiv')
```

This issues is actually related to a similar user-experience issue with GLUE. When several configurations are available and the first configuration is loaded by default (see issue #152 and #130), it seems to be unexpected for users.

I think we should maybe raise a (very explicit) error when there are several configurations available and the user doesn't specify one.

What do you think @lhoestq @patrickvonplaten @mariamabarham ?

Yes, it looks like the right thing to do 

Now if you don't specify which part you want, it raises an error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['pubmed', 'arxiv']
Example of usage:
	`load_dataset('scientific_papers', 'pubmed')`
```
Add Dataset: Qanta
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

Fixes dummy data for #169 @EntilZha

Comments: @lhoestq - the config name is rather special here: *E.g.* `mode=first,char_skip=25`. It includes `=` and `,` - will that be a problem for windows folders, you think? 

Apart from that good to merge for me.

It's ok to have `=` and `,`.
Windows doesn't like things like `?`, `:`, `/` etc.

I'll add some lines to raise an error if the config name is invalid.

Thanks for fixing things up! I'm curious to take a look at the zip files now to know the format for future reference.