helmo/github-issues huggingface dataset:
Fix manual download instructions
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

This PR replaces the static `DatasetBulider` variable `MANUAL_DOWNLOAD_INSTRUCTIONS` by a property function `manual_download_instructions()`. 

Some datasets like XTREME and all WMT need the manual data dir only for a small fraction of the possible configs.

After some brainstorming with @mariamabarham and @lhoestq, we came to the conclusion that having a property function `manual_download_instructions()` gives us more flexibility to decide on a per config basis in the dataset builder if manual download instructions are needed.

Also this PR should unblock solves a bug with `wmt16 - ro-en` 
@sshleifer from this branch you should be able to succesfully run

```python 
import nlp 
ds = nlp.load_dataset('./datasets/wmt16', 'ro-en')
```

and once this PR is merged S3 should be synched so that 

```python
import nlp
ds = nlp.load_dataset("wmt16", "ro-en")
```

works as well.

**Important**: Since `MANUAL_DOWNLOAD_INSTRUCTIONS` was not really exposed to the user, this PR should not be a problem regarding backward compatibility.

Comments: Verified that this works, thanks!

But I get
```python
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
```
When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).


Both machines can run
```bash
aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
```
but it seems one must be in the nlp directory to run the command?

(I ran `pip install -e . ` on this branch in both situations.)




`https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py` looks very weird.

(Also, S3 is not a file-system, it's a flat key-value store)

Good to merge I think @lhoestq 

> But I get
> 
> ```python
> ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
> ```
> 
> When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).
> 
> Both machines can run
> 
> ```shell
> aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
> ```
> 
> but it seems one must be in the nlp directory to run the command?
> 
> (I ran `pip install -e . ` on this branch in both situations.)

As soon as it is on master, the dataset script wmt16.py will be synced on S3 and you'll be able to do `load_dataset("wmt16")`
Update dataset_info from gcs
State: closed | Author: lhoestq (MEMBER)

Some datasets are hosted on gcs (wikipedia for example). In this PR I make sure that, when a user loads such datasets, the file_instructions are built using the dataset_info.json from gcs and not from the info extracted from the local `dataset_infos.json` (the one that contain the info for each config). Indeed local files may end up outdated.

Furthermore, to avoid outdated dataset_infos.json, I now make sure that each time you run `load_dataset` it also tries to update the file locally.
Private/sensitive data
State: closed | Author: MFreidank (CONTRIBUTOR)

Hi all,
Thanks for this fantastic library, it makes it very easy to do prototyping for NLP projects interchangeably between TF/Pytorch. 

Unfortunately, there is data that cannot easily be shared publicly as it may contain sensitive information. 
Is there support/a plan to support such data with NLP, e.g. by reading it from local sources?

Use case flow could look like this: use NLP to prototype an approach on similar, public data and apply the resulting prototype on sensitive/private data without the need to rethink data processing pipelines. 

Many thanks for your responses ahead of time and kind regards,
MFreidank

Comments: Hi @MFreidank, you should already be able to load a dataset from local sources, indeed. (ping @lhoestq and @jplu)

We're also thinking about the ability to host private datasets on a hosted bucket with permission management, but that's further down the road.

Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.

Hi @julien-c @jplu,
Thanks for sharing this solution with me, it helps, this is what I was looking for. 
If not already there and only missed by me, this could be a great addition in the docs.

Closing my issue as resolved, thanks again.
Error with SquadV2 Metrics
State: closed | Author: avinregmi (NONE)

I can't seem to import squad v2 metrics. 

**squad_metric = nlp.load_metric('squad_v2')**

**This throws me an error.:**


```
ImportError                               Traceback (most recent call last)
<ipython-input-8-170b6a170555> in <module>
----> 1 squad_metric = nlp.load_metric('squad_v2')

~/env/lib64/python3.6/site-packages/nlp/load.py in load_metric(path, name, process_id, num_process, data_dir, experiment_id, in_memory, download_config, **metric_init_kwargs)
    426     """
    427     module_path = prepare_module(path, download_config=download_config, dataset=False)
--> 428     metric_cls = import_main_class(module_path, dataset=False)
    429     metric = metric_cls(
    430         name=name,

~/env/lib64/python3.6/site-packages/nlp/load.py in import_main_class(module_path, dataset)
     55     """
     56     importlib.invalidate_caches()
---> 57     module = importlib.import_module(module_path)
     58 
     59     if dataset:

/usr/lib64/python3.6/importlib/__init__.py in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127 
    128 

/usr/lib64/python3.6/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib64/python3.6/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib64/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~/env/lib64/python3.6/site-packages/nlp/metrics/squad_v2/a15e787c76889174874386d3def75321f0284c11730d2a57e28fe1352c9b5c7a/squad_v2.py in <module>
     16 
     17 import nlp
---> 18 from .evaluate import evaluate
     19 
     20 _CITATION = """\

ImportError: cannot import name 'evaluate'
```
Dataset Preprocessing Cache with .map() function not working as expected
State: closed | Author: sarahwie (NONE)

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system. 

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file. 

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess. 
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!

Comments: When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?

Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.

Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?

Hi @sarahwie 
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?

I can't reproduce it, so I'll close for now.
Empty samples in glue/qqp
State: closed | Author: richarddwang (CONTRIBUTOR)

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string. 
BTW, I have checked and these two are the only naughty ones in all splits of qqp.

Comments: We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?

Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.
Fix metric compute (original_instructions missing)
State: closed | Author: lhoestq (MEMBER)

When loading arrow data we added in cc8d250 a way to specify the instructions that were used to store them with the loaded dataset.
However metrics load data the same way but don't need instructions (we use one single file).

In this PR I just make `original_instructions` optional when reading files to load a `Dataset` object.

This should fix #269 

Comments: Awesome! This is working now:

```python
import nlp 
seqeval = nlp.load_metric("seqeval") 
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']] 
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]                                                                                                                                                                                    

results = seqeval.compute(y_true, y_pred)
```

I heavily need this fix for an upcoming `nlp` integration PR for Transformers (token classification example)  😅

Haha nice ! We'll ship this fix with the next release that will probably come out on thursday :)
NonMatchingChecksumError when loading pubmed dataset
State: closed | Author: DavideStenner (NONE)
Labels: name: dataset bug

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38 

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.


Comments: For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
PG-19
State: closed | Author: lucidrains (CONTRIBUTOR)
Labels: name: dataset request

Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.

Comments: Sounds good! Do you want to give it a try?

Ok, I'll see if I can figure it out tomorrow!

Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?

Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
Fix allociné dataset configuration
State: closed | Author: TheophileBlard (CONTRIBUTOR)

This is a patch for #244. According to the [live nlp viewer](url), the Allociné dataset must be loaded with :
```python
dataset = load_dataset('allocine', 'allocine')
```
This is redundant, as there is only one "dataset configuration", and should only be:
```python
dataset = load_dataset('allocine')
```

This is my mistake, because the code for [`allocine.py`](https://github.com/huggingface/nlp/blob/master/datasets/allocine/allocine.py) was inspired by [`imdb.py`](https://github.com/huggingface/nlp/blob/master/datasets/imdb/imdb.py), which also force the user to specify the "dataset configuration" (even if there is only one).

I believe this PR should solve this issue, making the Allociné dataset more convenient to use.

Comments: Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
```python
dataset = load_dataset('allocine')
```
and it works.

Maybe we should take that into account in the nlp viewer @srush ?

@lhoestq Just to understand the exact semantics. Are you suggesting that if there is exactly 1 configuration I should not show the configuration menu and just treat it as if there were 0 configurations? 

The configuration menu is fine imo.
It was more about the code snippet presented in the viewer.
For example for Allociné it currently shows this snippet to load the dataset:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine', 'allocine')
```
However for datasets with one or zero configurations, the second argument in `load_dataset` is optional. For Allociné, that has one configuration, we can expect to show instead:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine')
```

> Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
> 
> ```python
> dataset = load_dataset('allocine')
> ```
> 
> and it works.
> 
> Maybe we should take that into account in the nlp viewer @srush ?

Oh ok, I didn't expect it would work! 

Anyway, I think it's intrinsically better to simply remove the optional parameter. 
The dummy data folder architecture seems also more logical this way.


Fixed in the viewer. Checked that allocine works.

Awesome thanks :)

Closing this.
c4 dataset is not viewable in nlpviewer demo
State: closed | Author: rajarsheem (NONE)
Labels: name: nlp-viewer

I get the following error when I try to view the c4 dataset in [nlpviewer](https://huggingface.co/nlp/viewer/)

```python
ModuleNotFoundError: No module named 'langdetect'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 54, in <module>
    configs = get_confs(option.id)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 48, in get_confs
    builder_cls = nlp.load.import_main_class(module_path, dataset=True)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4.py", line 29, in <module>
    from .c4_utils import (
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4_utils.py", line 29, in <module>
    import langdetect
```

Comments: C4 is too large to be shown in the viewer
Error in metric.compute: missing `original_instructions` argument
State: closed | Author: zphang (NONE)
Labels: name: metric bug

I'm running into an error using metrics for computation in the latest master as well as version 0.2.1. Here is a minimal example:

```python
import nlp
rte_metric = nlp.load_metric('glue', name="rte")
rte_metric.compute(
    [0, 0, 1, 1],
    [0, 1, 0, 1],
)
```

```
    181             # Read the predictions and references
    182             reader = ArrowReader(path=self.data_dir, info=None)
--> 183             self.data = reader.read_files(node_files)
    184 
    185             # Release all of our locks

TypeError: read_files() missing 1 required positional argument: 'original_instructions'
```

I believe this might have been introduced with cc8d2508b75f7ba0e5438d0686ee02dcec43c7f4, which added the `original_instructions` argument. Elsewhere, an empty-string default is provided--perhaps that could be done here too?
How can I load/find WMT en-romanian?
State: closed | Author: sshleifer (CONTRIBUTOR)

I believe it is from `wmt16`

When I run

```python
wmt = nlp.load_dataset('wmt16')
```
I get:
```python
AssertionError: The dataset wmt16 with config cs-en requires manual data. 
 Please follow the manual download instructions:   Some of the wmt configs here, require a manual download.
  Please look into wmt.py to see the exact path (and file name) that has to
  be downloaded.
  . 
 Manual data can be loaded with `nlp.load(wmt16, data_dir='<path/to/manual/data>')
```
There is no wmt.py,as the error message suggests, and wmt16.py doesn't have manual download instructions.

Any idea how to do this?

Thanks in advance!




Comments: I will take a look :-)