ahmedfarazsyk/datasets-github-issues huggingface dataset:
Empty samples in glue/qqp
Author: richarddwang
State: closed

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string. 
BTW, I have checked and these two are the only naughty ones in all splits of qqp.

Comments: We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?

Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.
NonMatchingChecksumError when loading pubmed dataset
Author: DavideStenner
State: closed
Labels: name: dataset bug

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38 

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.


Comments: For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
PG-19
Author: lucidrains
State: closed
Labels: name: dataset request

Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.

Comments: Sounds good! Do you want to give it a try?

Ok, I'll see if I can figure it out tomorrow!

Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?

Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
c4 dataset is not viewable in nlpviewer demo
Author: rajarsheem
State: closed
Labels: name: nlp-viewer

I get the following error when I try to view the c4 dataset in [nlpviewer](https://huggingface.co/nlp/viewer/)

```python
ModuleNotFoundError: No module named 'langdetect'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 54, in <module>
    configs = get_confs(option.id)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 48, in get_confs
    builder_cls = nlp.load.import_main_class(module_path, dataset=True)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4.py", line 29, in <module>
    from .c4_utils import (
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4_utils.py", line 29, in <module>
    import langdetect
```

Comments: C4 is too large to be shown in the viewer
Error in metric.compute: missing `original_instructions` argument
Author: zphang
State: closed
Labels: name: metric bug

I'm running into an error using metrics for computation in the latest master as well as version 0.2.1. Here is a minimal example:

```python
import nlp
rte_metric = nlp.load_metric('glue', name="rte")
rte_metric.compute(
    [0, 0, 1, 1],
    [0, 1, 0, 1],
)
```

```
    181             # Read the predictions and references
    182             reader = ArrowReader(path=self.data_dir, info=None)
--> 183             self.data = reader.read_files(node_files)
    184 
    185             # Release all of our locks

TypeError: read_files() missing 1 required positional argument: 'original_instructions'
```

I believe this might have been introduced with cc8d2508b75f7ba0e5438d0686ee02dcec43c7f4, which added the `original_instructions` argument. Elsewhere, an empty-string default is provided--perhaps that could be done here too?
How can I load/find WMT en-romanian?
Author: sshleifer
State: closed

I believe it is from `wmt16`

When I run

```python
wmt = nlp.load_dataset('wmt16')
```
I get:
```python
AssertionError: The dataset wmt16 with config cs-en requires manual data. 
 Please follow the manual download instructions:   Some of the wmt configs here, require a manual download.
  Please look into wmt.py to see the exact path (and file name) that has to
  be downloaded.
  . 
 Manual data can be loaded with `nlp.load(wmt16, data_dir='<path/to/manual/data>')
```
There is no wmt.py,as the error message suggests, and wmt16.py doesn't have manual download instructions.

Any idea how to do this?

Thanks in advance!




Comments: I will take a look :-)
Why is dataset after tokenization far more larger than the orginal one ?
Author: richarddwang
State: closed

I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?

Comments: Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`

Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔

Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.

Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
[Feature request] Add a feature to dataset
Author: sarahwie
State: closed

Is there a straightforward way to add a field to the arrow_dataset, prior to performing map?

Comments: Do you have an example of what you would like to do? (you can just add a field in the output of the unction you give to map and this will add this field in the output table)

Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).

This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```

Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```

Oh great. That should work. I missed that in the documentation- thanks :)
[Feature request] Be able to remove a specific sample of the dataset
Author: astariul
State: closed

As mentioned in #117, it's currently not possible to remove a sample of the dataset.

But it is a important use case : After applying some preprocessing, some samples might be empty for example. We should be able to remove these samples from the dataset, or at least mark them as `removed` so when iterating the dataset, we don't iterate these samples.

I think it should be a feature. What do you think ?

---

Any work-around in the meantime ?

Comments: Oh yes you can now do that with the `dataset.filter()` method that was added in #214
NonMatchingSplitsSizesError error when reading the IMDB dataset
Author: antmarakis
State: closed

Hi!

I am trying to load the `imdb` dataset with this line:

`dataset = nlp.load_dataset('imdb', data_dir='/A/PATH', cache_dir='/A/PATH')`

but I am getting the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/load.py", line 517, in load_dataset
    save_infos=save_infos,
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 363, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 421, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=5929447, num_examples=4537, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

Am I overlooking something? Thanks!

Comments: I just tried on my side and I didn't encounter your problem.
Apparently the script doesn't generate all the examples on your side.

Can you provide the version of `nlp` you're using ?
Can you try to clear your cache and re-run the code ?

I updated it, that was it, thanks!

Hello, I am facing the same problem... how do you clear the huggingface cache?

Hi ! The cache is at ~/.cache/huggingface
You can just delete this folder if needed :)
What is the best way to cache a dataset? 
Author: Mistobaan
State: closed

For example if I want to use streamlit with a nlp dataset:

```
@st.cache
def load_data():
    return nlp.load_dataset('squad')
```
This code raises the error "uncachable object"

Right now I just fixed with a constant for my specific case:
```
    @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
```
But I was curious to know what is the best way in general



Comments: Everything is already cached by default in 🤗nlp (in particular dataset
loading and all the “map()” operations) so I don’t think you need to do any
specific caching in streamlit.

Tell us if you feel like it’s not the case.

On Sat, 6 Jun 2020 at 13:02, Fabrizio Milo <notifications@github.com> wrote:

> For example if I want to use streamlit with a nlp dataset:
>
> @st.cache
> def load_data():
>     return nlp.load_dataset('squad')
>
> This code raises the error "uncachable object"
>
> Right now I just fixed with a constant for my specific case:
>
>     @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
>
> But I was curious to know what is the best way in general
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/246>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHKAKO7CWGX2QY55UXLRVIO3ZANCNFSM4NV333RQ>
> .
>


Closing this one. Feel free to re-open if you have other questions !
UnicodeDecodeError when downloading GLUE-MNLI
Author: patpizio
State: closed

When I run
```python
dataset = nlp.load_dataset('glue', 'mnli')
```
I get an encoding error (could it be because I'm using Windows?) :
```python
# Lots of error log lines later...
~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\5256cc2368cf84497abef1f1a5f66648522d5854b225162148cb8fc78a5a91cc\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    529 
--> 530                 for n, row in enumerate(reader):
    531                     if is_cola_non_test:

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6744: character maps to <undefined>
```
Anyway this can be solved by specifying to decode in UTF when reading the csv file. I am proposing a PR if that's okay.

Comments: It should be good now, thanks for noticing and fixing it ! I would say that it was because you are on windows but not 100% sure

On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly. 
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in [PEP 540](https://www.python.org/dev/peps/pep-0540/))

For me this fixed the problem described by the OP.
Deterministic dataset loading
Author: patrickvonplaten
State: closed

When calling:
```python 
import nlp
dataset = nlp.load_dataset("trivia_qa", split="validation[:1%]")
```

the resulting dataset is not deterministic over different google colabs. 
After talking to @thomwolf, I suspect the reason to be the use of `glob.glob` in line:

https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/datasets/trivia_qa/trivia_qa.py#L180

which seems to return an ordering of files that depends on the filesystem:
https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered

I think we should go through all the dataset scripts and make sure to have deterministic behavior.

A simple solution for `glob.glob()` would be to just replace it with `sorted(glob.glob())` to have everything sorted by name. 

What do you think @lhoestq?

Comments: Yes good point !

I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it

I'm pretty sure it would solve the problem too.

The only other dataset that is not deterministic right now is `blog_authorship_corpus` (see #215) but this is a problem related to string encodings.

I think we should do the same also for `os.list_dir`
[Metric] Bertscore : Warning : Empty candidate sentence; Setting recall to be 0.
Author: astariul
State: closed
Labels: name: metric bug

When running BERT-Score, I'm meeting this warning :

> Warning: Empty candidate sentence; Setting recall to be 0.

Code :

```
import nlp
metric = nlp.load_metric("bertscore")
scores = metric.compute(["swag", "swags"], ["swags", "totally something different"], lang="en", device=0)
```

---

**What am I doing wrong / How can I hide this warning ?**

Comments: This print statement comes from the official implementation of bert_score (see [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L343)). The warning shows up only if the attention mask outputs no candidate.
Right now we want to only use official code for metrics to have fair evaluations, so I'm not sure we can do anything about it. Maybe you can try to create an issue on their [repo](https://github.com/Tiiiger/bert_score) ?
Huggingface NLP, Uploading custom dataset
Author: Nouman97
State: closed

Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!

Comments: What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
`nlp.Dataset.from_file` and `load_dataset(..., cache_dir=)`

To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`

Also if you want to upload your script, you should be able to use the `nlp-cli`.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.


Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```