mengta666/huggingface_github huggingface dataset:
Issue: datasets.load_dataset() custom chaching directory bug

Hello,
I'm having issue with loading a dataset with a custom `cache_dir`. Despite specifying the output dir, it is still downloaded to 
 `~/.cache`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
from pathlib import Path

validation_dataset = datasets.load_dataset("natural_questions", split="validation[:5%]", cache_dir=Path("./data"))  
```

## The output:

* The dataset is downloaded to my home directory's `.cache` 
* A new empty directory named "`natural_questions` is created in the specified directory `.data`
* `tree data` in the shell outputs:
```
data
└── natural_questions
    └── default
        └── 0.0.2
3 directories, 0 files
```

The output:
```
Downloading: 8.61kB [00:00, 5.11MB/s]                                                                                                                                                                              
Downloading: 13.6kB [00:00, 7.89MB/s]                                                                                                                                                                              
Using custom data configuration default                                                                                                                                                                            
Downloading and preparing dataset natural_questions/default (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size, total: 134.92 GiB) to ./data/natural_questions/default/0.0.2/867dbbaf9137c1b8
3ecb19f5eb80559e1002ea26e702c6b919cfa81a17a8c531...                                                                                                                                                                
Downloading: 100%|██████████████████████████████████████████████████| 13.6k/13.6k [00:00<00:00, 1.51MB/s]                                                                                                          
Downloading:   7%|███▎                                            | 6.70G/97.4G [03:46<1:37:05, 15.6MB/s]
```

## Expected behaviour:
The dataset "Natural Questions" should be downloaded to the directory "./data"


Comments:
Thanks for reporting ! I'm looking into it.
Issue: Dataset viewer issues

I was looking through the dataset viewer and I like it a lot. Version numbers, citation information, everything's there! I've spotted a few issues/bugs though:

- the URL is still under `nlp`, perhaps an alias for `datasets` can be made
- when I remove a **feature** (and the feature list is empty), I get an error. This is probably expected, but perhaps a better error message can be shown to the user

```bash
IndexError: list index out of range
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 316, in <module>
    st.table(style)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 122, in wrapped_method
    return dg._enqueue_new_element_delta(marshall_element, delta_type, last_index)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 367, in _enqueue_new_element_delta
    rv = marshall_element(msg.delta.new_element)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 120, in marshall_element
    return method(dg, element, *args, **kwargs)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 2944, in table
    data_frame_proto.marshall_data_frame(data, element.table)
File "/home/sasha/streamlit/lib/streamlit/elements/data_frame_proto.py", line 54, in marshall_data_frame
    _marshall_styles(proto_df.style, df, styler)
File "/home/sasha/streamlit/lib/streamlit/elements/data_frame_proto.py", line 73, in _marshall_styles
    translated_style = styler._translate()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/pandas/io/formats/style.py", line 351, in _translate
    * (len(clabels[0]) - len(hidden_columns))
```

- there seems to be **an encoding issue** in the default view, the dataset examples are shown as raw monospace text, without a decent encoding. That makes it hard to read for languages that use a lot of special characters. Take for instance the [cs-en WMT19 set](https://huggingface.co/nlp/viewer/?dataset=wmt19&config=cs-en). This problem goes away when you enable "List view", because then some syntax highlighteris used, and the special characters are coded correctly.


Comments:
Thanks for reporting !
cc @srush for the empty feature list issue and the encoding issue
cc @julien-c maybe we can update the url and just have a redirection from the old url to the new one ?

Ok, I redirected on our side to a new url. ⚠️ @srush: if you update the Streamlit config too to `/datasets/viewer`, let me know because I'll need to change our nginx config at the same time

9

‏⠀‏‏‏⠀‏‏‏⠀ ‏⠀ 

‏⠀‏‏‏⠀‏‏‏⠀ ‏⠀
Issue: Nested lists are zipped unexpectedly

I might misunderstand something, but I expect that if I define:
```python
"top": datasets.features.Sequence({
  "middle": datasets.features.Sequence({
    "bottom": datasets.Value("int32")
  })
})
```

And I then create an example:
```python
yield 1, {
  "top": [{
    "middle": [
      {"bottom": 1},
      {"bottom": 2}
    ]
  }]
}
```

I then load my dataset:
```python
train = load_dataset("my dataset")["train"]
```

and expect to be able to access `data[0]["top"][0]["middle"][0]`.

That is not the case. Here is `data[0]` as JSON:

```json
{"top": {"middle": [{"bottom": [1, 2]}]}}
```

Clearly different than the thing I inputted.
```json
{"top": [{"middle": [{"bottom": 1},{"bottom": 2}]}]}
```

Comments:
Yes following the Tensorflow Datasets convention, objects with type `Sequence of a Dict` are actually stored as a `dictionary of lists`.
See the [documentation](https://huggingface.co/docs/datasets/features.html?highlight=features) for more details

Thanks.
This is a bit (very) confusing, but I guess if its intended, I'll just work with it as if its how my data was originally structured :)
Issue: Very slow cold-start

Hi,
I expect when importing `datasets` that nothing major happens in the background, and so the import should be insignificant.
When I load a metric, or a dataset, its fine that it takes time.

The following ranges from 3 to 9 seconds:
```
python -m timeit -n 1 -r 1 'from datasets import load_dataset'
```

edit:
sorry for the mis-tag, not sure how I added it.

Comments:
Good point!

Yes indeed. We can probably improve that by using lazy imports

#1690 added fast start-up of the library
Issue: Downloading/caching only a part of a datasets' dataset.

Hi,
I want to use the validation data *only* (of natural question).
I don't want to have the whole dataset cached in my machine, just the dev set.
Is this possible? I can't find a way to do it in the docs.

Thank you,
Sapir

Comments:
Not at the moment but we could likely support this feature.

?

I think it would be a very helpful feature, because sometimes one only wants to evaluate models on the dev set, and the whole training data may be many times bigger.
This makes the task impossible with limited memory resources.
Issue: DataLoader(datasets) become more and more slowly within iterations

Hello, when I for loop my dataloader, the loading speed is becoming more and more slowly!
```
dataset = load_from_disk(dataset_path)  # around 21,000,000 lines

lineloader = tqdm(DataLoader(dataset, batch_size=1))
for idx, line in enumerate(lineloader):
     # do some thing for each line
```
In the begining, the loading speed is around 2000it/s, but after 1 minutes later, the speed is much slower, just around 800it/s.

And when I set `num_workers=4` in DataLoader, the loading speed is much lower, just 130it/s.

Could you please help me with this problem?
Thanks a lot!

Comments:
Hi ! Thanks for reporting.
Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
It would be nice to know whether it comes from the dataloader or not

> Hi ! Thanks for reporting.
> Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> It would be nice to know whether it comes from the dataloader or not

I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`,  around 20000it/s.

> > Hi ! Thanks for reporting.
> > Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> > It would be nice to know whether it comes from the dataloader or not
> 
> I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`, around 20000it/s.

Hi ! I meet the same problem, how did you solve it?
Issue: imdb dataset cannot be loaded 

Hi
I am trying to load the imdb train dataset

`dataset = datasets.load_dataset("imdb", split="train")`

getting following errors, thanks for your help 
```
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=32660064, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='test', num_bytes=26476338, num_examples=20316, dataset_name='imdb')}, {'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
>>> dataset = datasets.load_dataset("imdb", split="train")

```


Comments:
It looks like there was an issue while building the imdb dataset.
Could you provide more information about your OS and the version of python and `datasets` ?

Also could you try again with 
```python
dataset = datasets.load_dataset("imdb", split="train", download_mode="force_redownload")
```
to make sure it's not a corrupted file issue ?

I was using version 1.1.2 and this resolved with version 1.1.3, thanks. 

Hello,
I have the same pb with 1.8.0

Hi ! I just tried in 1.8.0 and it worked fine. Can you try again ? Maybe the dataset host had some issues that are fixed now

Hello,
It works fine now :) !
Thanks !

Ran into the same issue on a different dataset. I workedaround this by passing 

        verification_mode='no_checks',

to load_dataset method. Ref: https://github.com/huggingface/datasets/blob/871eabc7b23c27d677bc06ae2cc1ec3a2a04b10f/src/datasets/builder.py#L1141

Note that this is a hack before the root cause is solved.
Issue: bug in boolq dataset loading

Hi
I am trying to load boolq dataset:

```
import datasets
datasets.load_dataset("boolq")
```

I am getting the following errors, thanks for your help 

```
>>> import datasets
2020-11-22 09:16:30.070332: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-11-22 09:16:30.070389: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
>>> datasets.load_dataset("boolq")
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
Using custom data configuration default
Downloading and preparing dataset boolq/default (download: 8.36 MiB, generated: 7.47 MiB, post-processed: Unknown size, total: 15.83 MiB) to /idiap/temp/rkarimi/cache_home/datasets/boolq/default/0.1.0/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11...
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
cahce dir  /idiap/temp/rkarimi/cache_home/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)
tensorflow.python.framework.errors_impl.AlreadyExistsError: file already exists


```

Comments:
I just opened a PR to fix this.
Thanks for reporting !
Issue: trec dataset unavailable 

Hi
when I try to load the trec dataset I am getting these errors, thanks for your help

`datasets.load_dataset("trec",  split="train")
`
```
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/trec/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7/trec.py", line 140, in _split_generators
    dl_files = dl_manager.download_and_extract(_URLs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 163, in _single_map_nested
    return function(data_struct)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 477, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label
```

Comments:
This was fixed in #740 
Could you try to update `datasets` and try again ?

This has been fixed in datasets 1.1.3
Issue: terminate called after throwing an instance of 'google::protobuf::FatalException'

Hi
I am using the dataset "iwslt2017-en-nl", and after downloading it I am getting this error when trying to evaluate it on T5-base with seq2seq_trainer.py in the huggingface repo could you assist me please? thanks 


100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [02:47<00:00,  2.18s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
run_t5_base_eval.sh: line 19:  5795 Aborted 

Comments:
Loading the iwslt2017-en-nl config of iwslt2017 works fine on my side. 
Maybe you can open an issue on transformers as well ? And also add more details about your environment (OS, python version, version of transformers and datasets etc.)

closing now, figured out this is because the max length of decoder was set smaller than the input_dimensions. thanks
Issue: [Feature Request] Add optional parameter in text loading script to preserve linebreaks

I'm working on a project about rhyming verse using phonetic poetry and song lyrics, and line breaks are a vital part of the data. 

I recently switched over to use the datasets library when my various corpora grew larger than my computer's memory. And so far, it is SO great. 

But the first time I processed all of my data into a dataset, I hadn't realized the text loader script was processing the source files line-by-line and stripping off the newlines. 

Once I caught the issue, I made my own data loader by modifying one line in the default text loader (changing `batch = batch.splitlines()` to `batch = batch.splitlines(True)` inside `_generate_tables`). And so I'm all set as far as my project is concerned.

But if my use case is more general, it seems like it'd be pretty trivial to add a kwarg to the default text loader called keeplinebreaks or something, which would default to False and get passed to `splitlines()`. 

Comments:
Hi ! Thanks for your message.
Indeed it's a free feature we can add and that can be useful.
If you want to contribute, feel free to open a PR to add it to the text dataset script :)

Resolved via #1913.
Issue: OSCAR from Inria group

## Adding a Dataset
- **Name:** *OSCAR* (Open Super-large Crawled ALMAnaCH coRpus), multilingual parsing of Common Crawl (separate crawls for many different languages), [here](https://oscar-corpus.com/).
- **Description:** *OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.*
- **Paper:** *[here](https://hal.inria.fr/hal-02148693)*
- **Data:** *[here](https://oscar-corpus.com/)*
- **Motivation:** *useful for unsupervised tasks in separate languages. In an ideal world, your team would be able to obtain the unshuffled version, that could be used to train GPT-2-like models (the shuffled version, I suppose, could be used for translation).*

I am aware that you do offer the "colossal" Common Crawl dataset already, but this has the advantage to be available in many subcorpora for different languages.


Comments:
PR is already open here : #348 
The only thing remaining is to compute the metadata of each subdataset (one per language + shuffled/unshuffled).
As soon as #863 is merged we can start computing them. This will take a bit of time though

Grand, thanks for this!
Issue: Have Trouble importing `datasets`

I'm failing to import transformers (v4.0.0-dev), and tracing the cause seems to be failing to import datasets.

I cloned the newest version of datasets (master branch), and do `pip install -e .`.

Then, `import datasets` causes the error below.

```
~/workspace/Clone/datasets/src/datasets/utils/file_utils.py in <module>
    116 sys.path.append(str(HF_MODULES_CACHE))
    117 
--> 118 os.makedirs(HF_MODULES_CACHE, exist_ok=True)
    119 if not os.path.exists(os.path.join(HF_MODULES_CACHE, "__init__.py")):
    120     with open(os.path.join(HF_MODULES_CACHE, "__init__.py"), "w"):

~/.pyenv/versions/anaconda3-2020.07/lib/python3.8/os.py in makedirs(name, mode, exist_ok)
    221             return
    222     try:
--> 223         mkdir(name, mode)
    224     except OSError:
    225         # Cannot rely on checking for EEXIST, since the operating system 

FileNotFoundError: [Errno 2] No such file or directory: '<MY_HOME_DIRECTORY>/.cache/huggingface/modules'
```

The error occurs in `os.makedirs` in `file_utils.py`, even though `exist_ok = True` option is set.
(I use Python 3.8, so `exist_ok` is expected to work.)

I've checked some environment variables, and they are set as below.

```
*** NameError: name 'HF_MODULES_CACHE' is not defined
*** NameError: name 'hf_cache_home' is not defined
*** NameError: name 'XDG_CACHE_HOME' is not defined
```

Should I set some environment variables before using this library?
And, do you have any idea why "No such file or directory" occurs even though the `exist_ok = True` option is set?

Thank you in advance.

Comments:
I'm sorry, this was a problem with my environment.
Now that I have identified the cause of environmental dependency, I would like to fix it and try it.
Excuse me for making a noise.
Issue: Load amazon dataset

Hi,
I was going through amazon_us_reviews dataset and found that example API usage given on website is different from the API usage while loading dataset. 

Eg. what API usage is on the [website](https://huggingface.co/datasets/amazon_us_reviews) 
```
from datasets import load_dataset
dataset = load_dataset("amazon_us_reviews")
```
How it is when I tried (the error generated does point me to the right direction though)
```
from datasets import load_dataset
dataset = load_dataset("amazon_us_reviews", 'Books_v1_00')
``` 
Also, there is some issue with formatting as it's not showing bullet list in description with new line. Can I work on it?

Comments:
Thanks for reporting !
We plan to show information about the different configs of the datasets on the website, with the corresponding `load_dataset` calls.

Also I think the bullet points formatting has been fixed
Issue: Add HoVer multi-hop fact verification dataset

## Adding a Dataset
- **Name:** HoVer
- **Description:** https://twitter.com/YichenJiang9/status/1326954363806429186 contains 20K claim verification examples
- **Paper:** https://arxiv.org/abs/2011.03088
- **Data:** https://hover-nlp.github.io/
- **Motivation:** There are still few multi-hop information extraction benchmarks (HotpotQA, which dataset wase based off, notwithstanding)

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hi @yjernite  I'm new but wanted to contribute. Has anyone already taken this problem and do you think it is suitable for newbies?

Hi @tenjjin! This dataset is still up for grabs! Here's the link with the guide to add it. You should play around with the library first (download and look at a few datasets), then follow the steps here:

https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md

Closed by #1399
Issue: use_custom_baseline still produces errors for bertscore

`metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2

Comments:
Thanks for reporting ! That's a bug indeed
If you want to contribute, feel free to fix this issue and open a PR :)

This error is because of a mismatch between `datasets` and `bert_score`. With `datasets=1.1.2` and `bert_score>=0.3.6` it works ok. So `pip install -U bert_score` should fix the problem. 

Thanks for the heads up @pvl and for the PR as well :)

Hello everyone,

I think the  problem is not solved: 

```
from datasets import load_metric
metric=load_metric('bertscore')
metric.compute(
    predictions=predictions,
    references=references,
    lang='fr',
    rescale_with_baseline=True
)
TypeError: get_hash() missing 2 required positional arguments: 'use_custom_baseline' and 'use_fast_tokenizer'
```
This code is produced using `Python 3.6.9 datasets==1.1.2 and bert_score==0.3.10`

Hi ! This has been fixed by https://github.com/huggingface/datasets/pull/2770, we'll do a new release soon to make the fix available :)

In the meantime please use an older version of `bert_score`
Issue: How to enable `.map()` pre-processing pipelines to support multi-node parallelism?

Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks!

Comments:
Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :)

Curious to hear if anything on that side changed or if you suggestions to do it changed @lhoestq :)

For our use-case, we are entering the regime where trading a few more instances to save a few days would be nice :)

Currently for multi-node setups we're mostly going towards a nice integration with Dask. But I wouldn't exclude exploring `pathos` more at one point

What is the current status of this feature? Is multi-node parallelism supported for `.map` function?

Hi ! `datasets` main focus in single node data processing, but you can manually `.shard()` a dataset to run `map()` on each dataset shard in a distributed setup !

Otherwise you can also explore `datatrove` which supports SLURM for distributed data processing, or tools like spark / dask
Issue: Wikipedia postprocessing

Hi, thanks for this library!

Running this code:

```py
import datasets
wikipedia = datasets.load_dataset("wikipedia", "20200501.de")
print(wikipedia['train']['text'][0])
```

I get:

```
mini|Ricardo Flores Magón
mini|Mexikanische Revolutionäre, Magón in der Mitte anführend, gegen die Diktatur von Porfirio Diaz, Ausschnitt des Gemälde „Tierra y Libertad“ von Idelfonso Carrara (?) von 1930.

Ricardo Flores Magón (* 16. September 1874 in San Antonio Eloxochitlán im mexikanischen Bundesstaat Oaxaca; † 22. November 1922 im Bundesgefängnis Leavenworth im US-amerikanischen Bundesstaat Kansas) war als Journalist, Gewerkschafter und Literat ein führender anarchistischer Theoretiker und Aktivist, der die revolutionäre mexikanische Bewegung radikal beeinflusste. Magón war Gründer der Partido Liberal Mexicano und Mitglied der Industrial Workers of the World.

Politische Biografie 
Journalistisch und politisch kämpfte er und sein Bruder sehr kompromisslos gegen die Diktatur Porfirio Diaz. Philosophisch und politisch orientiert an radikal anarchistischen Idealen und den Erfahrungen seiner indigenen Vorfahren bei der gemeinschaftlichen Bewirtschaftung des Gemeindelandes, machte er die Forderung „Land und Freiheit“ (Tierra y Libertad) populär. Besonders Francisco Villa und Emiliano Zapata griffen die Forderung Land und Freiheit auf. Seine Philosophie hatte großen Einfluss auf die Landarbeiter. 1904 floh er in die USA und gründete 1906 die Partido Liberal Mexicano. Im Exil lernte er u. a. Emma Goldman kennen. Er verbrachte die meiste Zeit seines Lebens in Gefängnissen und im Exil und wurde 1918 in den USA wegen „Behinderung der Kriegsanstrengungen“ zu zwanzig Jahren Gefängnis verurteilt. Zu seinem Tod gibt es drei verschiedene Theorien. Offiziell starb er an Herzversagen. Librado Rivera, der die Leiche mit eigenen Augen gesehen hat, geht davon aus, dass Magón von einem Mitgefangenen erdrosselt wurde. Die staatstreue Gewerkschaftszeitung CROM veröffentlichte 1923 einen Beitrag, nachdem Magón von einem Gefängniswärter erschlagen wurde.
mini|Die Brüder Ricardo (links) und Enrique Flores Magón (rechts) vor dem Los Angeles County Jail, 1917

[...]
```

so some Markup like `mini|` is still left. Should I run another parser on this text before feeding it to an ML model or is this a known imperfection of parsing Wiki markup?

Apologies if this has been asked before.

Comments:
Hi @bminixhofer ! Parsing WikiMedia is notoriously difficult: this processing used [mwparserfromhell](https://github.com/earwig/mwparserfromhell) which is pretty good but not perfect.

As an alternative, you can also use the Wiki40b dataset which was pre-processed using an un-released Google internal tool

Ok, thanks! I'll try the Wiki40b dataset.

If anyone else is concerned about this, `wiki40b` does indeed seem very well cleaned.
Issue: [GEM] add WikiLingua cross-lingual abstractive summarization dataset

## Adding a Dataset
- **Name:** WikiLingua
- **Description:** The dataset includes ~770k article and summary pairs in 18 languages from WikiHow. The gold-standard article-summary alignments across languages were extracted by aligning the images that are used to describe each how-to step in an article.
- **Paper:** https://arxiv.org/pdf/2010.03093.pdf
- **Data:** https://github.com/esdurmus/Wikilingua
- **Motivation:** Included in the GEM shared task. Multilingual.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hey @yjernite. This is a very interesting dataset. Would love to work on adding it but I see that the link to the data is to a gdrive folder. Can I just confirm wether dlmanager can handle gdrive urls or would this have to be a manual dl?

Hi @KMFODA ! A version of WikiLingua is actually already accessible in the [GEM dataset](https://huggingface.co/datasets/gem)

You can use it for example to load the French to English translation with:
```python
from datasets import load_dataset
wikilingua = load_dataset("gem", "wiki_lingua_french_fr")
```

Closed by https://github.com/huggingface/datasets/pull/1807
Issue: [GEM] add ToTTo Table-to-text dataset

## Adding a Dataset
- **Name:** ToTTo
- **Description:** ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description.
- **Paper:** https://arxiv.org/abs/2004.14373
- **Data:** https://github.com/google-research-datasets/totto
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
closed via #1098
Issue: [GEM] MultiWOZ dialogue dataset

## Adding a Dataset
- **Name:** MultiWOZ (Multi-Domain Wizard-of-Oz)
- **Description:** 10k annotated human-human dialogues. Each dialogue consists of a goal, multiple user and system utterances as well as a belief state. Only system utterances are annotated with dialogue acts – there are no annotations from the user side.
- **Paper:** https://arxiv.org/pdf/2007.12720.pdf
- **Data:** https://github.com/budzianowski/multiwoz
- **Motivation:** Will likely be part of the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hi @yjernite can I help in adding this dataset? 

I am excited about this because this will be my first contribution to the datasets library as well as to hugginface.

Resolved via https://github.com/huggingface/datasets/pull/979
Issue: datasets freezes 

Hi, I want to load these two datasets and convert them to Dataset format in torch and the code freezes for me, could you have a look please? thanks 

dataset1 = load_dataset("squad", split="train[:10]")
dataset1 = dataset1.set_format(type='torch', columns=['context', 'answers', 'question'])

dataset2 = load_dataset("imdb", split="train[:10]")
dataset2 = dataset2.set_format(type="torch", columns=["text", "label"])
print(len(dataset1))


Comments:
Pytorch is unable to convert strings to tensors unfortunately.
You can use `set_format(type="torch")` on columns that can be converted to tensors, such as token ids.

This makes me think that we should probably raise an error or at least a warning when one tries to create pytorch tensors out of text columns

Ultimately, we decided to return a list instead of an error when formatting a string column with the format type `"torch"`.

If you think an error would be more appropriate, please open a new issue.
Issue: Add MRQA dataset

## Adding a Dataset
- **Name:** MRQA
- **Description:** Collection of different (subsets of) QA datasets all converted to the same format to evaluate out-of-domain generalization (the datasets come from different domains, distributions, etc.). Some datasets are used for training and others are used for evaluation. This dataset was collected as part of MRQA 2019's shared task 
- **Paper:** https://arxiv.org/abs/1910.09753
- **Data:** https://github.com/mrqa/MRQA-Shared-Task-2019
- **Motivation:** Out-of-domain generalization is becoming (has become) a de-factor evaluation for NLU systems

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
Done! cf #1117 and #1022
Issue: [Caching] Dill globalvars() output order is not deterministic and can cause cache issues.

Dill uses `dill.detect.globalvars` to get the globals used by a function in a recursive dump. `globalvars` returns a dictionary of all the globals that a dumped function needs. However the order of the keys in this dict is not deterministic and can cause caching issues.

To fix that one could register an implementation of dill's `save_function` in the `datasets` pickler that sorts the globals keys before dumping a function.

Comments:
To show the issue:
```
python -c "from datasets.fingerprint import Hasher; a=[]; func = lambda : len(a); print(Hasher.hash(func))"
```
doesn't always return the same ouput since `globs` is a dictionary with "a" and "len" as keys but sometimes not in the same order
Issue: How to implement DistributedSampler with datasets 

Hi,
I am using your datasets to define my dataloaders, and I am training finetune_trainer.py in huggingface repo on them.
I need a distributedSampler to be able to train the models on TPUs being able to distribute the load across the TPU cores. Could you tell me how I can implement the distribued sampler when using datasets in which datasets are iterative? To give you more context, I have multiple of datasets and I need to write sampler for this case. thanks. 

Comments:
Hi Apparently I need to shard the data and give one host a chunk, could you provide me please with examples on how to do it? I want to use it jointly with finetune_trainer.py in huggingface repo seq2seq examples. thanks. 

Hey @rabeehkarimimahabadi I'm actually looking for the same feature. Did you manage to get somewhere?

@rabeehkarimimahabadi need the same feature

Hi! I think you can use the `accelerate` library for that, which implements distributed sampling.
Issue: nlp viewer error

Hello, 
when I select amazon_us_reviews in nlp viewer, it shows error.
https://huggingface.co/nlp/viewer/?dataset=amazon_us_reviews
![image](https://user-images.githubusercontent.com/30210529/98447334-4aa81200-2124-11eb-9dca-82c3ab34ccc2.png)


Comments:
and also for 'blog_authorship_corpus'
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
![image](https://user-images.githubusercontent.com/30210529/98557329-5c182800-22a4-11eb-9b01-5b910fb8fcd4.png)


Is this the problem of my local computer or ??

Related to:
- #673
Issue: Add Google Taskmaster dataset

## Adding a Dataset
- **Name:** Taskmaster
- **Description:** A large dataset of task-oriented dialogue with annotated goals (55K dialogues covering entertainment and travel reservations)
- **Paper:** https://arxiv.org/abs/1909.05358
- **Data:** https://github.com/google-research-datasets/Taskmaster
- **Motivation:** One of few annotated datasets of this size for goal-oriented dialogue

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hey @yjernite. Was going to start working on this but found taskmaster 1,2 & 3 in the datasets library already so think this can be closed now?

You are absolutely right :) 

Closed by https://github.com/huggingface/datasets/pull/1193 https://github.com/huggingface/datasets/pull/1197 https://github.com/huggingface/datasets/pull/1213
Issue: Quail dataset urls are out of date

<h3>Code</h3>

```
from datasets import load_dataset
quail = load_dataset('quail')
```

<h3>Error</h3>

```
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/text-machine-lab/quail/master/quail_v1.2/xml/ordered/quail_1.2_train.xml
```


As per [quail v1.3 commit](https://github.com/text-machine-lab/quail/commit/506501cfa34d9ec6c042d31026ba6fea6bcec8ff) it looks like the location and suggested ordering has changed. In [https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58](https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58) the quail v1.2 datasets are being pointed to, which don't exist anymore.

Comments:
Hi ! Thanks for reporting.
We should fix the urls and use quail 1.3.
If you want to contribute feel free to fix the urls and open a PR :) 

Done! PR [https://github.com/huggingface/datasets/pull/820](https://github.com/huggingface/datasets/pull/820)

Updated links and also regenerated the metadata and dummy data for v1.3 in order to pass verifications as described here: [https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset](https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset). 

Closing since #820 is merged.
Thanks again for fixing the urls :)
Issue: On loading a metric from datasets, I get the following error

`from datasets import load_metric`

`metric = load_metric('bleurt')`

Traceback:
210 class _ArrayXDExtensionType(pa.PyExtensionType):
    211 
    212     ndims: int = None

AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'

Any help will be appreciated. Thank you. 

Comments:
Hi ! We support only pyarrow > 0.17.1 so that we have access to the `PyExtensionType` object.
Could you update pyarrow and try again ?
```
pip install --upgrade pyarrow
```
Issue: Empty output/answer in TriviaQA test set (both in 'kilt_tasks' and 'trivia_qa')

# The issue

It's all in the title, it appears to be fine on the train and validation sets.

Is there some kind of mapping to do like for the questions (see https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md) ? 

# How to reproduce
```py
from datasets import load_dataset
kilt_tasks = load_dataset("kilt_tasks")
trivia_qa = load_dataset('trivia_qa', 'unfiltered.nocontext')
# both in "kilt_tasks"
In [18]: any([output['answer'] for output in kilt_tasks['test_triviaqa']['output']])                                                                                                                        
Out[18]: False
# and "trivia_qa"
In [13]: all([answer['value'] == '<unk>' for answer in trivia_qa['test']['answer']])                                                                                                                        
Out[13]: True
# appears to be fine on the train and validation sets.
In [14]: all([answer['value'] == '<unk>' for answer in trivia_qa['train']['answer']])                                                                                                                       
Out[14]: False

In [15]: all([answer['value'] == '<unk>' for answer in trivia_qa['validation']['answer']])                                                                                                                  
Out[15]: False

In [16]: any([output['answer'] for output in kilt_tasks['train_triviaqa']['output']])                                                                                                                       
Out[16]: True

In [17]: any([output['answer'] for output in kilt_tasks['validation_triviaqa']['output']])                                                                                                                  
Out[17]: True

```

Comments:
cc @yjernite is this expected ?

Yes: TriviaQA has a private test set for the leaderboard [here](https://competitions.codalab.org/competitions/17208)

For the KILT training and validation portions, you need to link the examples from the TriviaQA dataset as detailed here:
https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md

Oh ok, I guess I read the paper too fast 😅, thank you for your answer!
Issue: How to join two datasets?

Hi,

I'm wondering if it's possible to join two (preprocessed) datasets with the same number of rows but different labels? 

I'm currently trying to create paired sentences for BERT from `wikipedia/'20200501.en`, and I couldn't figure out a way to create a paired sentence using `.map()` where the second sentence is **not** the next sentence (i.e., from a different article) of the first sentence.

Thanks!

Comments:
Hi this is also my question. thanks 

Hi ! Currently the only way to add new fields to a dataset is by using `.map` and picking items from the other dataset


Closing this one. Feel free to re-open if you have other questions about this issue.

Also linking another discussion about joining datasets: #853
Issue: Token classification labels are strings and we don't have the list of labels

Not sure if this is an issue we want to fix or not, putting it here so it's not forgotten. Right now, in token classification datasets, the labels for NER, POS and the likes are typed as `Sequence` of `strings`, which is wrong in my opinion. These should be `Sequence` of `ClassLabel` or some types that gives easy access to the underlying labels.

The main problem for preprocessing those datasets is that the list of possible labels is not stored inside the `Dataset` object which makes converting the labels to IDs quite difficult (you either have to know the list of labels in advance or run a full pass through the dataset to get the list of labels, the `unique` method being useless with the type `Sequence[str]`).

Comments:
Indeed. Pinging @stefan-it here if he want to give an expert opinion :)

Related is https://github.com/huggingface/datasets/pull/636

Should definitely be a ClassLabel 👍 

Already done.
Issue: Descriptions of raw and processed versions of wikitext are inverted

Nothing of importance, but it looks like the descriptions of wikitext-n-v1 and wikitext-n-raw-v1 are inverted for both n=2 and n=103. I just verified by loading them and the `<unk>` tokens are present in the non-raw versions, which confirms that it's a mere inversion of the descriptions and not of the datasets themselves.

Also it would be nice if those descriptions appeared in the dataset explorer.

https://github.com/huggingface/datasets/blob/87bd0864845ea0a1dd7167918dc5f341bf807bd3/datasets/wikitext/wikitext.py#L52

Comments:
Yes indeed ! Thanks for reporting

Fixed by:
- #3241
Issue: Error running pip install -e ".[dev]" on MacOS 10.13.6: faiss/python does not exist

I was following along with https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset when I ran into this error.

```sh
git clone https://github.com/huggingface/datasets
cd datasets
virtualenv venv -p python3 --system-site-packages
source venv/bin/activate
pip install -e ".[dev]"
```


![image](https://user-images.githubusercontent.com/59632/97868518-72871800-1cd5-11eb-9cd2-37d4e9d20b39.png)

![image](https://user-images.githubusercontent.com/59632/97868592-977b8b00-1cd5-11eb-8f3c-0c409616149c.png)

Python 3.7.7


Comments:
I saw that `faiss-cpu` 1.6.4.post2 was released recently to fix the installation on macos. It should work now

Closing this one.
Feel free to re-open if you still have issues
Issue: feat(dataset): multiprocessing _generate_examples

forking this out of #741, this issue is only regarding multiprocessing

I'd love if there was a dataset configuration parameter `workers`, where when it is `1` it behaves as it does right now, and when its `>1` maybe `_generate_examples` can also get the `pool` and return an iterable using the pool.

In my use case, I would instead of:
```python
for datum in data:
     yield self.load_datum(datum)
```
do:
```python
return pool.map(self.load_datum, data)
```

As the dataset in question, as an example, has **only** 7000 rows, and takes 10 seconds to load each row on average, it takes almost 20 hours to load the entire dataset.
If this was a larger dataset (and many such datasets exist), it would take multiple days to complete.

Using multiprocessing, for example, 40 cores, could speed it up dramatically. For this dataset, hopefully to fully load in under an hour.

Comments:
I agree that would be cool :)
Right now the only distributed dataset builder is based on Apache Beam so you can use distributed processing frameworks like Dataflow, Spark, Flink etc. to build your dataset but it's not really well suited for single-worker parallel processing afaik

`_generate_examples` can now be run in parallel thanks to https://github.com/huggingface/datasets/pull/5107. You can find more info [here](https://huggingface.co/docs/datasets/dataset_script#sharding).