helmo/github-issues huggingface dataset:
Can't import cc100 dataset
State: closed | Author: alighofrani95 (NONE)

There is some issue to import cc100 dataset.

```
from datasets import load_dataset
dataset = load_dataset("cc100")
```

FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/cc100/cc100.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/cc100/cc100.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    280                 raise FileNotFoundError(
    281                     "Couldn't find file locally at {}, or remotely at {} or {}".format(
--> 282                         combined_path, github_file_path, file_path
    283                     )
    284                 )

FileNotFoundError: Couldn't find file locally at cc100/cc100.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/cc100/cc100.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/cc100/cc100.py

Comments: cc100 was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `cc100` with

```python
from datasets import load_dataset

lang = "en"
dataset = load_dataset("cc100", lang=lang, split="train")
```
Switchboard Dialog Act Corpus added under `datasets/swda`
State: closed | Author: gmihaila (CONTRIBUTOR)

Switchboard Dialog Act Corpus

Intro:
The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2,
with turn/utterance-level dialog-act tags. The tags summarize syntactic, semantic, and pragmatic information
about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.

Details:
[homepage](http://compprag.christopherpotts.net/swda.html)
[repo](https://github.com/NathanDuran/Switchboard-Corpus/raw/master/swda_data/)

I believe this is an important dataset to have since there is no dataset related to dialogue act added.

I didn't find any formatting for pull request. I hope all this information is enough.

For any support please contact me. 

Comments: @lhoestq Thank you for your detailed comments! I fixed everything you suggested.

Please let me know if I'm missing anything else.

It looks like the Transcript and Utterance objects are missing, maybe we can mention it in the README ? Or just add them ? @gmihaila @bhavitvyamalik 

Hi @lhoestq,
I'm working on this to add the full dataset

> It looks like the Transcript and Utterance objects are missing, maybe we can mention it in the README ? Or just add them ? @gmihaila @bhavitvyamalik

@lhoestq  Any info on how to add them?

@gmihaila, instead of using the current repo you should look into [this](https://github.com/cgpotts/swda). You can use the `csv` files uploaded in this repo (`swda.zip`) to access other fields and include them in this dataset. It has one dependency too, `swda.py`, you can download that separately and include it in your dataset's folder to be imported while reading the `csv` files.

Almost all the attributes of `Transcript` and `Utterance` objects are of the type str, int, or list. As far as `trees` attribute is concerned in utterance object you can simply parse it as string and user can maybe later convert it to nltk.tree object

@bhavitvyamalik Thank you for the clarification! 

I didn't use [that](https://github.com/cgpotts/swda) because it doesn't have the splits. I think in combination with [what I used](https://github.com/NathanDuran/Switchboard-Corpus) would help.

Let me know if I can help! I can make those changes if you don't have the time.

I'm a bit busy for the next 2 weeks. I'll be able to complete it by end of January only. Maybe you can start with it and I'll help you?
Also, I looked into the official train/val/test splits and not all the files are there in the repo I used so I think either we'll have to skip them or put all of that into just train

Yes, I can start working on it and ask you to do a code review.

Yes, not all files are there. I'll try to find papers that have the correct and full splits, if not, I'll do like you suggested.

Thank you again for your help @bhavitvyamalik !
Switchboard Dialog Act Corpus added under `datasets/swda`
State: closed | Author: gmihaila (CONTRIBUTOR)

Pleased to announced that I added my first dataset **Switchboard Dialog Act Corpus**.


I think this is an important datasets to be added since it is the only one related to dialogue act classification. 

Hope the pull request is ok. Wasn't able to see any special formatting for the pull request form.


The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2,
with turn/utterance-level dialog-act tags. The tags summarize syntactic, semantic, and pragmatic information
about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.


[webpage](http://compprag.christopherpotts.net/swda.html)

[repo](https://github.com/NathanDuran/Switchboard-Corpus/raw/master/swda_data/)

Please contact me for any support!

All tests passed and followed all steps in the contribution guide!



Comments: Need to fix code formatting.
new version of Ted Talks IWSLT (WIT3)
State: closed | Author: skyprince999 (CONTRIBUTOR)

In the previous iteration #1608 I had used language pairs. Which created 21,582 configs (109*108) !!! 

Now, TED talks in _each language_ is a separate config. So it's more cleaner with _just 109 configs_  (one for each language). Dummy files were created manually. 

Locally  I was able to clear the `python datasets-cli test datasets/......` . Which created the `dataset_info.json`  file . The test for the dummy files was also cleared. However couldn't figure out how to specify the local data folder for the real dataset


**Note: that this requires manual download of the dataset.** 
**Note2: The high number of _Files changed (112)_ is because of the large number of dummy files/configs!**

Comments: > Nice thank you ! Actually as it is a translation dataset we should probably have one configuration = one language pair no ?
> 
> Could you use the same trick for this dataset ?

I was looking for this input, infact I had written a long post on the Slack channel,...(_but unfortunately due to the holidays didn;t get a respones_). Initially I had tried with language pairs and then with specific language configs. 

I'll have a look at the `opus-gnomes` dataset


Oh sorry I must have missed your message then :/
I was off a few days during the holidays

Hopefully this trick can enable the use of any language pair (+ year ?) combination and also simplify a lot the dummy data creation since it will only require a few configs.

Updated it as per the comments. But couldn't figure out why the dummy tests are failing >> 
```
$RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_ted_talks_iwslt
.....
....
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```
Unable to Download Hindi Wikipedia Dataset
State: closed | Author: aditya3498 (NONE)

I used the Dataset Library in Python to load the wikipedia dataset with the Hindi Config 20200501.hi along with something called beam_runner='DirectRunner' and it keeps giving me the error that the file is not found. I have attached the screenshot of the error and the code both. Please help me to understand how to resolve this issue.

![Code](https://user-images.githubusercontent.com/30871963/103437466-1f3a3300-4c4e-11eb-9d54-fc9601abfeec.png)

![Error](https://user-images.githubusercontent.com/30871963/103437407-7ee40e80-4c4d-11eb-8151-a86eb664e6be.png)


Comments: Currently this dataset is only available when the library is installed from source since it was added after the last release.

We pin the dataset version with the library version so that people can have a reproducible dataset and processing when pinning the library.

We'll see if we can provide access to newer datasets with a warning that they are newer than your library version, that would help in cases like yours.

So for now, should i try and install the library from source and then try out the same piece of code? Will it work then, considering both the versions will match then?

Yes

Hey, so i tried installing the library from source using the commands : **git clone https://github.com/huggingface/datasets**,  **cd datasets** and then **pip3 install -e .**. But i still am facing the same error that file is not found. Please advise.

The Datasets library version now is 1.1.3 by installing from source as compared to the earlier 1.0.3 that i had loaded using pip command but I am still getting same error

![Error](https://user-images.githubusercontent.com/30871963/103479005-69f3b080-4df0-11eb-83ae-58d7bb56a90e.png)


Looks like the wikipedia dump for hindi at the date of 05/05/2020 is not available anymore.
You can try to load a more recent version of wikipedia
```python
from datasets import load_dataset

d = load_dataset("wikipedia", language="hi", date="20210101", split="train", beam_runner="DirectRunner")
```

Okay, thank you so much
wiki_dpr pre-processing performance
State: open | Author: dbarnhart (NONE)
Labels: name: enhancement

name: Dataset discussion

I've been working with wiki_dpr and noticed that the dataset processing is seriously impaired in performance [1]. It takes about 12h to process the entire dataset. Most of this time is simply loading and processing the data, but the actual indexing is also quite slow (3h).

I won't repeat the concerns around multiprocessing as they are addressed in other issues (#786), but this is the first obvious thing to do. Using cython to speed up the text manipulation may be also help. Loading and processing a dataset of this size in under 15 minutes does not seem unreasonable on a modern multi-core machine. I have hit such targets myself on similar tasks. Would love to see this improve.

The other issue is that it takes 3h to construct the FAISS index. If only we could use GPUs with HNSW, but we can't. My sharded GPU indexing code can build an IVF + PQ index in 10 minutes on 20 million vectors. Still, 3h seems slow even for the CPU.

It looks like HF is adding only 1000 vectors at a time by default [2], whereas the faiss benchmarks adds 1 million vectors at a time (effectively) [3]. It's possible the runtime could be reduced with a larger batch. Also, it looks like project dependencies ultimately use OpenBLAS, but this is known to have issues when combined with OpenMP, which HNSW does [3]. A workaround is to set the environment variable `OMP_WAIT_POLICY=PASSIVE` via `os.environ` or similar.

References:
[1] https://github.com/huggingface/datasets/blob/master/datasets/wiki_dpr/wiki_dpr.py
[2] https://github.com/huggingface/datasets/blob/master/src/datasets/search.py
[3] https://github.com/facebookresearch/faiss/blob/master/benchs/bench_hnsw.py
[4] https://github.com/facebookresearch/faiss/issues/422

Comments: Hi ! And thanks for the tips :) 

Indeed currently `wiki_dpr` takes some time to be processed.
Multiprocessing for dataset generation is definitely going to speed up things.

Regarding the index note that for the default configurations, the index is downloaded instead of being built, which avoid spending time on constructing the index. However in other cases it would be awesome to make the construction faster.

Any contribution that can help things faster are welcome. In particular in you have some code that can build a wiki_dpr IVF PQ index in a sharded GPU setup and would like to share it, we can add it to an `examples` folder. In particular since faiss is becoming the library of reference for dataset indexing for tasks like Open Domain Question Answering.



I'd be happy to contribute something when I get the time, probably adding multiprocessing and / or cython support to wiki_dpr. I've written cythonized apache beam code before as well.

For sharded index building, I used the FAISS example code for indexing 1 billion vectors as a start. I'm sure you're aware that the documentation isn't great, but the source code is fairly easy to follow.

Nice thanks ! That would be awesome to make its construction faster :)
wiki_dpr dataset pre-processesing performance
State: closed | Author: dbarnhart (NONE)

I've been working with wiki_dpr and noticed that the dataset processing is seriously impaired in performance [1]. It takes about 12h to process the entire dataset. Most of this time is simply loading and processing the data, but the actual indexing is also quite slow (3h).

I won't repeat the concerns around multiprocessing as they are addressed in other issues (#786), but this is the first obvious thing to do. Using cython to speed up the text manipulation may be also help. Loading and processing a dataset of this size in under 15 minutes does not seem unreasonable on a modern multi-core machine. I have hit such targets myself on similar tasks. Would love to see this improve.

The other issue is that it takes 3h to construct the FAISS index. If only we could use GPUs with HNSW, but we can't. My sharded GPU indexing code can build an IVF + PQ index in 10 minutes on 20 million vectors. Still, 3h seems slow even for the CPU.

It looks like HF is adding only 1000 vectors at a time by default [2], whereas the faiss benchmarks adds 1 million vectors at a time (effectively) [3]. It's possible the runtime could be reduced with a larger batch. Also, it looks like project dependencies ultimately use OpenBLAS, but this is known to have issues when combined with OpenMP, which HNSW does [3]. A workaround is to set the environment variable `OMP_WAIT_POLICY=PASSIVE` via `os.environ` or similar.

References:
[1] https://github.com/huggingface/datasets/blob/master/datasets/wiki_dpr/wiki_dpr.py
[2] https://github.com/huggingface/datasets/blob/master/src/datasets/search.py
[3] https://github.com/facebookresearch/faiss/blob/master/benchs/bench_hnsw.py
[4] https://github.com/facebookresearch/faiss/issues/422

Comments: Sorry, double posted.
Fix NER metric example in Overview notebook
State: closed | Author: jungwhank (CONTRIBUTOR)

Fix errors in `NER metric example` section in `Overview.ipynb`.

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-37-ee559b166e25> in <module>()
----> 1 ner_metric = load_metric('seqeval')
      2 references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      3 predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      4 ner_metric.compute(predictions, references)


/usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    340     if needs_to_be_installed:
    341         raise ImportError(
--> 342             f"To be able to use this {module_type}, you need to install the following dependencies"
    343             f"{[lib_name for lib_name, lib_path in needs_to_be_installed]} using 'pip install "
    344             f"{' '.join([lib_path for lib_name, lib_path in needs_to_be_installed])}' for instance'"

ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'
```

```
ValueError                                Traceback (most recent call last)
<ipython-input-39-ee559b166e25> in <module>()
      2 references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      3 predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
----> 4 ner_metric.compute(predictions, references)

/usr/local/lib/python3.6/dist-packages/datasets/metric.py in compute(self, *args, **kwargs)
    378         """
    379         if args:
--> 380             raise ValueError("Please call `compute` using keyword arguments.")
    381 
    382         predictions = kwargs.pop("predictions", None)

ValueError: Please call `compute` using keyword arguments.
```
update saving and loading methods for faiss index so to accept path l…
State: closed | Author: tslott (CONTRIBUTOR)

- Update saving and loading methods for faiss index so to accept path like objects from pathlib

The current code only supports using a string type to save and load a faiss index. This change makes it possible to use a string type OR a Path from [pathlib](https://docs.python.org/3/library/pathlib.html). The codes becomes a more intuitive this way I think.

Comments: Seems ok for me, what do you think @lhoestq ?
Arrow file is too large when saving vector data
State: closed | Author: weiwangorg (NONE)

I computed the sentence embedding of each sentence of bookcorpus data using bert base and saved them to disk. I used 20M sentences and the obtained arrow file is about 59GB while the original text file is only about 1.3GB. Are there any ways to reduce the size of the arrow file?

Comments: Hi !
The arrow file size is due to the embeddings. Indeed if they're stored as float32 then the total size of the embeddings is

20 000 000 vectors * 768 dimensions * 4 bytes per dimension ~= 60GB

If you want to reduce the size you can consider using quantization for example, or maybe using dimension reduction techniques.


Thanks for your reply @lhoestq.
I want to save original embedding for these sentences for subsequent calculations. So does arrow have a way to save in a compressed format to reduce the size of the file?

Arrow doesn't have compression since it is designed to have no serialization overhead

I see. Thank you.
NarrativeQA fails to load with `load_dataset`
State: closed | Author: eric-mitchell (NONE)

When loading the NarrativeQA dataset with `load_dataset('narrativeqa')` as given in the documentation [here](https://huggingface.co/datasets/narrativeqa), I receive a cascade of exceptions, ending with

    FileNotFoundError: Couldn't find file locally at narrativeqa/narrativeqa.py, or remotely at 
        https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/narrativeqa/narrativeqa.py or 
        https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/narrativeqa/narrativeqa.py

Workaround: manually copy the `narrativeqa.py` builder into my local directory with 

    curl https://raw.githubusercontent.com/huggingface/datasets/master/datasets/narrativeqa/narrativeqa.py -o narrativeqa.py

and load the dataset as `load_dataset('narrativeqa.py')` everything works fine. I'm on datasets v1.1.3 using Python 3.6.10.

Comments: Hi @eric-mitchell,
I think the issue might be that this dataset was added during the community sprint and has not been released yet. It will be available with the v2 of `datasets`.
For now, you should be able to load the datasets after installing the latest (master) version of `datasets` using pip:
`pip install git+https://github.com/huggingface/datasets.git@master`

@bhavitvyamalik Great, thanks for this! Confirmed that the problem is resolved on master at [cbbda53](https://github.com/huggingface/datasets/commit/cbbda53ac1520b01f0f67ed6017003936c41ec59).

Update: HuggingFace did an intermediate release yesterday just before the v2.0.

To load it you can just update `datasets`

`pip install --upgrade datasets`
HoVeR dataset fails to load
State: closed | Author: urikz (NONE)

Hi! I'm getting an error when trying to load **HoVeR** dataset. Another one (**SQuAD**) does work for me. I'm using the latest (1.1.3) version of the library.

Steps to reproduce the error:

```python
>>> from datasets import load_dataset
>>> dataset = load_dataset("hover")
Traceback (most recent call last):
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 486, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/hover/hover.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/load.py", line 278, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 486, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/hover/hover.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/Users/urikz/anaconda/envs/mentionmemory/lib/python3.7/site-packages/datasets/load.py", line 282, in prepare_module
    combined_path, github_file_path, file_path
FileNotFoundError: Couldn't find file locally at hover/hover.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/hover/hover.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/hover/hover.py
```

Comments: Hover was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `hover` with

```python
from datasets import load_dataset

dataset = load_dataset("hover")
```
Dataset social_bias_frames 404
State: closed | Author: atemate (NONE)

```
>>> from datasets import load_dataset
>>> dataset = load_dataset("social_bias_frames")
...
Downloading and preparing dataset social_bias_frames/default
...
~/.pyenv/versions/3.7.6/lib/python3.7/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag)
    484             )
    485         elif response is not None and response.status_code == 404:
--> 486             raise FileNotFoundError("Couldn't find file at {}".format(url))
    487         raise ConnectionError("Couldn't reach {}".format(url))
    488 

FileNotFoundError: Couldn't find file at https://homes.cs.washington.edu/~msap/social-bias-frames/SocialBiasFrames_v2.tgz
```
[Here](https://homes.cs.washington.edu/~msap/social-bias-frames/) we find button `Download data` with the correct URL for the data: https://homes.cs.washington.edu/~msap/social-bias-frames/SBIC.v2.tgz

Comments: I see, master is already fixed in https://github.com/huggingface/datasets/commit/9e058f098a0919efd03a136b9b9c3dec5076f626