jamesLeeeeeee/datasets-github-issues huggingface dataset:
Starting to add some real doc
Author: thomwolf | Pull request: yes | State: closed

Adding a lot of documentation for:
- load a dataset
- explore the dataset object
- process data with the dataset
- add a new dataset script
- share a dataset script
- full package reference

This version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html

Also:
- fix a bug in `train_test_split`
- update the `csv` script
- add a verbose argument to the dataset processing methods

Still missing:
- doc for the metrics
- how to directly upload a community provided dataset with the CLI
- clean up more docstrings
- add the `features` argument to `load_dataset` (should be another PR)
Add text dataset
Author: jarednielsen | Pull request: yes | State: closed

Usage:

```python
from nlp import load_dataset
dset = load_dataset("text", data_files="/path/to/file.txt")["train"]
```


I created a dummy_data.zip which contains three files: `train.txt`, `test.txt`, `dev.txt`. Each of these contains two lines. It passes

```bash
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_text
```

but I would like a second set of eyes to ensure I did it right.
can't load SNLI dataset
Author: jxmorris12 | Pull request: no | State: closed

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```
[Dataset requests] New datasets for Text Classification
Author: thomwolf | Pull request: no | State: open

We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705
add from_pandas and from_dict
Author: lhoestq | Pull request: yes | State: closed

I added two new methods to the `Dataset` class:
- `from_pandas()` to create a dataset from a pandas dataframe
- `from_dict()` to create a dataset from a dictionary (keys = columns)

It uses the `pa.Table.from_pandas` and `pa.Table.from_pydict` funcitons to do so.
It is also possible to specify the features types via `features=...` if there are ambiguities (null/nan values), otherwise the arrow schema is infered from the data automatically by pyarrow.

One question that I have right now:
+ Should we also add a `save()` method that would write the dataset on the disk ? Right now if we create a `Dataset` using those two new methods, the data are kept in RAM. Then to reload it we can call the `from_file()` method.
Hyperpartisan news detection
Author: ghomasHudson | Pull request: yes | State: closed

Adding the hyperpartisan news detection dataset from PAN. This contains news article text, labelled with whether they're hyper-partisan and why kinds of biases they display.

Implementation notes:
- As with many PAN tasks, the data is hosted on [Zenodo](https://zenodo.org/record/1489920) and must be requested before use. I've used the manual download stuff for this, although the dataset is provided under a Creative Commons Attribution 4.0 International License, so we could host a version if we wanted to?
- The 'bias' attribute doesn't exist for the 'byarticle' configuration. I've added an empty string to the class labels to deal with this. Is there a more standard value for empty data?
- Should we always subclass `nlp.BuilderConfig`?
'cp950' codec error from load_dataset('xtreme', 'tydiqa')
Author: jerryIsHere | Pull request: no | State: closed

![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
Supporting documents in ELI5
Author: saverymax | Pull request: no | State: closed

I was attempting to use the ELI5 dataset, when I realized that huggingface does not provide the supporting documents (the source documents from the common crawl). Without the supporting documents, this makes the dataset about as useful for my project as a block of cheese, or some other more apt metaphor.  According to facebook, the entire document collection is quite large. However, it would still be helpful to at least include a subset of the supporting documents i.e., having some data is better than having a block of cheese, in my case at least.

If you choose not to include them, it would be helpful to have documentation mentioning this specifically. It is especially confusing because the hf nlp ELI5 dataset has the key `'document'` but there are no documents to be found :(
Fix nested tensorflow format
Author: lhoestq | Pull request: yes | State: closed

In #339 and #337 we are thinking about adding a way to export datasets to tfrecords.

However I noticed that it was not possible to do `dset.set_format("tensorflow")` on datasets with nested features like `squad`. I fixed that using a nested map operations to convert features to `tf.ragged.constant`.

I also added tests on the `set_format` function.
Add dataset.export() to TFRecords
Author: jarednielsen | Pull request: yes | State: closed

Fixes https://github.com/huggingface/nlp/issues/337

Some design decisions:

- Simplified the function API to not handle sharding. It writes the entire dataset as a single TFRecord file. This simplifies the function logic and users can use other functions (`select`, `shard`, etc) to handle custom sharding or splitting.
- Use `from_generator()` instead of `from_tensor_slices()` to address the memory issues discussed in https://github.com/huggingface/nlp/issues/315 and https://github.com/huggingface/nlp/issues/193.
- Performs introspection using the values from `dataset.set_format()` to identify the TF datatypes. Currently it supports string, float, and int. If this should be extended for other datatypes, let me know.
- There are quite a few helper functions required within the `export()` method. If these are better placed in a utils file somewhere, let me know.

Also, I noticed that 
```python
dataset = dataset.select(indices)
dataset.set_format("tensorflow")
# dataset._format_type is "tensorflow"
```
gives a different output than
```python
dataset.set_format("tensorflow")
dataset = dataset.select(indices)
# dataset._format_type is None
```
The latter loses the format of its parent dataset. Is there interest in making `set_format` a functional method that returns itself (can be chained), and that derived datasets maintain the format of their parent?
[Feature request] Export Arrow dataset to TFRecords
Author: jarednielsen | Pull request: no | State: closed

The TFRecord generation process is error-prone and requires complex separate Python scripts to download and preprocess the data. I propose to combine the user-friendly features of `nlp` with the speed and efficiency of TFRecords. Sample API:

```python
# use these existing methods
ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
ds = ds.map(lambda ex: tokenizer(ex))
ds.set_format("tensorflow", columns=["input_ids", "token_type_ids", "attention_mask"])
# then add this method
ds.export(folder="/my/tfrecords", prefix="myrecord", num_shards=8, format="tfrecord")
```
which would create files like so:
```bash
/my/tfrecords/myrecord_1.tfrecord
/my/tfrecords/myrecord_2.tfrecord
...
```

I would be happy to contribute this method. We could use a similar approach for PyTorch. Thoughts?
[Dataset requests] New datasets for Open Question Answering
Author: thomwolf | Pull request: no | State: closed

We are still a few datasets missing for Open-Question Answering which is currently a field in strong development.

Namely, it would be really nice to add:
- WebQuestions (Berant et al., 2013) [done]
- CuratedTrec (Baudis et al. 2015) [not open-source]
- MS-MARCO (NGuyen et al. 2016) [done]
- SearchQA (Dunn et al. 2017) [done]
- FEVER (Thorne et al. 2018) - [ done]

 

All these datasets are cited in http://arxiv.org/abs/2005.11401
Add wiki_dpr
Author: lhoestq | Pull request: yes | State: closed

Presented in the [Dense Passage Retrieval paper](https://arxiv.org/pdf/2004.04906.pdf), this dataset consists in 21M passages from the english wikipedia along with their 768-dim embeddings computed using DPR's context encoder.

Note on the implementation:
- There are two configs: with and without the embeddings (73GB vs 14GB)
- I used a non-fixed-size sequence of floats to describe the feature format of the embeddings. I wanted to use fixed-size sequences but I had issues with reading the arrow file afterwards (for example `dataset[0]` was crashing)
- I added the case for lists of urls as input of the download_manager
Loading CNN/Daily Mail dataset produces `nlp.utils.info_utils.NonMatchingSplitsSizesError`
Author: jxmorris12 | Pull request: no | State: closed

```
>>> import nlp
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
```
Doc red
Author: ghomasHudson | Pull request: yes | State: closed

Adding [DocRED](https://github.com/thunlp/DocRED) - a relation extraction dataset which tests document-level RE. A few implementation notes:

- There are 2 separate versions of the training set - *annotated* and *distant*. Instead of `nlp.Split.Train` I've used the splits `"train_annotated"` and `"train_distant"` to reflect this.
- As well as the relation id, the full relation name is mapped from `rel_info.json`
- I renamed the 'h', 'r', 't' keys to 'head', 'relation' and 'tail' to make them more readable.
- Used the fix from #319 to allow nested sequences of dicts.
[Bug] FileLock dependency incompatible with filesystem
Author: jarednielsen | Pull request: no | State: closed

I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that.
Fork dataset
Author: timothyjlaurent | Pull request: no | State: closed

We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL
Large dataset in Squad2-format
Author: flozi00 | Pull request: no | State: closed

At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ?
Error when calculating glue score
Author: D-i-l-r-u-k-s-h-i | Pull request: no | State: closed

I was trying glue score along with other metrics here. But glue gives me this error;

```
import nlp
glue_metric = nlp.load_metric('glue',name="cola")

glue_score = glue_metric.compute(predictions, references)
```

```
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-b9210a524504> in <module>()
----> 1 glue_score = glue_metric.compute(predictions, references)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195 

/usr/local/lib/python3.6/dist-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210 
    211     def add(self, prediction=None, reference=None, **kwargs):

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/usr/local/lib/python3.6/dist-packages/pyarrow/types.pxi in __iter__()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

TypeError: an integer is required (got type str)
```
I'm not sure whether I'm doing this wrong or whether it's an issue. I would like to know a workaround. Thank you.
Add package path to sys when downloading package as github archive
Author: yjernite | Pull request: yes | State: closed

This fixes the `coval.py` metric so that imports within the downloaded module work correctly. We can use a similar trick to add the BLEURT metric (@ankparikh)

@thomwolf not sure how you feel about adding to the `PYTHONPATH` from the script. This is the only way I could make it work with my understanding of `importlib` but there might be a more elegant method.

This PR fixes https://github.com/huggingface/nlp/issues/305
output nested dict in get_nearest_examples
Author: lhoestq | Pull request: yes | State: closed

As we are using a columnar format like arrow as the backend for datasets, we expect to have a dictionary of columns when we slice a dataset like in this example:
```python
my_examples = dataset[0:10]
print(type(my_examples))
# >>> dict
print(my_examples["my_column"][0]
# >>> this is the first element of the column 'my_column'
```

Therefore I wanted to keep this logic when calling `get_nearest_examples` that returns the top 10 nearest examples:
```python
dataset.add_faiss_index(column="embeddings")
scores, examples = dataset.get_nearest_examples("embeddings", query=my_numpy_embedding)
print(type(examples))
# >>> dict
```

Previously it was returning a list[dict]. It was the only place that was using this output format.

To make it work I had to implement `__getitem__(key)` where `key` is a list.
This is different from `.select` because `.select` is a dataset transform (it returns a new dataset object) while `__getitem__` is an extraction method (it returns python dictionaries).
ERROR:root:mwparserfromhell
Author: Shiro-LK | Pull request: no | State: closed

Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`
Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer
Author: mariamabarham | Pull request: no | State: closed

Selecting `blog_authorship_corpus` in the nlp viewer throws the following error: 

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq