jamesLeeeeeee/datasets-github-issues huggingface dataset:
Bookcorpus data contains pretokenized text
Author: orsharir | Pull request: no | State: closed

It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
PAWS dataset first item is header
Author: jxmorris12 | Pull request: no | State: closed

```
import nlp
dataset = nlp.load_dataset('xtreme', 'PAWS-X.en')
dataset['test'][0]
```

prints the following

```
{'label': 'label', 'sentence1': 'sentence1', 'sentence2': 'sentence2'}
```

dataset['test'][0] should probably be the first item in the dataset, not just a dictionary mapping the column names to themselves. Probably just need to ignore the first row in the dataset by default or something like that.
Bugs : dataset.map() is frozen on ELI5
Author: ratthachat | Pull request: no | State: closed

Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) 

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
Apply utf-8 encoding to all datasets
Author: lewtun | Pull request: yes | State: closed

## Description
This PR applies utf-8 encoding for all instances of `with open(...) as f` to all Python files in `datasets/`. As suggested by @thomwolf in #468 , we use regular expressions and the following function

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""
    
    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"(?!.*\b(?:encoding|rb|w|wb|w+|wb+|ab|ab+)\b)(?<=\s)(open)\((.*)\)")
        input_text = input_file.read()
        match = regexp.search(input_text)
        
        if match:
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
```

to perform the replacement. 

Note:

1. I excluded all _**binary files**_ from the search since it's possible some objects are opened for which the encoding doesn't make sense. Please correct me if I'm wrong and I'll tweak the regexp accordingly
2. There were two edge cases where the regexp failed (e.g. two `open` instances on a single line), but I decided to just fix these manually in the interest of time.
3. I only applied the replacement to files in `datasets/`. Let me know if this should be extended to other places like `metrics/`
4. I have implemented a unit test that should catch missing encodings in future CI runs

Closes #468  and possibly #347
add METEOR metric
Author: vegarab | Pull request: yes | State: closed

Added the METEOR metric. Can be used like this:

```python
import nlp
meteor = nlp.load_metric('metrics/meteor')
meteor.compute(["some string", "some string"], ["some string", "some similar string"])
# {'meteor': 0.6411637931034483}
meteor.add("some string", "some string")
meteor.add('some string", "some similar string")
meteor.compute()
# {'meteor': 0.6411637931034483}
```

Uses [NLTK's implementation](https://www.nltk.org/api/nltk.translate.html#module-nltk.translate.meteor_score), [(source)](https://github.com/nltk/nltk/blob/develop/nltk/translate/meteor_score.py)
Export TFRecord to GCP bucket
Author: astariul | Pull request: no | State: closed

Previously, I was writing TFRecords manually to GCP bucket with : `with tf.io.TFRecordWriter('gs://my_bucket/x.tfrecord')`

Since `0.4.0` is out with the `export()` function, I tried it. But it seems TFRecords cannot be directly written to GCP bucket.

`dataset.export('local.tfrecord')` works fine,  
but `dataset.export('gs://my_bucket/x.tfrecord')` does not work. 

There is no error message, I just can't find the file on my bucket...

---

Looking at the code, `nlp` is using `tf.data.experimental.TFRecordWriter`, while I was using `tf.io.TFRecordWriter`.  

**What's the difference between those 2 ? How can I write TFRecords files directly to GCP bucket ?**

@jarednielsen @lhoestq
Overview.ipynb throws exceptions with nlp 0.4.0
Author: mandy-li | Pull request: no | State: closed

with nlp 0.4.0, the TensorFlow example in Overview.ipynb throws the following exceptions:


---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-48907f2ad433> in <module>
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

<ipython-input-5-48907f2ad433> in <dictcomp>(.0)
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

AttributeError: 'numpy.ndarray' object has no attribute 'to_tensor'
misc. bugs and quality of life
Author: joeddav | Pull request: yes | State: closed

A few misc. bugs and QOL improvements that I've come across in using the library. Let me know if you don't like any of them and I can adjust/remove them.

1. Printing datasets without a description field throws an error when formatting the `single_line_description`. This fixes that, and also adds some formatting to the repr to make it slightly more readable.
```
>>> print(list_datasets()[0])
nlp.ObjectInfo(
	id='aeslc',
	description='A collection of email messages of employees in the Enron Corporation.There are two features:  - email_body: email body text.  - subject_line: email subject text.',
	files=[nlp.S3Object('aeslc.py'), nlp.S3Object('dataset_infos.json'), nlp.S3Object('dummy/1.0.0/dummy_data-zip-extracted/dummy_data/AESLC-master/enron_subject_line/dev/allen-p_inbox_29.subject'), nlp.S3Object('dummy/1.0.0/dummy_data-zip-extracted/dummy_data/AESLC-master/enron_subject_line/test/allen-p_inbox_24.subject'), nlp.S3Object('dummy/1.0.0/dummy_data-zip-extracted/dummy_data/AESLC-master/enron_subject_line/train/allen-p_inbox_20.subject'), nlp.S3Object('dummy/1.0.0/dummy_data.zip'), nlp.S3Object('urls_checksums/checksums.txt')]
)
```

2. Add id-only option to `list_datasets` and `list_metrics` to allow the user to easily print out just the names of the datasets & metrics. I often found myself annoyed that this took so many strokes to do.

```python
[dataset.id for dataset in list_datasets()] # before
list_datasets(id_only=True) # after
```

3. Fix null-seed randomization caching. When using `train_test_split` and `shuffle`, the computation was being cached even without a seed or generator being passed. The result was that calling `.shuffle` more than once on the same dataset didn't do anything without passing a distinct seed or generator. Likewise with `train_test_split`.

4. Indexing by iterables of bool. I added support for passing an iterable of type bool to `_getitem` as a numpy/pandas-like indexing method. Let me know if you think it's redundant with `filter` (I know it's not optimal memory-wise), but I think it's nice to have as a lightweight alternative to do simple things without having to create a copy of the entire dataset, e.g.

```python
dataset[dataset['label'] == 0] # numpy-like bool indexing to look at instances with labels of 0
```

5. Add an `input_column` argument to `map` and `filter`, which allows you to filter/map on a particular column rather than passing the whole dict to the function. Also adds `fn_kwargs` to be passed to the function. I think these together make mapping much cleaner in many cases such as mono-column tokenization:

```python
# before
dataset = dataset.map(lambda batch: tokenizer(batch["text"])
# after
dataset = dataset.map(tokenizer, input_column="text")
dataset = dataset.map(tokenizer, input_column="text", fn_kwargs={"truncation": True, "padding": True})
```
test_load_real_dataset when config has BUILDER_CONFIGS that matter
Author: marcotcr | Pull request: no | State: closed

It a dataset has custom `BUILDER_CONFIGS` with non-keyword arguments (or keyword arguments with non default values), the config is not loaded during the test and causes an error.
I think the problem is that `test_load_real_dataset` calls `load_dataset` with `data_dir=temp_data_dir` ([here](https://github.com/huggingface/nlp/blob/master/tests/test_dataset_common.py#L200)). This causes [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L201) to always be false because `config_kwargs` is not `None`. [This line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L222) will be run instead, which doesn't use `BUILDER_CONFIGS`.

For an example, you can try running the test for lince:
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lince`
which yields
> E           TypeError: __init__() missing 3 required positional arguments: 'colnames', 'classes', and 'label_column'
Adding IWSLT 2017  dataset.
Author: Narsil | Pull request: yes | State: closed

Created a [IWSLT 2017](https://sites.google.com/site/iwsltevaluation2017/TED-tasks) dataset script for the *multilingual data*.

```
Bilingual data:  {Arabic, German, French, Japanese, Korean, Chinese} <-> English
Multilingual data:   German, English, Italian, Dutch, Romanian. (Any pair)
```

I'm unsure how to handle bilingual vs multilingual. Given `nlp` architecture a Config option seems to be the way to go, however, it might be a bit confusing to have different language pairs with different option. Using just language pairs is not viable as English to German exists in both.

Any opinion on how that should be done ?
EDIT: I decided to just omit de-en from multilingual as it's only a subset of the bilingual one. That way only language pairs exist.
EDIT : Could be interesting for #438
invalid data type 'str' at _convert_outputs in arrow_dataset.py
Author: Murgates | Pull request: no | State: closed

I trying to build multi label text classifier model using Transformers lib. 

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error 

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.  
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'
Keep features after transform
Author: lhoestq | Pull request: yes | State: closed

When applying a transform like `map`, some features were lost (and inferred features were used).
It was the case for ClassLabel, Translation, etc.

To fix that, I did some modifications in the `ArrowWriter`:

- added the `update_features` parameter. When it's `True`, then the features specified by the user (if any) can be updated with inferred features if their type don't match. `map` transform sets `update_features=True` when writing to cache file or buffer. Features won't change by default in `map`.

- added the `with_metadata` parameter. If `True`, the `features` (after update) will be written inside the metadata of the schema in this format:
```
{
    "huggingface": {"features" : <serialized Features exactly like dataset_info.json>}
} 
```
Then, once a dataset is instantiated without info/features, these metadata are used to set the features of the dataset.
Add rename, remove and cast in-place operations
Author: thomwolf | Pull request: yes | State: closed

Add a bunch of in-place operation leveraging the Arrow back-end to rename and remove columns and cast to new features without using the more expensive `map` method.

These methods are added to `Dataset` as well as `DatasetDict`.

Added tests for these new methods and add the methods to the doc.

Naming follows the new pattern with a trailing underscore indicating in-place methods.
[Breaking] Update Dataset and DatasetDict API
Author: thomwolf | Pull request: yes | State: closed

This PR contains a few breaking changes so it's probably good to keep it for the next (major) release:
- rename the `flatten`, `drop` and `dictionary_encode_column` methods in `flatten_`, `drop_` and `dictionary_encode_column_` to indicate that these methods have in-place effects as discussed in #166. From now on we should keep the convention of having a trailing underscore for methods which have an in-place effet. I also adopt the conversion of not returning the (self) dataset for these methods. This is different than what PyTorch does for instance (`model.to()` is in-place but return the self model) but I feel like it's a safer approach in terms of UX.
- remove the `dataset.columns` property which returns a low-level Apache Arrow object and should not be used by users. Similarly, remove `dataset. nbytes` which we don't really want to expose in this bare-bone format.
- add a few more properties and methods to `DatasetDict`
Add bleurt
Author: yjernite | Pull request: yes | State: closed

This PR adds the BLEURT metric to the library.

The BLEURT `Metric` downloads a TF checkpoint corresponding to its `config_name` at creation (in the `_info` function). Default is set to `bleurt-base-128`.

Note that the default in the original package is `bleurt-tiny-128`, but they throw a warning and recommend using `bleurt-base-128` instead. I think it's safer to have our users have a functioning metric when they call the default behavior, we'll address discrepancies in the issues/discussions if it comes up.

In addition to the BLEURT file, `load.py` was changed so we can ask users to pip install the required packages from git when they have a `setup.py` but are not on PyPL

cc @ankparikh @tsellam
Guardian authorship dataset
Author: malikaltakrori | Pull request: yes | State: closed

A new dataset: Guardian news articles for authorship attribution

**tests passed:**
python nlp-cli dummy_data datasets/guardian_authorship --save_infos --all_configs

RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_guardian_authorship

**Tests failed:**
Real data: RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_guardian_authorship
output: __init__() missing 3 required positional arguments: 'train_folder', 'valid_folder', and 'tes...' 

Remarks: This is the init function of my class. I am not sure why it passes in both my tests and with nlp-cli, but fails here. By the way, I ran this command with another 2 datasets and they failed:
* _glue - OSError: Cannot find data file.
*_newsgroup - FileNotFoundError: Local file datasets/newsgroup/dummy/18828_comp.graphics/3.0.0/dummy_data.zip doesn't exist

Thank you for letting us contribute to such a huge and important library! 

EDIT:
I was able to fix the dummy_data issue. This dataset has around 14 configurations. I was testing with only 2, but their versions were not in a sequence, they were V1.0.0 and V.12.0.0. It seems that the testing code generates testes for all the versions from 0 to MAX, and was testing for versions (and dummy_data.zip files) that do not exist. I fixed that by changing the versions to 1 and 2.
add reuters21578 dataset
Author: mariamabarham | Pull request: yes | State: closed

This PR adds the `Reuters_21578` dataset https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html  
#353 

The datasets is a lit of `.sgm` files which are a bit different from xml file indeed `xml.etree` couldn't be used to read files. I consider them as text file (to avoid using external library)  and read line by line (maybe there is a better way to do, happy to get your opinion on it)

In the Readme file 3 ways to split the dataset are given.:

- The Modified Lewis ("ModLewis") Split: train, test and unused-set

- The Modified Apte ("ModApte") Split : train, test and unused-set

- The Modified Hayes ("ModHayes") Split: train and test

Here I consider the last one as the readme file highlight that this split provides the ability to compare results with those of the 2 first splits.
DEFAULT_TOKENIZER import error in sacrebleu
Author: idoh | Pull request: no | State: closed

Latest Version 0.3.0

When loading the metric "sacrebleu" there is an import error due to the wrong path
![image](https://user-images.githubusercontent.com/5303103/88633063-2c5e5f00-d0bd-11ea-8ca8-4704dc975433.png)
Keep loading old file even I specify a new file in load_dataset
Author: joshhu | Pull request: no | State: closed

I used load a file called 'a.csv' by 
```
dataset = load_dataset('csv', data_file='./a.csv')
```
And after a while, I tried to load another csv called 'b.csv'
```
dataset = load_dataset('csv', data_file='./b.csv')
```
However, the new dataset seems to remain the old 'a.csv' and not loading new csv file.

Even worse, after I load a.csv, the load_dataset function keeps loading the 'a.csv' afterward. 

Is this a cache problem?
Cannot unpickle saved .pt dataset with torch.save()/load()
Author: vegarab | Pull request: no | State: closed

Saving a formatted torch dataset to file using `torch.save()`. Loading the same file fails during unpickling:

```python
>>> import torch
>>> import nlp

>>> squad = nlp.load_dataset("squad.py", split="train")
>>> squad
Dataset(features: {'source_text': Value(dtype='string', id=None), 'target_text': Value(dtype='string', id=None)}, num_rows: 87599)
>>> squad = squad.map(create_features, batched=True)
>>> squad.set_format(type="torch", columns=["source_ids", "target_ids", "attention_mask"])
>>> torch.save(squad, "squad.pt")

>>> squad_pt = torch.load("squad.pt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 773, in _legacy_load
    result = unpickler.load()
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/splits.py", line 493, in __setitem__
    raise ValueError("Cannot add elem. Use .add() instead.")
ValueError: Cannot add elem. Use .add() instead.
```
where `create_features` is a function that tokenizes the data using `batch_encode_plus` and returns a Dict with `input_ids`, `target_ids` and `attention_mask`. 
```python
def create_features(batch):
    source_text_encoding = tokenizer.batch_encode_plus(
        batch["source_text"],
        max_length=max_source_length,
        pad_to_max_length=True,
        truncation=True)

    target_text_encoding = tokenizer.batch_encode_plus(
        batch["target_text"],
        max_length=max_target_length,
        pad_to_max_length=True,
        truncation=True)

    features = {
        "source_ids": source_text_encoding["input_ids"],
        "target_ids": target_text_encoding["input_ids"],
        "attention_mask": source_text_encoding["attention_mask"]
    }

    return features
```

I found a similar issue in [issue 5267 in the huggingface/transformers repo](https://github.com/huggingface/transformers/issues/5267) which was solved by downgrading to `nlp==0.2.0`. That did not solve this problem, however.
[Suggestion] Glue Diagnostic Data with Labels 
Author: ggbetz | Pull request: no | State: open

Hello! First of all, thanks for setting up this useful project!

I've just realised you provide the the [Glue Diagnostics Data](https://huggingface.co/nlp/viewer/?dataset=glue&config=ax) without labels, indicating in the `GlueConfig` that you've only a test set.

Yet, the data with labels is available, too (see also [here](https://gluebenchmark.com/diagnostics#introduction)):

https://www.dropbox.com/s/ju7d95ifb072q9f/diagnostic-full.tsv?dl=1 

Have you considered incorporating it?
Issues: Adding a FAISS or Elastic Search index to a Dataset
Author: nsankar | Pull request: no | State: closed

It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?