helmo/github-issues huggingface dataset:
`list_datasets()` is broken.
State: closed | Author: ashutosh-dwivedi-e3502 (NONE)

version = '0.4.0'

`list_datasets()` is broken. It results in the following error : 

```
In [3]: nlp.list_datasets()
Out[3]: ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    375                 if cls in self.type_pprinters:
    376                     # printer registered in self.type_pprinters
--> 377                     return self.type_pprinters[cls](obj, self, cycle)
    378                 else:
    379                     # deferred printer

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in inner(obj, p, cycle)
    553                 p.text(',')
    554                 p.breakable()
--> 555             p.pretty(x)
    556         if len(obj) == 1 and type(obj) is tuple:
    557             # Special case for 1-item tuples.

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    392                         if cls is not object \
    393                                 and callable(cls.__dict__.get('__repr__')):
--> 394                             return _repr_pprint(obj, self, cycle)
    395
    396             return _default_pprint(obj, self, cycle)

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    698     """A pprint that just redirects to the normal repr function."""
    699     # Find newlines and replace them with p.break_()
--> 700     output = repr(obj)
    701     lines = output.splitlines()
    702     with p.group():

~/.virtualenvs/san-lgUCsFg_/lib/python3.8/site-packages/nlp/hf_api.py in __repr__(self)
    110
    111     def __repr__(self):
--> 112         single_line_description = self.description.replace("\n", "")
    113         return f"nlp.ObjectInfo(id='{self.id}', description='{single_line_description}', files={self.siblings})"
    114

AttributeError: 'NoneType' object has no attribute 'replace'
```

Comments: Thanks for reporting !
This has been fixed in #475 and the fix will be available in the next release

What you can do instead to get the list of the datasets is call

```python
print([dataset.id for dataset in nlp.list_datasets()])
```

Thanks @lhoestq .
Add MLSUM
State: closed | Author: RachelKer (CONTRIBUTOR)

Hello (again :) !), 

So, I started a new branch because of a [rebase issue](https://github.com/huggingface/nlp/pull/463), sorry for the mess. 

However, the command `pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_mlsum` still fails because there is no default language dataset : the script throws an error as a specific config language is necessary. 

I think that setting a default language would be a bad workaround for this so I kept it as it is. Putting all the train files across languages together would also be a bad idea because of the size. 

Thanks for your help, 

Rachel


Comments: Could you test to run the test using the changes in #527 and let me know if it fixes the issue ? If so I'll merge it and we'll be good to go :)

Hello, it does work on the fixing real dataset branch. Merci Quentin :)

Nice, glad to hear that :)
de rien !
Fix config used for slow test on real dataset
State: closed | Author: lhoestq (MEMBER)

As noticed in #470, #474, #476, #504 , the slow test `test_load_real_dataset` couldn't run on datasets that require config parameters.

To fix that I replaced it with one test with the first config of BUILDER_CONFIGS `test_load_real_dataset`, and another test that runs all of the configs in BUILDER_CONFIGS `test_load_real_dataset_all_configs`
Returning None instead of "python" if dataset is unformatted
State: closed | Author: TevenLeScao (CONTRIBUTOR)

Following the discussion on Slack, this small fix ensures that calling `dataset.set_format(type=dataset.format["type"])` works properly. Slightly breaking as calling `dataset.format` when the dataset is unformatted will return `None` instead of `python`.

Comments: We have to change the tests to expect `None` instead of `python` then

Merging!
wmt download speed example
State: closed | Author: sshleifer (CONTRIBUTOR)

Continuing from the slack 1.0 roadmap thread w @lhoestq , I realized the slow downloads is only a thing sometimes. Here are a few examples, I suspect there are multiple issues. All commands were run from the same gcp us-central-1f machine.

```
import nlp
nlp.load_dataset('wmt16', 'de-en')
```
Downloads at 49.1 KB/S

Whereas 
```
pip install gdown # download from google drive
!gdown https://drive.google.com/uc?id=1iO7um-HWoNoRKDtw27YUSgyeubn9uXqj
```
Downloads at 127 MB/s. (The file is a copy of wmt-en-de raw).


```
nlp.load_dataset('wmt16', 'ro-en')
```
goes at 27 MB/s, much faster. 

if we wget the same data from s3 is the same download speed, but ¼ the file size:
```
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_packed_200_rand.tgz
```

Finally,
```
nlp.load_dataset('wmt19', 'zh-en')
```
Starts fast, but broken. (duplicate of #493 )




Comments: Thanks for creating the issue :)
The download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.
Is this mirror official ?

Also it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool

Also cc @patrickvonplaten 

Mirror is not official.

Shall we host the files ourselves or it is fine to use this mirror in your opinion ?

Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?

This could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).

@lhoestq I think we should host it ourselves. I'll put the subset of wmt (without preprocessed files) that we need on s3 and post a link over the weekend.

Is there a solution yet? The download speed is still too slow. 60-70kbps download for wmt16 and around 100kbps for wmt19. @sshleifer 

I'm working on mirror links which will provide high download speed :)
See https://github.com/huggingface/datasets/issues/1892

Resolved via https://github.com/huggingface/datasets/pull/1912
Some docs are missing parameter names
State: closed | Author: jarednielsen (CONTRIBUTOR)

See https://huggingface.co/nlp/master/package_reference/main_classes.html#nlp.Dataset.map. I believe this is because the parameter names are enclosed in backticks in the docstrings, maybe it's an old docstring format that doesn't work with the current Sphinx version.

Comments: Indeed, good catch!
Speed up Tokenization by optimizing cast_to_python_objects
State: closed | Author: lhoestq (MEMBER)

I changed how `cast_to_python_objects` works to make it faster.
It is used to cast numpy/pytorch/tensorflow/pandas objects to python lists, and it works recursively.

To avoid iterating over possibly long lists, it first checks if the first element that is not None has to be casted.
If the first element needs to be casted, then all the elements of the list will be casted, otherwise they'll stay the same.
This trick allows to cast objects that contain tokenizers outputs without iterating over every single token for example.

Speed improvement:


```python
import transformers
import nlp

tok = transformers.BertTokenizerFast.from_pretrained("bert-base-uncased")
txt = ["a " * 512] * 1000
dataset = nlp.Dataset.from_dict({"txt": txt})

# Tokenization using .map is now faster. Previously it was taking 3.5s
%time _ = dataset.map(lambda x: tok(x["txt"]), batched=True, load_from_cache_file=False)
# 450ms

# for comparison
%time _ = tok(txt)
# 280ms

```

Comments: I took your comments into account and added tests for `cast_to_python_objects`
Transform references for sacrebleu
State: closed | Author: jbragg (CONTRIBUTOR)

Currently it is impossible to use sacrebleu when len(predictions) != the number of references per prediction (very uncommon), due to a strange format expected by sacrebleu. If one passes in the data to `nlp.metric.compute()` in sacrebleu format, `nlp` throws an error due to mismatching lengths between predictions and references. If one uses a more standard format where predictions and references are lists of the same length, sacrebleu throws an error.

This PR transforms reference data in a more standard format into the [unusual format](https://github.com/mjpost/sacreBLEU#using-sacrebleu-from-python) expected by sacrebleu.

Comments: I think I agree @lhoestq so I pushed a change.
Thanks for your work on the library!
[BUG] Metrics throwing new error on master since 0.4.0
State: closed | Author: jbragg (CONTRIBUTOR)

The following error occurs when passing in references of type `List[List[str]]` to metrics like bleu.
Wasn't happening on 0.4.0 but happening now on master.

```
  File "/usr/local/lib/python3.7/site-packages/nlp/metric.py", line 226, in compute
    self.add_batch(predictions=predictions, references=references)
  File "/usr/local/lib/python3.7/site-packages/nlp/metric.py", line 242, in add_batch
    batch = self.info.features.encode_batch(batch)
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 527, in encode_batch
    encoded_batch[key] = [encode_nested_example(self[key], cast_to_python_objects(obj)) for obj in column]
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 527, in <listcomp>
    encoded_batch[key] = [encode_nested_example(self[key], cast_to_python_objects(obj)) for obj in column]
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 456, in encode_nested_example
    raise ValueError("Got a string but expected a list instead: '{}'".format(obj))
```

Comments: Update - maybe this is only failing on bleu because I was not tokenizing inputs to the metric

Closing - seems to be just forgetting to tokenize. And found the helpful discussion in huggingface/evaluate#105
[METRICS, breaking] Refactor caching behavior, pickle/cloudpickle metrics and dataset, add tests on metrics
State: closed | Author: thomwolf (MEMBER)

Move the acquisition of the filelock at a later stage during metrics processing so it can be pickled/cloudpickled after instantiation.

Also add some tests on pickling, concurrent but separate metric instances and concurrent and distributed metric instances.

Changes significantly the caching behavior for the metrics:
- if the metric is used in a non-distributed setup (most common case) we try to find a free cache file using UUID instead of asking for an `experiment_id` if we can't lock the cache file this allows to use several instances of the same metrics in parallel.
- if the metrics is used in a distributed setup we ask for an `experiment_id` if we can't lock the cache file (because all the nodes need to have related cache file names for the final sync.
- after the computation, we free the locks and delete all the cache files.

Breaking: Some arguments for Metrics initialization have been removed for simplicity (`version`...) and some have been renamed for consistency with the rest of the library (`in_memory` => `keep_in_memory`).

Also remove the `_has_transformers` detection in utils to avoid importing transformers everytime during loading.

Comments: (test failure is unrelated)

As discussed with @thomwolf merging since the hyperparameter-search has been merged in transformers.
add MLDoc dataset
State: open | Author: jxmorris12 (CONTRIBUTOR)
Labels: name: dataset request

Hi,

I am recommending that someone add MLDoc, a multilingual news topic classification dataset.

- Here's a link to the Github: https://github.com/facebookresearch/MLDoc
- and the paper: http://www.lrec-conf.org/proceedings/lrec2018/pdf/658.pdf

Looks like the dataset contains news stories in multiple languages that can be classified into four hierarchical groups: CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). There are 13 languages: Dutch, French, German, Chinese, Japanese, Russian, Portuguese, Spanish, Latin American Spanish, Italian, Danish, Norwegian, and Swedish

Comments: Any updates on this?

This request is still an open issue waiting to be addressed by any community member, @GuillemGSubies.
[speedup] Use indices mappings instead of deepcopy for all the samples reordering methods
State: closed | Author: thomwolf (MEMBER)

Use an indices mapping instead of rewriting the dataset for all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`).

Added a `flatten_indices` method which copy the dataset to a new table to remove the indices mapping with tests.

All the samples re-ordering/selection methods should be a lot faster. The downside is that iterating on very large batch of the dataset might be a little slower when we have changed the order of the samples since with in these case we use `pyarrow.Table.take` instead of `pyarrow.Table.slice`. There is no free lunch but the speed of iterating over the dataset is rarely the bottleneck.

*Backward breaking change*: the `cache_file_name` argument in all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`) is now called `indices_cache_file_name` on purpose to make it explicit to the user that this caching file is used for caching the indices mapping and not the dataset itself.

Comments: Ok I fixed `concatenate_datasets` and added tests
Feel free to merge if it's good for you @thomwolf 

Ok, adding some benchmarks for map/filters and then I'll merge

Warning from pytorch that we should maybe consider at some point @lhoestq:
```
/__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
and PyTorch does not support non-writeable tensors. This means you can write to the underlying
(supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
protect its data or make it writeable before converting it to a tensor. This type of warning will be
suppressed for the rest of this program.
(Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
532
  return torch.tensor(x, **format_kwargs)
```

> Warning from pytorch that we should maybe consider at some point @lhoestq:
> 
> ```
> /__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
> and PyTorch does not support non-writeable tensors. This means you can write to the underlying
> (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
> protect its data or make it writeable before converting it to a tensor. This type of warning will be
> suppressed for the rest of this program.
> (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
> 532
>   return torch.tensor(x, **format_kwargs)
> ```

Not sure why we have that, it's probably linked to zero copy from arrow to numpy
Version of numpy to use the library
State: closed | Author: isspek (NONE)

Thank you so much for your excellent work! I would like to use nlp library in my project. While importing nlp, I am receiving the following error `AttributeError: module 'numpy.random' has no attribute 'Generator'` Numpy version in my project is 1.16.0. May I learn which numpy version is used for the nlp library.

Thanks in advance.

Comments: Seems like this method was added in 1.17. I'll add a requirement on this.

Thank you so much. After upgrading the numpy library, it worked.
Converting TensorFlow dataset example
State: closed | Author: saareliad (NONE)

Hi,
I want to use TensorFlow datasets with this repo, I noticed you made some conversion script,
can you give a simple example of using it?

Thanks


Comments: Do you want to convert a dataset script to the tfds format ?
If so, we currently have a comversion script nlp/commands/convert.py but it is a conversion script that goes from tfds to nlp.
I think it shouldn't be too hard to do the changes in reverse (at some manual adjustments).
If you manage to make it work in reverse, feel free to open a PR to share it with the community :)

In our docs: [Using a Dataset with PyTorch/Tensorflow](https://huggingface.co/docs/datasets/torch_tensorflow.html).
TypeError: Receiver() takes no arguments
State: closed | Author: sebastiantomac (NONE)

I am trying to load a wikipedia data set

```
import nlp
from nlp import load_dataset

dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=data_path, beam_runner='DirectRunner')
#dataset = load_dataset('wikipedia', '20200501.sv', cache_dir=data_path, beam_runner='DirectRunner')
```

This fails in the apache beam runner. 

```
Traceback (most recent call last):
  File "D:/ML/wikiembedding/gpt2_sv.py", line 36, in <module>
    dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=my_cache_dir, beam_runner='DirectRunner')
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 462, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 969, in _download_and_prepare
    pipeline_results = pipeline.run()
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\pipeline.py", line 534, in run
    return self.runner.run_pipeline(self, self._options)
....
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 218, in process_encoded
    self.output(decoded_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\operations.py", line 332, in output
    cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\Cython\Shadow.py", line 167, in cast
    return type(*args)
TypeError: Receiver() takes no arguments

```

This is run on a Windows 10 machine with python 3.8. I get the same error loading the swedish wikipedia dump.

Comments: Which version of Apache Beam do you have (can you copy your full environment info here)?

apache-beam==2.23.0
nlp==0.4.0

For me this was resolved by running the same python script on Linux (or really WSL). 

Do you manage to run a dummy beam pipeline with python on windows ? 
You can test a dummy pipeline with [this code](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py)

If you get the same error, it means that the issue comes from apache beam.
Otherwise we'll investigate what went wrong here

Still, same error, so I guess it is on apache beam then. 
Thanks for the investigation.

Thanks for trying
Let us know if you find clues of what caused this issue, or if you find a fix
Errors when I use 
State: closed | Author: mchari (NONE)

I tried the following example code from https://huggingface.co/deepset/roberta-base-squad2 and got errors 
I am using **transformers 3.0.2** code .


from transformers.pipelines import pipeline
from transformers.modeling_auto import AutoModelForQuestionAnswering
from transformers.tokenization_auto import AutoTokenizer

model_name = "deepset/roberta-base-squad2"

nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
QA_input = {
    'question': 'Why is model conversion important?',
    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
}
res = nlp(QA_input)

The errors are :

res = nlp(QA_input)
  File ".local/lib/python3.6/site-packages/transformers/pipelines.py", line 1316, in __call__
    for s, e, score in zip(starts, ends, scores)
  File ".local/lib/python3.6/site-packages/transformers/pipelines.py", line 1316, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 0



Comments: Looks like an issue with 3.0.2 transformers version. Works fine when I use "master" version of transformers.
tmp_file referenced before assignment
State: closed | Author: avloss (NONE)

Just learning about this library - so might've not set up all the flags correctly, but was getting this error about "tmp_file".

Comments: Thanks for reporting the issue ! I'm creating a new PR to fix it and add tests.
(I'm doing a new PR because I know there's some other place where it needs to be fixed)

I'm closing this one as I created the other PR.
Added downloading to Hyperpartisan news detection
State: closed | Author: ghomasHudson (CONTRIBUTOR)

Following the discussion on Slack and #349, I've updated the hyperpartisan dataset to pull directly from Zenodo rather than manual install, which should make this dataset much more accessible. Many thanks to @johanneskiesel !

Currently doesn't pass `test_load_real_dataset` - I'm using `self.config.name` which is `default` in this test. Might be related to #474

Comments: Thank you @ghomasHudson for making our dataset available! This is great!

The test passes since #527 :)