helmo/github-issues huggingface dataset:
Replace seqeval code with original classification_report for simplicity
State: closed | Author: Hironsan (CONTRIBUTOR)

Recently, the original seqeval has enabled us to get per type scores and overall scores as a dictionary.

This PR replaces the current code with the original function(`classification_report`) to simplify it.

Also, the original code has been updated to fix #352.
- Related issue: https://github.com/chakki-works/seqeval/pull/38


```python
from datasets import load_metric
metric = load_metric("seqeval")
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
metric.compute(predictions=y_pred, references=y_true)
# Output: {'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

Comments: Hello,

I ran https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh

And received this error:
```
100%|██████████| 407/407 [21:37<00:00,  3.44s/it]Traceback (most recent call last):
  File "run_ner.py", line 445, in <module>
    main()
  File "run_ner.py", line 398, in main
    results = trainer.evaluate()
  File "/data/2021/transformers/src/transformers/trainer.py", line 1470, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/data/2021/transformers/src/transformers/trainer.py", line 1622, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "run_ner.py", line 345, in compute_metrics
    results = metric.compute(predictions=true_predictions, references=true_labels)
  File "/usr/local/lib/python3.6/dist-packages/datasets/metric.py", line 398, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py", line 97, in _compute
    report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

I'm still trying multiple things to see if I can work around this, but I thought it might be useful to mention it here.

```
Name: transformers
Version: 4.3.0.dev0

Name: datasets
Version: 1.2.1
```

Hi, can you try to update your local installation of `seqeval` ?

```
pip install --upgrade seqeval
```

@lhoestq thanks for the reply. Indeed it was some issue with my setup. I removed the "transformers" and "datasets" (that I had previously installed from the source code), cleared the cache and installed everything again. It works great now!
Trec Dataset Connection Error
State: closed | Author: aychang95 (NONE)

**Datasets Version:**
1.1.2

**Python Version:**
3.6/3.7


**Code:**
```python
from datasets import load_dataset
load_dataset("trec")
```

**Expected behavior:**
Download Trec dataset and load Dataset object

**Current Behavior:**
Get a connection error saying it couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label (but the link doesn't seem broken)

<details>
  <summary>Error Logs</summary>
 

Using custom data configuration default
Downloading and preparing dataset trec/default (download: 350.79 KiB, generated: 403.39 KiB, post-processed: Unknown size, total: 754.18 KiB) to /root/.cache/huggingface/datasets/trec/default/1.1.0/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-8-66bf1242096e> in <module>()
----> 1 load_dataset("trec")

10 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag)
    473         elif response is not None and response.status_code == 404:
    474             raise FileNotFoundError("Couldn't find file at {}".format(url))
--> 475         raise ConnectionError("Couldn't reach {}".format(url))
    476 
    477     # Try a second time

ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label

</details>

Comments: Thanks for reporting.
That's because the download url has changed. The old url now redirects to the new one but we don't support redirection for downloads.

I'm opening a PR to update the url
Start community-provided dataset docs
State: closed | Author: sshleifer (CONTRIBUTOR)

This is one I did to get the pseudo-labels updated. Not sure if it generalizes, but I figured I would write it down. It was pretty easy because all I had to do was make properly formatted directories and change URLs.

+ In slack @thomwolf  called it a `user-namespace` dataset, but the docs call it `community dataset`.
I think the first naming is clearer, but I didn't address that here.


+ I didn't add metadata, will try that.

Comments: can you also reference the `--organization` flag like in https://github.com/huggingface/transformers/blob/master/docs/source/model_sharing.rst#upload-your-model-with-the-cli ?

done!

Not sure if the changes in `datasets/wmt_t2t/wmt_utils.py` are intentional.
If you want to add more configs to wmt, could you do it in a serapate PR ?

I don't think I changed wmt_utils (I think github is wrong or my setup is poorly configured).

Locally git diff master --name-only says one file. Master is up to date.
Tried to make a new PR #755 and the same thing happened.

Trying new fork.
Throw error when an unexpected key is used in data_files
State: closed | Author: BramVanroy (CONTRIBUTOR)

I have found that only "train", "validation" and "test" are valid keys in the `data_files` argument. When you use any other ones, those attached files are silently ignored - leading to unexpected behaviour for the users.

So the following, unintuitively, returns only one key (namely `train`).

```python
datasets = load_dataset("text", data_files={"train": train_f, "valid": valid_f})
print(datasets.keys())
# dict_keys(['train'])
```

whereas using `validation` instead, does return the expected result:

```python
datasets = load_dataset("text", data_files={"train": train_f, "validation": valid_f})
print(datasets.keys())
# dict_keys(['train', 'validation'])
```

I would like to see more freedom in which keys one can use, but if that is not possible at least an error should be thrown when using an unexpected key.

Comments: Thanks for reporting !
We'll add support for other keys
dataset(wlasl): initial loading script
State: closed | Author: AmitMY (CONTRIBUTOR)

takes like 9-10 hours to download all of the videos for the dataset, but it does finish :) 

Comments: Followup: 
From the info in https://github.com/huggingface/datasets/pull/722, I probably should load the videos as array of frames directly into the database. 
This will make the dataset generation time very long, but will make working with the dataset much easier.

When I run:
```
python datasets-cli dummy_data datasets/wlasl
```

I get:
```
Checking datasets/wlasl/wlasl.py for additional imports. 
Found main folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl 
Found specific version folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786  
Found script file from datasets/wlasl/wlasl.py to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.py 
Found dataset infos file from datasets/wlasl/dataset_infos.json to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/dataset_infos.json 
Found metadata file for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.json 
Using custom data configuration default 
Loading Dataset Infos from /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786
Creating dummy folder structure for datasets/wlasl/dummy/0.3.0...   
Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data. 
Traceback (most recent call last):      
File "datasets-cli", line 36, in         
service.run()                                                                                                                                                                                                                                                                                                                                                                                                            File "/home/nlp/amit/anaconda2/envs/meta-scholar/lib/python3.7/site-packages/datasets-1.1.2-py3.7.egg/datasets/commands/dummy_data.py", line 73, in run 
for split in generator_splits: 
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```
dataset(aslg_pc12): initial loading script
State: closed | Author: AmitMY (CONTRIBUTOR)

This contains the only current public part of this corpus.

The rest of the corpus is not yet been made public, but this sample is still being used by researchers.

Comments: Thanks @lhoestq 
Are there any guidelines for the dummy data?
In this particular case for example, the dataset fetches from two hardcoded URLs. 
Do I just `head -n 10` both files and zip them?



> Thanks @lhoestq
> Are there any guidelines for the dummy data?
> In this particular case for example, the dataset fetches from two hardcoded URLs.
> Do I just `head -n 10` both files and zip them?

Yes the idea is just to have a few examples to properly test the script and make sure it keeps working in the long run.

And FYI there's a command to help you name the dummy data files correctly. More info in the documentation [here](https://huggingface.co/docs/datasets/share_dataset.html#adding-dummy-data)

@lhoestq passes all tests
Better error message when one forgets to call `add_batch` before `compute`
State: closed | Author: sgugger (CONTRIBUTOR)

When using metrics, if for some reason a user forgets to call `add_batch` to a metric before `compute` (with no arguments), the error message is a bit cryptic and could probably be made clearer.

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
     pass # User forgets to call `add_batch`
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-267729d187fa> in <module>
      3      pass
      4     # metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 5 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383 
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    343         elif self.process_id == 0:
    344             # Let's acquire a lock on each node files to be sure they are finished writing
--> 345             file_paths, filelocks = self._get_all_cache_files()
    346 
    347             # Read the predictions and references

~/git/datasets/src/datasets/metric.py in _get_all_cache_files(self)
    280         filelocks = []
    281         for process_id, file_path in enumerate(file_paths):
--> 282             filelock = FileLock(file_path + ".lock")
    283             try:
    284                 filelock.acquire(timeout=self.timeout)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
```
Parallel downloads progress bar flickers
State: open | Author: lhoestq (MEMBER)

When there are parallel downloads using the download manager, the tqdm progress bar flickers since all the progress bars are on the same line.

To fix that we could simply specify `position=i` for i=0 to n the number of files to download when instantiating the tqdm progress bar. 

Another way would be to have one "master" progress bar that tracks the number of finished downloads, and then one progress bar per process that show the current downloads.
pretty print dataset objects
State: closed | Author: stas00 (CONTRIBUTOR)

Currently, if I do:
```
from datasets import load_dataset
load_dataset("wikihow", 'all', data_dir="/hf/pegasus-datasets/wikihow/")
```
I get:
```

DatasetDict({'train': Dataset(features: {'text': Value(dtype='string', id=None),
'headline': Value(dtype='string', id=None), 'title': Value(dtype='string',
id=None)}, num_rows: 157252), 'validation': Dataset(features: {'text':
Value(dtype='string', id=None), 'headline': Value(dtype='string', id=None),
'title': Value(dtype='string', id=None)}, num_rows: 5599), 'test':
Dataset(features: {'text': Value(dtype='string', id=None), 'headline':
Value(dtype='string', id=None), 'title': Value(dtype='string', id=None)},
num_rows: 5577)})
```

This is not very readable. 

Can we either have a better `__repr__` or have a custom method to nicely pprint the dataset object? 

Here is my very simple attempt. With this PR, it produces:
```
DatasetDict({
  train:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 157252
  })
  validation:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5599
  })
  test:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
  })
})
```
I did omit the data types on purpose to make it more readable, but it shouldn't be too difficult to integrate those too.

note that this PR also fixes the inconsistency in output that in master misses enclosing `{}` for Dataset, but it is there for `DatasetDict` - or perhaps it was by design.

I'm totally not attached to this format, just wanting something more readable. One approach could be to serialize to `json.dumps` or something similar. It'd make the indentation simpler.

Thank you.

Comments: Great, as you found it useful I improved the code a bit to automate indentation in the parent class, so that the child repr doesn't need to guess the indentation level, while repr'ing nicely on its own.

- do we want indent=4 or 2?
- do we want `{` ... `}` or w/o?

currently it's indent4 and w/ curly braces, so it looks:

```
DatasetDict({
    train: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 157252
    })
    validation: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5599
    })
    test: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5577
    })
})
```
just child:
```
Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
})
```



Yes!  A lot better indeed!
need to redirect /nlp to /datasets and remove outdated info
State: closed | Author: stas00 (CONTRIBUTOR)

It looks like the website still has all the `nlp` data, e.g.: https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all

should probably redirect to: https://huggingface.co/datasets/wikihow

also for some reason the new information is slightly borked. If you look at the old one it was nicely formatted and had the links marked up, the new one is just a jumble of text in one chunk and no markup for links (i.e. not clickable).

Comments: Should be fixed now: 

![image](https://user-images.githubusercontent.com/35882/95917301-040b0600-0d78-11eb-9655-c4ac0e788089.png)

Not sure I understand what you mean by the second part?


Thank you!

> Not sure I understand what you mean by the second part?

Compare the 2:
* https://huggingface.co/datasets/wikihow
* https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all
Can you see the difference? 2nd has formatting, 1st doesn't.


For context, those are two different pages (not an old vs new one), one is from the dataset viewer (you can browse data inside the datasets) while the other is just a basic reference page displayed some metadata about the dataset.

For the second one, we'll move to markdown parsing soon, so it'll be formatted better.

I understand. I was just flagging the lack of markup issue.
Adding pseudo-labels to datasets
State: closed | Author: sshleifer (CONTRIBUTOR)

I recently [uploaded pseudo-labels](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/precomputed_pseudo_labels.md) for CNN/DM, XSUM and WMT16-en-ro to s3, and thom mentioned I should add them to this repo.
Since pseudo-labels are just a large model's generations on an existing dataset, what is the right way to structure this contribution.
I read https://huggingface.co/docs/datasets/add_dataset.html, but it doesn't really cover this type of contribution.

I could, for example, make a new directory, `xsum_bart_pseudolabels` for each set of pseudolabels or add some sort of parametrization to `xsum.py`: https://github.com/huggingface/datasets/blob/5f4c6e830f603830117877b8990a0e65a2386aa6/datasets/xsum/xsum.py

What do you think @lhoestq ?




Comments: Nice ! :)
It's indeed the first time we have such contributions so we'll have to figure out the appropriate way to integrate them.
Could you add details on what they could be used for ?


They can be used as training data for a smaller model.

Sounds just like a regular dataset to me then, no?

A new configuration for those datasets should do the job then.
Note that until now datasets like xsum only had one configuration. It means that users didn't have to specify the configuration name when loading the dataset. If we add new configs, users that update the lib will have to update their code to specify the default/standard configuration name (not the one with pseudo labels).

Could also be a `user-namespace` dataset maybe?

Oh yes why not. I'm more in favor of this actually since pseudo labels are things that users (not dataset authors in general) can compute by themselves and share with the community

![image](https://user-images.githubusercontent.com/6045025/96045248-b528a380-0e3f-11eb-9124-bd55afa031bb.png)

I assume I should (for example) rename the xsum dir, change the URL, and put the modified dir somewhere in S3?

You can use the `datasets-cli` to upload the folder with your version of xsum with the pseudo labels.

```
datasets-cli upload_dataset path/to/xsum
```
datasets(RWTH-PHOENIX-Weather 2014 T): add initial loading script
State: closed | Author: AmitMY (CONTRIBUTOR)
Labels: name: dataset contribution

This is the first sign language dataset in this repo as far as I know.
Following an old issue I opened https://github.com/huggingface/datasets/issues/302.

I added the dataset official REAMDE file, but I see it's not very standard, so it can be removed.



Comments: This might be interesting to @kayoyin the author of https://github.com/kayoyin/transformer-slt – pinging you just in case :)

Thanks Amit, this is a great idea! I'm thinking of porting the SLT models from my paper here as well, having this dataset would be perfect for that :)

Thanks for your contribution, @AmitMY. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Fix train_test_split output format
State: closed | Author: lhoestq (MEMBER)

There was an issue in the `transmit_format` wrapper that returned bad formats when using train_test_split.
This was due to `column_names` being handled as a List[str] instead of Dict[str, List[str]] when the dataset transform (train_test_split) returns a DatasetDict (one set of column names per split).

This should fix @timothyjlaurent 's issue in #620 and fix #676 

I added tests for `transmit_format` so that it doesn't happen again
Don't use tqdm 4.50.0
State: closed | Author: lhoestq (MEMBER)

tqdm 4.50.0 introduced permission errors on windows
see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111) for the error details.

For now I just added `<4.50.0` in the setup.py
Hopefully we can find what's wrong with this version soon
Use python read for text dataset
State: closed | Author: lhoestq (MEMBER)

As mentioned in #622 the pandas reader used for text dataset doesn't work properly when there are \r characters in the text file.

Instead I switched to pure python using `open` and `read`.
From my benchmark on a 100MB text file, it's the same speed as the previous pandas reader.

Comments: One thing though, could we try to read the files in parallel?

We could but I'm not sure this would help a lot since the bottleneck is the drive IO if the files are big enough.
It could make sense for very small files.

Looks like windows is not a big fan of this approach
I'm working on a fix

I remember issue https://github.com/huggingface/datasets/issues/546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

I think it's worth testing multiprocessing. It could also be something we add to our speed benchmarks

> I remember issue #546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

It still would be interesting I think, especially in scenarios where IO is less of an issue (SSDs particularly) and where there are many smaller files. Wrapping this function in a `pool.map` is perhaps an easy thing to try. 

Merging this one for now for the patch release