ahmedfarazsyk/datasets-github-issues huggingface dataset:
Add SD task for SUPERB
Author: albertvillanova
State: closed
Labels: name: dataset request

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun 


Comments: Note that this subset requires us to:

* generate the LibriMix corpus from LibriSpeech
* prepare the corpus for diarization

As suggested by @lhoestq we should perform these steps locally and add the prepared data to this public repo on the Hub: https://huggingface.co/datasets/superb/superb-data

Then we can use the URLs for the files to load the data in `superb`'s dataset loading script.

For consistency, I suggest we name the folders in `superb-data` in the same way as the configs in the dataset loading script - e.g. use `sd` for speech diarization in both places :)

@lewtun @lhoestq: 

I have already generated the LibriMix corpus and prepared the corpus for diarization. The output is 3 dirs (train, dev, test), each one containing 6 files: reco2dur  rttm  segments  spk2utt  utt2spk  wav.scp

Next steps:
- Upload these files to the superb-data repo
- Transcribe the corresponding s3prl processing of these files into our superb loading script

Note that processing of these files is a bit more intricate than usual datasets: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/dataset.py#L233
Setting log level higher than warning does not suppress progress bar
Author: Isa-rentacs
State: closed
Labels: name: bug

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown 

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1


Comments: Hi,

you can suppress progress bars by patching logging as follows:
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda: logging.NOTSET
# map call ...
```
EDIT: now you have to use `disable_progress_bar `

Thank you, it worked :)

See https://github.com/huggingface/datasets/issues/2528 for reference

Note also that you can disable the progress bar with

```python
from datasets.utils import disable_progress_bar
disable_progress_bar()
```

See https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/src/datasets/utils/tqdm_utils.py#L84

Now the library officially recommends `set_progress_bar_enabled(False)`

```py
from datasets.utils import set_progress_bar_enabled

set_progress_bar_enabled(False)
```

source:

https://github.com/huggingface/datasets/blob/1fd47120ace13626c528367787ffa13e1a26e6c0/src/datasets/utils/tqdm_utils.py#L83-L88



From https://github.com/huggingface/datasets/pull/3897, `disable_progress_bar` is the function you should use

Now ``disable_progress_bar`` function is in ``datasets/src/datasets/utils/logging.py``.
https://github.com/huggingface/datasets/blob/aa555a299ad73c65e3f997a764e9d211675ab05d/src/datasets/utils/logging.py#L233-L236
And the method mentioned in https://github.com/huggingface/datasets/issues/2651#issuecomment-880270774 is not working now.
[load_dataset] shard and parallelize the process
Author: stas00
State: closed
Labels: name: enhancement

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq 

Comments: I need the same feature for distributed training

I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :)

Posted a PR to address the building side, still needs something to load sharded arrow files + tests

Closing as this feature has been implemented in #5107
adding progress bar / ETA for `load_dataset`
Author: stas00
State: open
Labels: name: enhancement

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq 


Comments: Is this done now? I see progress bars when using `load_dataset`.

There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
Add web_split dataset for Paraphase and Rephrase benchmark
Author: bhadreshpsavani
State: open
Labels: name: enhancement

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase



Comments: #take
downloading of yahoo_answers_topics dataset failed
Author: vikrant7k
State: closed
Labels: name: bug

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.


## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files



Comments: Hi ! I just tested and it worked fine today for me.

I think this is because the dataset is stored on Google Drive which has a quota limit for the number of downloads per day, see this similar issue https://github.com/huggingface/datasets/issues/996 

Feel free to try again today, now that the quota was reset

Fixed once data URL was replaced:
- #4023
load_dataset processing failed with OS error after downloading a dataset
Author: fake-warrior8
State: closed
Labels: name: bug

## Describe the bug
After downloading a dataset like opus100, there is a bug that 
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS


## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0




Comments: Hi ! It looks like an issue with pytorch.

Could you try to run `import torch` and see if it raises an error ?

> Hi ! It looks like an issue with pytorch.
> 
> Could you try to run `import torch` and see if it raises an error ?

It works. Thank you!
Support multi-worker with streaming dataset (IterableDataset).
Author: changjonathanc
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


Comments: Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

Any update on this feature request?

Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
Progress bars are not properly rendered in Jupyter notebook
Author: albertvillanova
State: closed
Labels: name: bug

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf 

Comments: To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Load datasets from the Hub without requiring a dataset script
Author: lhoestq
State: closed

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

Comments: This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
can't set verbosity for `metric.py`
Author: thomas-happify
State: closed
Labels: name: bug

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places. 
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected 
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation. 

I had to do 
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
``` 
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS 
- Python version: 3.8.10
- PyArrow version: 3.0.0


Comments: Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.
Integration with AugLy
Author: Darktex
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Comments: Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

Preprocessing functions/augmentations, unless super generic, should be defined in separate libraries, so I'm closing this issue.
Jsonlines export error
Author: TevenLeScao
State: closed
Labels: name: bug

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1

Comments: Thanks for reporting @TevenLeScao! I'm having a look...

(not sure what just happened on the assignations sorry)

For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.

@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898

Thanks ! I'm creating a PR

Well I though it was me who has taken on this issue... 😅 

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄 

The fix is available on `master` @TevenLeScao , thanks for reporting
[Metrics] addition of wiki_split metrics
Author: bhadreshpsavani
State: closed
Labels: name: enhancement

name: metric request

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this 
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py) 

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign

Comments: #take
Crash when using multiprocessing (`num_proc` > 1) on `filter` and all samples are discarded
Author: mxschmdt
State: closed
Labels: name: bug

## Describe the bug
If `filter` is applied to a dataset using multiprocessing (`num_proc` > 1) and all sharded datasets are empty afterwards (due to all samples being discarded), the program crashes.

## Steps to reproduce the bug
```python
from datasets import Dataset
data = Dataset.from_dict({'id': [0,1]})
data.filter(lambda x: False, num_proc=2)
```

## Expected results
An empty table should be returned without crashing.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2143, in filter
    return self.map(
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1738, in map
    result = concatenate_datasets(transformed_shards)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3267, in concatenate_datasets
    table = concat_tables(tables_to_concat, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 853, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 713, in from_tables
    blocks = to_blocks(tables[0])
IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.12.11-300.fc34.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.10
- PyArrow version: 3.0.0