helmo/github-issues huggingface dataset:
`to_json` reporting enhancements
State: open | Author: stas00 (CONTRIBUTOR)
Labels: name: enhancement

While using `to_json` 2 things came to mind that would have made the experience easier on the user:

1. Could we have a `desc` arg for the tqdm use and a fallback to just `to_json` so that it'd be clear to the user what's happening? Surely, one can just print the description before calling json, but I thought perhaps it'd help to have it self-identify like you did for other progress bars recently.

2. It took me a while to make sense of the reported numbers:
```
 22%|██▏       | 1536/7076 [12:30:57<44:09:42, 28.70s/it]
```
So iteration here happens to be 10K samples, and the total is 70M records. But the user does't know that, so the progress bar is perfect, but the numbers it reports are meaningless until one discovers that 1it=10K samples. And one still has to convert these in the head - so it's not quick. Not exactly sure what's the best way to approach this, perhaps it can be part of `desc`? or report M or K, so it'd be built-in if it were to print, e.g.:
```
 22%|██▏       | 15360K/70760K [12:30:57<44:09:42, 28.70s/it]
```
or 
```
 22%|██▏       | 15.36M/70.76M [12:30:57<44:09:42, 28.70s/it]
```
(while of course remaining friendly to small datasets)

I forget if tqdm lets you add a magnitude identifier to the running count.

Thank you!
Change `from_csv` default arguments
State: closed | Author: SBrandeis (CONTRIBUTOR)

Passing `sep=None` to pandas's `read_csv` lets pandas guess the CSV file's separator

This PR allows users to use this pandas's feature by passing `sep=None` to `Dataset.from_csv`:

```python
Dataset.from_csv(
    ...,
    sep=None
)
```

Comments: This is not the default in pandas right ?
We try to align our CSV loader with the pandas API.

Moreover according to their documentation, the python parser is used when sep is None, which might not be the fastest one.

Maybe users could just specify `sep=None` themselves ?
In this case we should add some documentation about this
Allow the selection of multiple columns at once
State: closed | Author: Dref360 (CONTRIBUTOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

Similar to pandas, it would be great if we could select multiple columns at once.


**Describe the solution you'd like**
```python
my_dataset = ...  # Has columns ['idx', 'sentence', 'label']
idx, label = my_dataset[['idx', 'label']]
```

**Describe alternatives you've considered**
we can do `[dataset[col] for col in ('idx', 'label')]`

**Additional context**
This is of course very minor.


Comments: Hi! I was looking into this and hope you can clarify a point. Your my_dataset variable would be of type DatasetDict which means the alternative you've described (dict comprehension) is what makes sense. 
Is there a reason why you wouldn't want to convert my_dataset to a pandas df if you'd like to use it like one? Please let me know if I'm missing something.

Hi! Sorry for the delay.

In this case, the dataset would be a `datasets.Dataset` and we want to select multiple columns, the `idx` and `label` columns for example.

My issue is that my dataset is too big for memory if I load everything into pandas.

Hello - any update on this? Thank you.

Please note that you could use the method `Dataset.select_columns`: https://huggingface.co/docs/datasets/v2.15.0/en/package_reference/main_classes#datasets.Dataset.select_columns

Thank you!
Give a user feedback if the dataset he loads is streamable or not
State: open | Author: philschmid (CONTRIBUTOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not. 

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive. 

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"


Comments: #self-assign

I understand it already raises a `NotImplementedError` exception, eg:

```
>>> dataset = load_dataset("journalists_questions", name="plain_text", split="train", streaming=True)

[...]
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```
Add SD task for SUPERB
State: closed | Author: albertvillanova (MEMBER)
Labels: name: dataset request

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun 


Comments: Note that this subset requires us to:

* generate the LibriMix corpus from LibriSpeech
* prepare the corpus for diarization

As suggested by @lhoestq we should perform these steps locally and add the prepared data to this public repo on the Hub: https://huggingface.co/datasets/superb/superb-data

Then we can use the URLs for the files to load the data in `superb`'s dataset loading script.

For consistency, I suggest we name the folders in `superb-data` in the same way as the configs in the dataset loading script - e.g. use `sd` for speech diarization in both places :)

@lewtun @lhoestq: 

I have already generated the LibriMix corpus and prepared the corpus for diarization. The output is 3 dirs (train, dev, test), each one containing 6 files: reco2dur  rttm  segments  spk2utt  utt2spk  wav.scp

Next steps:
- Upload these files to the superb-data repo
- Transcribe the corresponding s3prl processing of these files into our superb loading script

Note that processing of these files is a bit more intricate than usual datasets: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/dataset.py#L233
Setting log level higher than warning does not suppress progress bar
State: closed | Author: Isa-rentacs (NONE)
Labels: name: bug

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown 

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1


Comments: Hi,

you can suppress progress bars by patching logging as follows:
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda: logging.NOTSET
# map call ...
```
EDIT: now you have to use `disable_progress_bar `

Thank you, it worked :)

See https://github.com/huggingface/datasets/issues/2528 for reference

Note also that you can disable the progress bar with

```python
from datasets.utils import disable_progress_bar
disable_progress_bar()
```

See https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/src/datasets/utils/tqdm_utils.py#L84

Now the library officially recommends `set_progress_bar_enabled(False)`

```py
from datasets.utils import set_progress_bar_enabled

set_progress_bar_enabled(False)
```

source:

https://github.com/huggingface/datasets/blob/1fd47120ace13626c528367787ffa13e1a26e6c0/src/datasets/utils/tqdm_utils.py#L83-L88



From https://github.com/huggingface/datasets/pull/3897, `disable_progress_bar` is the function you should use

Now ``disable_progress_bar`` function is in ``datasets/src/datasets/utils/logging.py``.
https://github.com/huggingface/datasets/blob/aa555a299ad73c65e3f997a764e9d211675ab05d/src/datasets/utils/logging.py#L233-L236
And the method mentioned in https://github.com/huggingface/datasets/issues/2651#issuecomment-880270774 is not working now.
[load_dataset] shard and parallelize the process
State: closed | Author: stas00 (CONTRIBUTOR)
Labels: name: enhancement

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq 

Comments: I need the same feature for distributed training

I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :)

Posted a PR to address the building side, still needs something to load sharded arrow files + tests

Closing as this feature has been implemented in #5107
adding progress bar / ETA for `load_dataset`
State: open | Author: stas00 (CONTRIBUTOR)
Labels: name: enhancement

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq 


Comments: Is this done now? I see progress bars when using `load_dataset`.

There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
Add web_split dataset for Paraphase and Rephrase benchmark
State: open | Author: bhadreshpsavani (CONTRIBUTOR)
Labels: name: enhancement

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase



Comments: #take
downloading of yahoo_answers_topics dataset failed
State: closed | Author: vikrant7k (NONE)
Labels: name: bug

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.


## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files



Comments: Hi ! I just tested and it worked fine today for me.

I think this is because the dataset is stored on Google Drive which has a quota limit for the number of downloads per day, see this similar issue https://github.com/huggingface/datasets/issues/996 

Feel free to try again today, now that the quota was reset

Fixed once data URL was replaced:
- #4023
load_dataset processing failed with OS error after downloading a dataset
State: closed | Author: fake-warrior8 (NONE)
Labels: name: bug

## Describe the bug
After downloading a dataset like opus100, there is a bug that 
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS


## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0




Comments: Hi ! It looks like an issue with pytorch.

Could you try to run `import torch` and see if it raises an error ?

> Hi ! It looks like an issue with pytorch.
> 
> Could you try to run `import torch` and see if it raises an error ?

It works. Thank you!
Support multi-worker with streaming dataset (IterableDataset).
State: open | Author: changjonathanc (CONTRIBUTOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


Comments: Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

Any update on this feature request?

Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
Streaming for the Json loader
State: closed | Author: lhoestq (MEMBER)

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.

Comments: A note is that I think we should add a few indicator of status (as mentioned by @stas00 in #2649), probably at the (1) downloading, (2) extracting and (3) reading steps. In particular when loading many very large files it's interesting to know a bit where we are in the process.

I tested locally, and the builtin `json` loader is 4x slower than `pyarrow.json`. Thanks for the comment @albertvillanova !

Therefore I switched back to using `pyarrow.json`, but only on the batch that is read. This way we don't have to deal with its `block_size`, and it only loads in memory one batch at a time.
add image-classification task template
State: closed | Author: nateraw (CONTRIBUTOR)

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```

Comments: Awesome!

Thanks for adding a new task template - great work @nateraw  🚀 !
Progress bars are not properly rendered in Jupyter notebook
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf 

Comments: To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Load datasets from the Hub without requiring a dataset script
State: closed | Author: lhoestq (MEMBER)

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

Comments: This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
can't set verbosity for `metric.py`
State: closed | Author: thomas-happify (NONE)
Labels: name: bug

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places. 
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected 
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation. 

I had to do 
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
``` 
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS 
- Python version: 3.8.10
- PyArrow version: 3.0.0


Comments: Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.