mengta666/huggingface_github huggingface dataset:
Issue: 404 Error when loading remote data files from private repo

## Describe the bug
When loading remote data files from a private repo, a 404 error is raised.

## Steps to reproduce the bug
```python
url = hf_hub_url("lewtun/asr-preds-test", "preds.jsonl", repo_type="dataset")
dset = load_dataset("json", data_files=url, use_auth_token=True)
# HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/datasets/lewtun/asr-preds-test/resolve/main/preds.jsonl
```

## Expected results
Load dataset.

## Actual results
404 Error.



Comments:
I guess the issue is when computing the ETags of the remote files. Indeed `use_auth_token` must be passed to `request_etags` here:

https://github.com/huggingface/datasets/blob/35b5e4bc0cb2ed896e40f3eb2a4aa3de1cb1a6c5/src/datasets/builder.py#L160-L160

Yes, I remember having properly implemented that: 
- https://github.com/huggingface/datasets/commit/7a9c62f7cef9ecc293f629f859d4375a6bd26dc8#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R160
- https://github.com/huggingface/datasets/pull/2628/commits/6350a03b4b830339a745f7b1da46ece784ca734c

But a subsequent refactoring accidentally removed it...

I have opened a PR to fix it @lewtun.
Issue: Missing cache file

Strangely missing cache file after I  restart my program again.

`glue_dataset = datasets.load_dataset('glue', 'sst2')`

`FileNotFoundError: [Errno 2] No such file or directory: /Users/chris/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96d6053ad/dataset_info.json'`


Comments:
This could be solved by going to the glue/ directory and delete sst2 directory, then load the dataset again will help you redownload the dataset.

Hi ! Not sure why this file was missing, but yes the way to fix this is to delete the sst2 directory and to reload the dataset
Issue: Calling shuffle on IterableDataset will disable batching in case any functions were mapped

When using dataset in streaming mode, if one applies `shuffle` method on the dataset and `map` method for which `batched=True` than the batching operation will not happen, instead `batched` will be set to `False`

I did RCA on the dataset codebase, the problem is emerging from [this line of code](https://github.com/huggingface/datasets/blob/d25a0bf94d9f9a9aa6cabdf5b450b9c327d19729/src/datasets/iterable_dataset.py#L197) here as it is
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batch_size=self.batch_size`, as one can see it is missing batched argument, which means that the iterator fallsback to default constructor value, which in this case is `False`.
To remedy the problem we can change this line to
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batched=self.batched, batch_size=self.batch_size`


Comments:
Hi :) Good catch ! Feel free to open a PR if you want to contribute, this would be very welcome ;)

Have raised the PR [here](https://github.com/huggingface/datasets/pull/2717)

Fixed by #2717.
Issue: add more precise information for size

For the import into ELG, we would like a more precise description of the size of the dataset, instead of the current size categories. The size can be expressed in bytes, or any other preferred size unit. As suggested in the slack channel, perhaps this could be computed with a regex for existing datasets.

Comments:
We already have this information in the dataset_infos.json files of each dataset.
Maybe we can parse these files in the backend to return their content with the endpoint at huggingface.co/api/datasets

For now if you want to access this info you have to load the json for each dataset. For example:
- for a dataset on github like `squad` 
-   https://raw.githubusercontent.com/huggingface/datasets/master/datasets/squad/dataset_infos.json
- for a community dataset on the hub like `lhoestq/squad`:
  https://huggingface.co/datasets/lhoestq/squad/resolve/main/dataset_infos.json
Issue: Missing documentation for wnut_17 (ner_tags)

On the info page of the wnut_17 data set (https://huggingface.co/datasets/wnut_17), the model output of ner-tags is only documented for these 5 cases:

`ner_tags: a list of classification labels, with possible values including O (0), B-corporation (1), I-corporation (2), B-creative-work (3), I-creative-work (4).`

I trained a model with the data and it gives me 13 classes:

```
"id2label": {
    "0": 0,
    "1": 1,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9,
    "10": 10,
    "11": 11,
    "12": 12
  }

  "label2id": {
    "0": 0,
    "1": 1,
    "10": 10,
    "11": 11,
    "12": 12,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9
  }
```
The paper (https://www.aclweb.org/anthology/W17-4418.pdf) explains those 6 categories, but the ordering does not match:

```
1. person
2. location (including GPE, facility)
3. corporation
4. product (tangible goods, or well-defined
services)
5. creative-work (song, movie, book and
so on)
6. group (subsuming music band, sports team,
and non-corporate organisations)
```
I would be very helpful for me, if somebody could clarify the model ouputs and explain the "B-" and "I-" prefixes to me.

Really great work with that and the other packages, I couldn't believe that training the model with that data was basically a one-liner!

Comments:
Hi @maxpel, thanks for reporting this issue.

Indeed, the documentation in the dataset card is not complete. I’m opening a Pull Request to fix it.

As the paper explains, there are 6 entity types and we have ordered them alphabetically: `corporation`, `creative-work`, `group`, `location`, `person` and `product`. 

Each of these entity types has 2 possible IOB2 format tags: 
- `B-`: to indicate that the token is the beginning of an entity name, and the 
- `I-`: to indicate that the token is inside an entity name. 

Additionally, there is the standalone IOB2 tag 
- `O`: that indicates that the token belongs to no named entity. 

In total there are 13 possible tags, which correspond to the following integer numbers:

0. `O`
1. `B-corporation`
2. `I-corporation`
3. `B-creative-work`
4. `I-creative-work`
5. `B-group`
6. `I-group`
7. `B-location`
8. `I-location`
9. `B-person`
10. `I-person`
11. `B-product`
12. `I-product`
Issue: 404 Not Found Error when loading LAMA dataset

The [LAMA](https://huggingface.co/datasets/viewer/?dataset=lama) probing dataset is not available for download:  

Steps to Reproduce: 

1. `from datasets import load_dataset`
2. `dataset = load_dataset('lama', 'trex')`. 


Results:  
`FileNotFoundError: Couldn't find file locally at lama/lama.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/lama/lama.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/lama/lama.py`

Comments:
Hi @dwil2444! I was able to reproduce your error when I downgraded to v1.1.2. Updating to the latest version of Datasets fixed the error for me :)

Hi @dwil2444, thanks for reporting.

Could you please confirm which `datasets` version you were using and if the problem persists after you update it to the latest version: `pip install -U datasets`?

Thanks @stevhliu for the hint to fix this! ;)

@stevhliu @albertvillanova  updating to the latest version of datasets did in fact fix this issue. Thanks a lot for your help!
Issue: 404 not found error on loading WIKIANN dataset

## Describe the bug
Unable to retreive wikiann English dataset

## Steps to reproduce the bug
```python
from datasets import list_datasets, load_dataset, list_metrics, load_metric
WIKIANN = load_dataset("wikiann","en")
```

## Expected results
Colab notebook should display successful download status

## Actual results
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyArrow version: 3.0.0


Comments:
Hi @ronbutan, thanks for reporting.

You are right: we have recently found that the link to the original PAN-X dataset (also called WikiANN), hosted at Dropbox, is no longer working.

We have opened an issue in the GitHub repository of the original dataset (afshinrahimi/mmner#4) and we have also contacted the author by email to ask if they are planning to fix this issue. See the details here: https://github.com/huggingface/datasets/issues/2691#issuecomment-885463027

I close this issue because it is the same as in #2691. Feel free to subscribe to that other issue to be informed about any updates.
Issue: from datasets import Dataset is failing 

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import Dataset
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28 
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: latest version as of 07/21/2021
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


Comments:
Hi @kswamy15, thanks for reporting.

We are fixing this critical issue and making an urgent patch release of the `datasets` library today.

In the meantime, you can circumvent this issue by updating the `tqdm` library: `!pip install -U tqdm`
Issue: cannot combine splits merging and streaming?

this does not work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation',streaming=True)`
with error:
`ValueError: Bad split: train+validation. Available splits: ['train', 'validation']`

these work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation')`
`dataset = datasets.load_dataset('mc4','iw',split='train',streaming=True)`
`dataset = datasets.load_dataset('mc4','iw',split='validation',streaming=True)`

i could not find a reference to this in the documentation and the error message is confusing. also would be nice to allow streaming for the merged splits

Comments:
Hi ! That's missing indeed. We'll try to implement this for the next version :)

I guess we just need to implement #2564 first, and then we should be able to add support for splits combinations

is there an update on this? ran into the same issue on 2.17.1.

On a similar note, the keyword `split="all"` also does not work as intended when `streaming=True`. 

No update so far, especially since we haven't implemented an efficient way to query `split=train[50%:]` for example. The addition of two splits should be easy though, since we have `concatenate_datasets()`

Can you concatenate_datasets that are being streamed now? I was led to believe concatenation works on non streaming datasets only.

Yes `concatenate_datasets` works for datasets loaded in streaming mode as well
Issue: xtreme / pan-x cannot be downloaded

## Describe the bug

Dataset xtreme / pan-x cannot be loaded

Seems related to https://github.com/huggingface/datasets/pull/2326

## Steps to reproduce the bug

```python
dataset = load_dataset("xtreme", "PAN-X.fr")
```

## Expected results

Load the dataset

## Actual results

```
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1
```

## Environment info

- `datasets` version: 1.9.0
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Comments:
Hi @severo, thanks for reporting.

However I have not been able to reproduce this issue. Could you please confirm if the problem persists for you?

Maybe Dropbox (where the data source is hosted) was temporarily unavailable when you tried.

Hmmm, the file (https://www.dropbox.com/s/dl/12h3qqog6q4bjve/panx_dataset.tar) really seems to be unavailable... I tried from various connexions and machines and got the same 404 error. Maybe the dataset has been loaded from the cache in your case?

Yes @severo, weird... I could access the file when I answered to you, but now I cannot longer access it either... Maybe it was from the cache as you point out.

Anyway, I have opened an issue in the GitHub repository responsible for the original dataset: https://github.com/afshinrahimi/mmner/issues/4
I have also contacted the maintainer by email.

I'll keep you informed with their answer.

Reply from the author/maintainer: 
> Will fix the issue and let you know during the weekend.

The author told that apparently Dropbox has changed their policy and no longer allow downloading the file without having signed in first. The author asked Hugging Face to host their dataset.
Issue: cannot save the dataset to disk after rename_column

## Describe the bug
If you use `rename_column` and do no other modification, you will be unable to save the dataset using `save_to_disk`

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
In [1]: from datasets import Dataset, load_from_disk
In [5]: dataset=Dataset.from_dict({'foo': [0]})
In [7]: dataset.save_to_disk('foo')
In [8]: dataset=load_from_disk('foo')
In [10]: dataset=dataset.rename_column('foo', 'bar')
In [11]: dataset.save_to_disk('foo')
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-11-a3bc0d4fc339> in <module>
----> 1 dataset.save_to_disk('foo')

/mnt/beegfs/projects/meerqat/anaconda3/envs/meerqat/lib/python3.7/site-packages/datasets/arrow_dataset.py in save_to_disk(self, dataset_path
, fs)
    597             if Path(dataset_path, config.DATASET_ARROW_FILENAME) in cache_files_paths:
    598                 raise PermissionError(
--> 599                     f"Tried to overwrite {Path(dataset_path, config.DATASET_ARROW_FILENAME)} but a dataset can't overwrite itself."
    600                 )
    601             if Path(dataset_path, config.DATASET_INDICES_FILENAME) in cache_files_paths:

PermissionError: Tried to overwrite foo/dataset.arrow but a dataset can't overwrite itself.
```

N. B. I created the dataset from dict to enable easy reproduction but the same happens if you load an existing dataset (e.g. starting from `In [8]`)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.10
- PyArrow version: 3.0.0



Comments:
Hi ! That's because you are trying to overwrite a file that is already open and being used.
Indeed `foo/dataset.arrow` is open and used by your `dataset` object.

When you do `rename_column`, the resulting dataset reads the data from the same arrow file.
In other cases like when using `map` on the other hand, the resulting dataset reads the data from another arrow file that is the result of the map transform.

Therefore overwriting a dataset after `rename_column` is not possible, but it is possible after `map`, since `rename_column` doesn't switch to using another arrow file (the actual data stay the same).

Ok, thanks for clearing it up :)

so what would be the right way to read a dataset, then change something and then overwrite it with the new version?

You have to write to a new directory, then delete the former directory, and finally rename the new directory.

This worked for me; I know it's hacky but it works:

```
ds.add_column...
ds = Dataset.from_dict(ds.to_dict())
ds.save_to_disk...
```
Issue: hebrew language codes he and iw should be treated as aliases

https://huggingface.co/datasets/mc4 not listed when searching for hebrew datasets (he) as it uses the older language code iw, preventing discoverability. 

Comments:
Hi @eyaler, thanks for reporting.

While you are true with respect the Hebrew language tag ("iw" is deprecated and "he" is the preferred value), in the "mc4" dataset (which is a derived dataset) we have kept the language tags present in the original dataset: [Google C4](https://www.tensorflow.org/datasets/catalog/c4).

For discoverability on the website I updated the YAML tags at the top of the mC4 dataset card https://github.com/huggingface/datasets/commit/38288087b1b02f97586e0346e8f28f4960f1fd37

Once the website is updated, mC4 will be listed in https://huggingface.co/datasets?filter=languages:he
Issue: 5 duplicate datasets

## Describe the bug

In 5 cases, I could find a dataset on Paperswithcode which references two Hugging Face datasets as dataset loaders. They are:

- https://paperswithcode.com/dataset/multinli -> https://huggingface.co/datasets/multi_nli and https://huggingface.co/datasets/multi_nli_mismatch
  
  <img width="838" alt="Capture d’écran 2021-07-20 à 16 33 58" src="https://user-images.githubusercontent.com/1676121/126342757-4625522a-f788-41a3-bd1f-2a8b9817bbf5.png">

- https://paperswithcode.com/dataset/squad -> https://huggingface.co/datasets/squad and https://huggingface.co/datasets/squad_v2
- https://paperswithcode.com/dataset/narrativeqa -> https://huggingface.co/datasets/narrativeqa and https://huggingface.co/datasets/narrativeqa_manual
- https://paperswithcode.com/dataset/hate-speech-and-offensive-language -> https://huggingface.co/datasets/hate_offensive and https://huggingface.co/datasets/hate_speech_offensive
- https://paperswithcode.com/dataset/newsph-nli -> https://huggingface.co/datasets/newsph and https://huggingface.co/datasets/newsph_nli

Possible solutions:
- don't fix (it works)
- for each pair of duplicate datasets, remove one, and create an alias to the other.

## Steps to reproduce the bug

Visit the Paperswithcode links, and look at the "Dataset Loaders" section

## Expected results

There should only be one reference to a Hugging Face dataset loader

## Actual results

Two Hugging Face dataset loaders


Comments:
Yes this was documented in the PR that added this hf->paperswithcode mapping (https://github.com/huggingface/datasets/pull/2404) and AFAICT those are slightly distinct datasets so I think it's a wontfix

For context on the paperswithcode mapping you can also refer to https://github.com/huggingface/huggingface_hub/pull/43 which contains a lot of background discussion 

Thanks for the antecedents. I close.
Issue: Error when downloading C4

Hi,
I am trying to download `en` corpus from C4 dataset. However, I get an error caused by validation files download (see image). My code is very primitive:
`datasets.load_dataset('c4', 'en')`

Is this a bug or do I have some configurations missing on my server? 
Thanks!


<img width="1014" alt="Снимок экрана 2021-07-20 в 11 37 17" src="https://user-images.githubusercontent.com/36672861/126289448-6e0db402-5f3f-485a-bf74-eb6e0271fc25.png">

Comments:
Hi Thanks for reporting !
It looks like these files are not correctly reported in the list of expected files to download, let me fix that ;)

Alright this is fixed now. We'll do a new release soon to make the fix available.

In the meantime feel free to simply pass `ignore_verifications=True` to `load_dataset` to skip this error

@lhoestq thank you for such a quick feedback!
Issue: [`to_json`] add multi-proc sharding support

As discussed on slack it appears that `to_json` is quite slow on huge datasets like OSCAR.

I implemented sharded saving, which is much much faster - but the tqdm bars all overwrite each other, so it's hard to make sense of the progress, so if possible ideally this multi-proc support could be implemented internally in `to_json` via `num_proc` argument. I guess `num_proc` will be the number of shards?

I think the user will need to use this feature wisely, since too many processes writing to say normal style HD is likely to be slower than one process.

I'm not sure whether the user should be responsible to concatenate the shards at the end  or `datasets`, either way works for my needs.

The code I was using:

```
from multiprocessing import cpu_count, Process, Queue

[...]

filtered_dataset = concat_dataset.map(filter_short_documents, batched=True, batch_size=256, num_proc=cpu_count())

DATASET_NAME = "oscar"
SHARDS = 10
def process_shard(idx):
    print(f"Sharding {idx}")
    ds_shard = filtered_dataset.shard(SHARDS, idx, contiguous=True)
    # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
    print(f"Saving {DATASET_NAME}-{idx}.jsonl")
    ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

queue = Queue()
processes = [Process(target=process_shard, args=(idx,)) for idx in range(SHARDS)]
for p in processes:
    p.start()

for p in processes:
    p.join()
```

Thank you!

@lhoestq 

Comments:
Hi @stas00, 
I want to work on this issue and I was thinking why don't we use `imap` [in this loop](https://github.com/huggingface/datasets/blob/440b14d0dd428ae1b25881aa72ba7bbb8ad9ff84/src/datasets/io/json.py#L99)? This way, using offset (which is being used to slice the pyarrow table) we can convert  pyarrow table to `json` using multiprocessing. I've a small code snippet for some clarity:
```
result = list(
            pool.imap(self._apply_df, [(offset, batch_size) for offset in range(0, len(self.dataset), batch_size)])
        )
```
`_apply_df` is a function which will return `batch.to_pandas().to_json(path_or_buf=None, orient="records", lines=True)` which is basically json version of the batched pyarrow table. Later on we can concatenate it to form json file? 

I think the only downside here is to write file from `imap` output (output would be a list and we'll need to iterate over it and write in a file) which might add a little overhead cost. What do you think about this?

Followed up in https://github.com/huggingface/datasets/pull/2747
Issue: Allow the selection of multiple columns at once

**Is your feature request related to a problem? Please describe.**

Similar to pandas, it would be great if we could select multiple columns at once.


**Describe the solution you'd like**
```python
my_dataset = ...  # Has columns ['idx', 'sentence', 'label']
idx, label = my_dataset[['idx', 'label']]
```

**Describe alternatives you've considered**
we can do `[dataset[col] for col in ('idx', 'label')]`

**Additional context**
This is of course very minor.


Comments:
Hi! I was looking into this and hope you can clarify a point. Your my_dataset variable would be of type DatasetDict which means the alternative you've described (dict comprehension) is what makes sense. 
Is there a reason why you wouldn't want to convert my_dataset to a pandas df if you'd like to use it like one? Please let me know if I'm missing something.

Hi! Sorry for the delay.

In this case, the dataset would be a `datasets.Dataset` and we want to select multiple columns, the `idx` and `label` columns for example.

My issue is that my dataset is too big for memory if I load everything into pandas.

Hello - any update on this? Thank you.

Please note that you could use the method `Dataset.select_columns`: https://huggingface.co/docs/datasets/v2.15.0/en/package_reference/main_classes#datasets.Dataset.select_columns

Thank you!
Issue: Give a user feedback if the dataset he loads is streamable or not

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not. 

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive. 

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"


Comments:
#self-assign

I understand it already raises a `NotImplementedError` exception, eg:

```
>>> dataset = load_dataset("journalists_questions", name="plain_text", split="train", streaming=True)

[...]
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```
Issue: Add SD task for SUPERB

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun 


Comments:
Note that this subset requires us to:

* generate the LibriMix corpus from LibriSpeech
* prepare the corpus for diarization

As suggested by @lhoestq we should perform these steps locally and add the prepared data to this public repo on the Hub: https://huggingface.co/datasets/superb/superb-data

Then we can use the URLs for the files to load the data in `superb`'s dataset loading script.

For consistency, I suggest we name the folders in `superb-data` in the same way as the configs in the dataset loading script - e.g. use `sd` for speech diarization in both places :)

@lewtun @lhoestq: 

I have already generated the LibriMix corpus and prepared the corpus for diarization. The output is 3 dirs (train, dev, test), each one containing 6 files: reco2dur  rttm  segments  spk2utt  utt2spk  wav.scp

Next steps:
- Upload these files to the superb-data repo
- Transcribe the corresponding s3prl processing of these files into our superb loading script

Note that processing of these files is a bit more intricate than usual datasets: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/dataset.py#L233
Issue: Setting log level higher than warning does not suppress progress bar

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown 

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1


Comments:
Hi,

you can suppress progress bars by patching logging as follows:
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda: logging.NOTSET
# map call ...
```
EDIT: now you have to use `disable_progress_bar `

Thank you, it worked :)

See https://github.com/huggingface/datasets/issues/2528 for reference

Note also that you can disable the progress bar with

```python
from datasets.utils import disable_progress_bar
disable_progress_bar()
```

See https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/src/datasets/utils/tqdm_utils.py#L84

Now the library officially recommends `set_progress_bar_enabled(False)`

```py
from datasets.utils import set_progress_bar_enabled

set_progress_bar_enabled(False)
```

source:

https://github.com/huggingface/datasets/blob/1fd47120ace13626c528367787ffa13e1a26e6c0/src/datasets/utils/tqdm_utils.py#L83-L88



From https://github.com/huggingface/datasets/pull/3897, `disable_progress_bar` is the function you should use

Now ``disable_progress_bar`` function is in ``datasets/src/datasets/utils/logging.py``.
https://github.com/huggingface/datasets/blob/aa555a299ad73c65e3f997a764e9d211675ab05d/src/datasets/utils/logging.py#L233-L236
And the method mentioned in https://github.com/huggingface/datasets/issues/2651#issuecomment-880270774 is not working now.
Issue: [load_dataset] shard and parallelize the process

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq 

Comments:
I need the same feature for distributed training

I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :)

Posted a PR to address the building side, still needs something to load sharded arrow files + tests

Closing as this feature has been implemented in #5107
Issue: adding progress bar / ETA for `load_dataset`

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq 


Comments:
Is this done now? I see progress bars when using `load_dataset`.

There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
Issue: Add web_split dataset for Paraphase and Rephrase benchmark

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase



Comments:
#take
Issue: downloading of yahoo_answers_topics dataset failed

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.


## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files



Comments:
Hi ! I just tested and it worked fine today for me.

I think this is because the dataset is stored on Google Drive which has a quota limit for the number of downloads per day, see this similar issue https://github.com/huggingface/datasets/issues/996 

Feel free to try again today, now that the quota was reset

Fixed once data URL was replaced:
- #4023
Issue: load_dataset processing failed with OS error after downloading a dataset

## Describe the bug
After downloading a dataset like opus100, there is a bug that 
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS


## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0




Comments:
Hi ! It looks like an issue with pytorch.

Could you try to run `import torch` and see if it raises an error ?

> Hi ! It looks like an issue with pytorch.
> 
> Could you try to run `import torch` and see if it raises an error ?

It works. Thank you!
Issue: Support multi-worker with streaming dataset (IterableDataset).

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


Comments:
Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

Any update on this feature request?

Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
Issue: Progress bars are not properly rendered in Jupyter notebook

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf 

Comments:
To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Issue: Load datasets from the Hub without requiring a dataset script

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

Comments:
This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
Issue: can't set verbosity for `metric.py`

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places. 
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected 
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation. 

I had to do 
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
``` 
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS 
- Python version: 3.8.10
- PyArrow version: 3.0.0


Comments:
Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.
Issue: Integration with AugLy

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Comments:
Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

Preprocessing functions/augmentations, unless super generic, should be defined in separate libraries, so I'm closing this issue.
Issue: Jsonlines export error

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1

Comments:
Thanks for reporting @TevenLeScao! I'm having a look...

(not sure what just happened on the assignations sorry)

For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.

@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898

Thanks ! I'm creating a PR

Well I though it was me who has taken on this issue... 😅 

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄 

The fix is available on `master` @TevenLeScao , thanks for reporting
Issue: [Metrics] addition of wiki_split metrics

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this 
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py) 

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign

Comments:
#take
Issue: ModuleNotFoundError: No module named 'datasets.tasks' while importing common voice datasets

Error traceback:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-8-a7b592d3bca0> in <module>()
      1 from datasets import load_dataset, load_metric
      2 
----> 3 common_voice_train = load_dataset("common_voice", "pa-IN", split="train+validation")
      4 common_voice_test = load_dataset("common_voice", "pa-IN", split="test")

9 frames
/root/.cache/huggingface/modules/datasets_modules/datasets/common_voice/078d412587e9efeb0ae2e574da99c31e18844c496008d53dc5c60f4159ed639b/common_voice.py in <module>()
     19 
     20 import datasets
---> 21 from datasets.tasks import AutomaticSpeechRecognition
     22 
     23 

ModuleNotFoundError: No module named 'datasets.tasks'

Comments:
Hi @profsatwinder.

It looks like you are using an old version of `datasets`. Please update it with `pip install -U datasets` and indicate if the problem persists.

@albertvillanova Thanks for the information. I updated it to 1.9.0 and the issue is resolved. Thanks again.