404 Error when loading remote data files from private repo

Something isn't working

I guess the issue is when computing the ETags of the remote files. Indeed `use_auth_token` must be passed to `request_etags` here:

https://github.com/huggingface/datasets/blob/35b5e4bc0cb2ed896e40f3eb2a4aa3de1cb1a6c5/src/datasets/builder.py#L160-L160

Yes, I remember having properly implemented that:
- https://github.com/huggingface/datasets/commit/7a9c62f7cef9ecc293f629f859d4375a6bd26dc8#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R160
- https://github.com/huggingface/datasets/pull/2628/commits/6350a03b4b830339a745f7b1da46ece784ca734c

But a subsequent refactoring accidentally removed it...

## Describe the bug
When loading remote data files from a private repo, a 404 error is raised.

## Steps to reproduce the bug
```python
url = hf_hub_url("lewtun/asr-preds-test", "preds.jsonl", repo_type="dataset")
dset = load_dataset("json", data_files=url, use_auth_token=True)
# HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/datasets/lewtun/asr-preds-test/resolve/main/preds.jsonl
```

## Expected results
Load dataset.

## Actual results
404 Error.


Fix en subset by modifying dataset_info with correct validation infos

- Related to: #2682

We correct the values of `en` subset concerning the expected validation values (both `num_bytes` and `num_examples`.

Instead of having:

`{"name": "validation", "num_bytes": 828589180707, "num_examples": 364868892, "dataset_name": "c4"}`

We replace with correct values:

`{"name": "validation", "num_bytes": 825767266, "num_examples": 364608, "dataset_name": "c4"}`

There are still issues with validation with other subsets, but I can't download all the files, unzip to check for the correct number of bytes. (If you have a fast way to obtain those values for other subsets, I can do this in this PR ... otherwise I can't spend those resources)


Missing cache file

bug

Something isn't working

This could be solved by going to the glue/ directory and delete sst2 directory, then load the dataset again will help you redownload the dataset.

Hi ! Not sure why this file was missing, but yes the way to fix this is to delete the sst2 directory and to reload the dataset

Strangely missing cache file after I  restart my program again.

`glue_dataset = datasets.load_dataset('glue', 'sst2')`

`FileNotFoundError: [Errno 2] No such file or directory: /Users/chris/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96d6053ad/dataset_info.json'`


Deal with the bad check in test_load.py

Hi ! I did a change for this test already in #2662 :

https://github.com/huggingface/datasets/blob/00686c46b7aaf6bfcd4102cec300a3c031284a5a/tests/test_load.py#L312-L316

(though I have to change the variable name `m_combined_path` to `m_url` or something)

I guess it's ok to remove this check for now :)

This PR removes a check that's been added in #2684. My intention with this check was to capture an URL in the error message, but instead, it captures a substring of the previous regex match in the test function. Another option would be to replace this check with:
```python
m_paths = re.findall(r"\S*_dummy/_dummy.py\b", str(exc_info.value))  # on Linux this will match an URL as well as a local_path due to different os.sep, so take the last element (an URL always comes last in the list)
assert len(m_paths) > 0 and is_remote_url(m_paths[-1])  # is_remote_url comes from datasets.utils.file_utils
```

@lhoestq Let me know which one of these two approaches (delete or replace) do you prefer?


fix: 🐛 fix two typos


Use ETag in streaming mode to detect resource updates

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

I want to cache data I generate from processing a dataset I've loaded in streaming mode, but I've currently no way to know if the remote data has been updated or not, thus I don't know when to invalidate my cache.

**Describe the solution you'd like**

Take the ETag of the data files into account and provide it (directly or through a hash) to give a signal that I can invalidate my cache.

**Describe alternatives you've considered**

None


Docs structure

Organize Datasets documentation into four documentation types to improve clarity and discoverability of content.

- [x] Tutorials
- [x] How-to guides
- [ ] Conceptual guides (mostly done, need Quentin's help with a few sections)
- [x] Reference
- [x] Review


Fix shuffle on IterableDataset that disables batching in case any functions were mapped

Made a very minor change to fix the issue#2716. Added the missing argument in the constructor call.

As discussed in the bug report, the change is made to prevent the `shuffle` method call from resetting the value of `batched` attribute in `MappedExamplesIterable`

Fix #2716.


Calling shuffle on IterableDataset will disable batching in case any functions were mapped

bug

Something isn't working

Hi :) Good catch ! Feel free to open a PR if you want to contribute, this would be very welcome ;)

Have raised the PR [here](https://github.com/huggingface/datasets/pull/2717)

When using dataset in streaming mode, if one applies `shuffle` method on the dataset and `map` method for which `batched=True` than the batching operation will not happen, instead `batched` will be set to `False`

I did RCA on the dataset codebase, the problem is emerging from [this line of code](https://github.com/huggingface/datasets/blob/d25a0bf94d9f9a9aa6cabdf5b450b9c327d19729/src/datasets/iterable_dataset.py#L197) here as it is
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batch_size=self.batch_size`, as one can see it is missing batched argument, which means that the iterator fallsback to default constructor value, which in this case is `False`.
To remedy the problem we can change this line to
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batched=self.batched, batch_size=self.batch_size`


Update PAN-X data URL in XTREME dataset

Merging since the CI is just about missing infos in the dataset card

Related to #2710, #2691.


add more precise information for size

enhancement

New feature or request

We already have this information in the dataset_infos.json files of each dataset.
Maybe we can parse these files in the backend to return their content with the endpoint at huggingface.co/api/datasets

For now if you want to access this info you have to load the json for each dataset. For example:
- for a dataset on github like `squad`
-   https://raw.githubusercontent.com/huggingface/datasets/master/datasets/squad/dataset_infos.json
- for a community dataset on the hub like `lhoestq/squad`:
  https://huggingface.co/datasets/lhoestq/squad/resolve/main/dataset_infos.json

For the import into ELG, we would like a more precise description of the size of the dataset, instead of the current size categories. The size can be expressed in bytes, or any other preferred size unit. As suggested in the slack channel, perhaps this could be computed with a regex for existing datasets.


Enumerate all ner_tags values in WNUT 17 dataset

This PR does:
- Enumerate all ner_tags in dataset card Data Fields section
- Add all metadata tags to dataset card

Close #2709.


Update WikiANN data URL

We have to update the URL in the XTREME benchmark as well:

https://github.com/huggingface/datasets/blob/0dfc639cec450ed8762a997789a2ed63e63cdcf2/datasets/xtreme/xtreme.py#L411-L411

WikiANN data source URL is no longer accessible: 404 error from Dropbox.

We have decided to host it at Hugging Face. This PR updates the data source URL, the metadata JSON file and the dataset card.

Close #2691.


Missing documentation for wnut_17 (ner_tags)

bug

Something isn't working

Hi @maxpel, thanks for reporting this issue.

Indeed, the documentation in the dataset card is not complete. I’m opening a Pull Request to fix it.

As the paper explains, there are 6 entity types and we have ordered them alphabetically: `corporation`, `creative-work`, `group`, `location`, `person` and `product`.

Each of these entity types has 2 possible IOB2 format tags:
- `B-`: to indicate that the token is the beginning of an entity name, and the
- `I-`: to indicate that the token is inside an entity name.

Additionally, there is the standalone IOB2 tag
- `O`: that indicates that the token belongs to no named entity.

In total there are 13 possible tags, which correspond to the following integer numbers:

0. `O`
1. `B-corporation`
2. `I-corporation`
3. `B-creative-work`
4. `I-creative-work`
5. `B-group`
6. `I-group`
7. `B-location`
8. `I-location`
9. `B-person`
10. `I-person`
11. `B-product`
12. `I-product`

On the info page of the wnut_17 data set (https://huggingface.co/datasets/wnut_17), the model output of ner-tags is only documented for these 5 cases:

`ner_tags: a list of classification labels, with possible values including O (0), B-corporation (1), I-corporation (2), B-creative-work (3), I-creative-work (4).`

I trained a model with the data and it gives me 13 classes:

```
"id2label": {
    "0": 0,
    "1": 1,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9,
    "10": 10,
    "11": 11,
    "12": 12
  }

  "label2id": {
    "0": 0,
    "1": 1,
    "10": 10,
    "11": 11,
    "12": 12,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9
  }
```
The paper (https://www.aclweb.org/anthology/W17-4418.pdf) explains those 6 categories, but the ordering does not match:

```
1. person
2. location (including GPE, facility)
3. corporation
4. product (tangible goods, or well-defined
services)
5. creative-work (song, movie, book and
so on)
6. group (subsuming music band, sports team,
and non-corporate organisations)
```
I would be very helpful for me, if somebody could clarify the model ouputs and explain the "B-" and "I-" prefixes to me.

Really great work with that and the other packages, I couldn't believe that training the model with that data was basically a one-liner!


QASC: incomplete training set

Something isn't working

Hi @danyaljj, thanks for reporting.

Unfortunately, I have not been able to reproduce your problem. My train split has 8134 examples:
```ipython
In [10]: ds["train"]
Out[10]:
Dataset({
    features: ['id', 'question', 'choices', 'answerKey', 'fact1', 'fact2', 'combinedfact', 'formatted_question'],
    num_rows: 8134
})

In [11]: ds["train"].shape
Out[11]: (8134, 8)
```
and the content of the last 5 examples is:
```ipython
In [12]: for i in range(8129, 8134):
    ...:     print(json.dumps(ds["train"][i]))
    ...:
{"id": "3KAKFY4PGU1LGXM77JAK2700NGCI3X", "question": "Chitin can be used for protection by whom?", "choices": {"text": ["Fungi", "People", "Man", "Fish", "trees", "Dogs", "animal", "Birds"], "label": ["A", "B",
 "C", "D", "E", "F", "G", "H"]}, "answerKey": "D", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish scales are also composed of chitin.", "combinedfact": "Chitin can be used for prote
ction by fish.", "formatted_question": "Chitin can be used for protection by whom? (A) Fungi (B) People (C) Man (D) Fish (E) trees (F) Dogs (G) animal (H) Birds"}
{"id": "336YQZE83VDAQVZ26HW59X51JZ9M5M", "question": "Which type of animal uses plates for protection?", "choices": {"text": ["squids", "reptiles", "sea urchins", "fish", "amphibians", "Frogs", "mammals", "salm
on"], "label": ["A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "B", "fact1": "scales are used for protection by scaled animals", "fact2": "Reptiles have scales or plates.", "combinedfact": "Reptiles use
 their plates for protection.", "formatted_question": "Which type of animal uses plates for protection? (A) squids (B) reptiles (C) sea urchins (D) fish (E) amphibians (F) Frogs (G) mammals (H) salmon"}
{"id": "3WZ36BJEV3FGS66VGOOUYX0LN8GTBU", "question": "What are used for protection by fish?", "choices": {"text": ["scales", "fins", "streams.", "coral", "gills", "Collagen", "mussels", "whiskers"], "label": ["
A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "A", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish are backboned aquatic animals.", "combinedfact": "scales are used for prote
ction by fish ", "formatted_question": "What are used for protection by fish? (A) scales (B) fins (C) streams. (D) coral (E) gills (F) Collagen (G) mussels (H) whiskers"}
{"id": "3Z2R0DQ0JHDKFAO2706OYIXGNA4E28", "question": "What are pangolins covered in?", "choices": {"text": ["tunicates", "Echinoids", "shells", "exoskeleton", "blastoids", "barrel-shaped", "protection", "white"
], "label": ["A", "B", "C", "D", "E", "F", "G", "H"]}, "answerKey": "G", "fact1": "scales are used for protection by scaled animals", "fact2": "Pangolins have an elongate and tapering body covered above with ov
erlapping scales.", "combinedfact": "Pangolins are covered in overlapping protection.", "formatted_question": "What are pangolins covered in? (A) tunicates (B) Echinoids (C) shells (D) exoskeleton (E) blastoids
 (F) barrel-shaped (G) protection (H) white"}
{"id": "3PMBY0YE272GIWPNWIF8IH5RBHVC9S", "question": "What are covered with protection?", "choices": {"text": ["apples", "trees", "coral", "clams", "roses", "wings", "hats", "fish"], "label": ["A", "B", "C", "D
", "E", "F", "G", "H"]}, "answerKey": "H", "fact1": "scales are used for protection by scaled animals", "fact2": "Fish are covered with scales.", "combinedfact": "Fish are covered with protection", "formatted_q
uestion": "What are covered with protection? (A) apples (B) trees (C) coral (D) clams (E) roses (F) wings (G) hats (H) fish"}
```

Could you please load again your dataset and print its shape, like this:
```python
ds = load_dataset("qasc", split="train)
print(ds.shape)
```
and confirm which is your output?

Hmm .... it must have been a mistake on my side. Sorry for the hassle!

## Describe the bug
The training instances are not loaded properly.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("qasc", script_version='1.10.2')

def load_instances(split):
    instances = dataset[split]
    print(f"split: {split} - size: {len(instances)}")
    for x in instances:
        print(json.dumps(x))

load_instances('test')
load_instances('validation')
load_instances('train')
```

##  results
For test and validation, we can see the examples in the output (which is good!):
```
split: test - size: 920
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Anthax", "under water", "uterus", "wombs", "two", "moles", "live", "embryo"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What type of birth do therian mammals have? (A) Anthax (B) under water (C) uterus (D) wombs (E) two (F) moles (G) live (H) embryo", "id": "3C44YUNSI1OBFBB8D36GODNOZN9DPA", "question": "What type of birth do therian mammals have?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Corvidae", "arthropods", "birds", "backbones", "keratin", "Jurassic", "front paws", "Parakeets."]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "By what time had mouse-sized viviparous mammals evolved? (A) Corvidae (B) arthropods (C) birds (D) backbones (E) keratin (F) Jurassic (G) front paws (H) Parakeets.", "id": "3B1NLC6UGZVERVLZFT7OUYQLD1SGPZ", "question": "By what time had mouse-sized viviparous mammals evolved?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Reduced friction", "causes infection", "vital to a good life", "prevents water loss", "camouflage from consumers", "Protection against predators", "spur the growth of the plant", "a smooth surface"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What does a plant's skin do? (A) Reduced friction (B) causes infection (C) vital to a good life (D) prevents water loss (E) camouflage from consumers (F) Protection against predators (G) spur the growth of the plant (H) a smooth surface", "id": "3QRYMNZ7FYGITFVSJET3PS0F4S0NT9", "question": "What does a plant's skin do?"}
...
```
However, only a few instances are loaded for the training split, which is not correct.

## Environment info
- `datasets` version: '1.10.2'
- Platform: MaxOS
- Python version:3.7
- PyArrow version: 3.0.0


404 Not Found Error when loading LAMA dataset

Hi @dwil2444! I was able to reproduce your error when I downgraded to v1.1.2. Updating to the latest version of Datasets fixed the error for me :)

Hi @dwil2444, thanks for reporting.

Could you please confirm which `datasets` version you were using and if the problem persists after you update it to the latest version: `pip install -U datasets`?

Thanks @stevhliu for the hint to fix this! ;)

@stevhliu @albertvillanova  updating to the latest version of datasets did in fact fix this issue. Thanks a lot for your help!

The [LAMA](https://huggingface.co/datasets/viewer/?dataset=lama) probing dataset is not available for download:

Steps to Reproduce:

1. `from datasets import load_dataset`
2. `dataset = load_dataset('lama', 'trex')`.

Results:
`FileNotFoundError: Couldn't find file locally at lama/lama.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/lama/lama.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/lama/lama.py`


Update BibTeX entry

Update BibTeX entry.


404 not found error on loading WIKIANN dataset

Something isn't working

Hi @ronbutan, thanks for reporting.

You are right: we have recently found that the link to the original PAN-X dataset (also called WikiANN), hosted at Dropbox, is no longer working.

We have opened an issue in the GitHub repository of the original dataset (afshinrahimi/mmner#4) and we have also contacted the author by email to ask if they are planning to fix this issue. See the details here: https://github.com/huggingface/datasets/issues/2691#issuecomment-885463027

I close this issue because it is the same as in #2691. Feel free to subscribe to that other issue to be informed about any updates.

## Describe the bug
Unable to retreive wikiann English dataset

## Steps to reproduce the bug
```python
from datasets import list_datasets, load_dataset, list_metrics, load_metric
WIKIANN = load_dataset("wikiann","en")
```

## Expected results
Colab notebook should display successful download status

## Actual results
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyArrow version: 3.0.0


Fix pick default config name message

The error message to tell which config name to load is not displayed.

This is because in the code it was considering the config kwargs to be non-empty, which is a special case for custom configs created on the fly. It appears after this change: https://github.com/huggingface/datasets/pull/2659

I fixed that by making the config kwargs empty by default, even if default parameters are passed

Fix https://github.com/huggingface/datasets/issues/2703


Bad message when config name is missing

When loading a dataset that have several configurations, we expect to see an error message if the user doesn't specify a config name.

However in `datasets` 1.10.0 and 1.10.1 it doesn't show the right message:

```python
import datasets

datasets.load_dataset("glue")
```
raises
```python
AttributeError: 'BuilderConfig' object has no attribute 'text_features'
```
instead of
```python
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
        `load_dataset('glue', 'cola')`
```


Fix download_mode docstrings

documentation

Improvements or additions to documentation

Fix `download_mode` docstrings.


from datasets import Dataset is failing

bug

Something isn't working

Hi @kswamy15, thanks for reporting.

We are fixing this critical issue and making an urgent patch release of the `datasets` library today.

In the meantime, you can circumvent this issue by updating the `tqdm` library: `!pip install -U tqdm`

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import Dataset
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: latest version as of 07/21/2021
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


cannot combine splits merging and streaming?

Something isn't working

Hi ! That's missing indeed. We'll try to implement this for the next version :)

I guess we just need to implement #2564 first, and then we should be able to add support for splits combinations

this does not work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation',streaming=True)`
with error:
`ValueError: Bad split: train+validation. Available splits: ['train', 'validation']`

these work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation')`
`dataset = datasets.load_dataset('mc4','iw',split='train',streaming=True)`
`dataset = datasets.load_dataset('mc4','iw',split='validation',streaming=True)`

i could not find a reference to this in the documentation and the error message is confusing. also would be nice to allow streaming for the merged splits


Ignore empty batch when writing

This prevents an schema update with unknown column types, as reported in #2644.

This is my first attempt at fixing the issue. I tested the following:
- First batch returned by a batched map operation is empty.
- An intermediate batch is empty.
- `python -m unittest tests.test_arrow_writer` passes.

However, `arrow_writer` looks like a pretty generic interface, I'm not sure if there are other uses I may have overlooked. Let me know if that's the case, or if a better approach would be preferable.


Fix import on Colab

@lhoestq @albertvillanova - It might be a good idea to have a patch release after this gets merged (presumably tomorrow morning when you're around). The Colab issue linked to this PR is a pretty big blocker.

Fix #2695, fix #2700.


Add support for disable_progress_bar on Windows

The CI failure seems unrelated to this PR (probably has something to do with Transformers).

This PR is a continuation of #2667 and adds support for `utils.disable_progress_bar()` on Windows when using multiprocessing. This [answer](https://stackoverflow.com/a/6596695/14095927) on SO explains it nicely why the current approach (with calling `utils.is_progress_bar_enabled()` inside `Dataset._map_single`) would not work on Windows.


Cannot import load_dataset on Colab

Something isn't working

I'm facing the same issue on Colab today too.

```
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-4-5833ac0f5437> in <module>()
      3
      4 from ray import tune
----> 5 from datasets import DatasetDict, Dataset
      6 from datasets import load_dataset, load_metric
      7 from dataclasses import dataclass

7 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
```

@phosseini
I think it is related to [1.10.0](https://github.com/huggingface/datasets/actions/runs/1052653701) release done 3 hours ago. (cc: @lhoestq )
For now I just downgraded to 1.9.0 and it is working fine.

> @phosseini
> I think it is related to [1.10.0](https://github.com/huggingface/datasets/actions/runs/1052653701) release done 3 hours ago. (cc: @lhoestq )
> For now I just downgraded to 1.9.0 and it is working fine.

Same here, downgraded to 1.9.0 for now and works fine.

Hi,

updating tqdm to the newest version resolves the issue for me. You can do this as follows in Colab:
```
!pip install tqdm --upgrade
```

Hi @bayartsogt-ya and @phosseini, thanks for reporting.

We are fixing this critical issue and making an urgent patch release of the `datasets` library today.

In the meantime, as pointed out by @mariosasko, you can circumvent this issue by updating the `tqdm` library:
```
!pip install -U tqdm
```

## Describe the bug
Got tqdm concurrent module not found error during importing load_dataset from datasets.

## Steps to reproduce the bug
Here [colab notebook](https://colab.research.google.com/drive/1pErWWnVP4P4mVHjSFUtkePd8Na_Qirg4?usp=sharing) to reproduce the error

On colab:
```python
!pip install datasets
from datasets import load_dataset
```

## Expected results
Works without error

## Actual results
Specify the actual results or traceback.
```
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-8cc7de4c69eb> in <module>()
----> 1 from datasets import load_dataset, load_metric, Metric, MetricInfo, Features, Value
      2 from sklearn.metrics import mean_squared_error

/usr/local/lib/python3.7/dist-packages/datasets/__init__.py in <module>()
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in <module>()
     40 from tqdm.auto import tqdm
     41
---> 42 from datasets.tasks.text_classification import TextClassification
     43
     44 from . import config, utils

/usr/local/lib/python3.7/dist-packages/datasets/tasks/__init__.py in <module>()
      1 from typing import Optional
      2
----> 3 from ..utils.logging import get_logger
      4 from .automatic_speech_recognition import AutomaticSpeechRecognition
      5 from .base import TaskTemplate

/usr/local/lib/python3.7/dist-packages/datasets/utils/__init__.py in <module>()
     19
     20 from . import logging
---> 21 from .download_manager import DownloadManager, GenerateMode
     22 from .file_utils import DownloadConfig, cached_path, hf_bucket_url, is_remote_url, temp_seed
     23 from .mock_download_manager import MockDownloadManager

/usr/local/lib/python3.7/dist-packages/datasets/utils/download_manager.py in <module>()
     24
     25 from .. import config
---> 26 from .file_utils import (
     27     DownloadConfig,
     28     cached_path,

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.0
- Platform: Colab
- Python version: 3.7.11
- PyArrow version: 3.0.0


fix: 🐛 change string format to allow copy/paste to work in bash

Before: copy/paste resulted in an error because the square bracket
characters `[]` are special characters in bash


Fix OSCAR Esperanto

The Esperanto part (original) of OSCAR has the wrong number of examples:
```python
from datasets import load_dataset
raw_datasets = load_dataset("oscar", "unshuffled_original_eo")
```
raises
```python
NonMatchingSplitsSizesError:
[{'expected': SplitInfo(name='train', num_bytes=314188336, num_examples=121171, dataset_name='oscar'),
'recorded': SplitInfo(name='train', num_bytes=314064514, num_examples=121168, dataset_name='oscar')}]
```

I updated the number of expected examples in dataset_infos.json

cc @sgugger


Update BibTeX entry

Update BibTeX entry


xtreme / pan-x cannot be downloaded

Something isn't working

Hi @severo, thanks for reporting.

However I have not been able to reproduce this issue. Could you please confirm if the problem persists for you?

Maybe Dropbox (where the data source is hosted) was temporarily unavailable when you tried.

Hmmm, the file (https://www.dropbox.com/s/dl/12h3qqog6q4bjve/panx_dataset.tar) really seems to be unavailable... I tried from various connexions and machines and got the same 404 error. Maybe the dataset has been loaded from the cache in your case?

Yes @severo, weird... I could access the file when I answered to you, but now I cannot longer access it either... Maybe it was from the cache as you point out.

Anyway, I have opened an issue in the GitHub repository responsible for the original dataset: https://github.com/afshinrahimi/mmner/issues/4
I have also contacted the maintainer by email.

I'll keep you informed with their answer.

Reply from the author/maintainer:
> Will fix the issue and let you know during the weekend.

The author told that apparently Dropbox has changed their policy and no longer allow downloading the file without having signed in first. The author asked Hugging Face to host their dataset.

## Describe the bug

Dataset xtreme / pan-x cannot be loaded

Seems related to https://github.com/huggingface/datasets/pull/2326

## Steps to reproduce the bug

```python
dataset = load_dataset("xtreme", "PAN-X.fr")
```

## Expected results

Load the dataset

## Actual results

```
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1
```

## Environment info

- `datasets` version: 1.9.0
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Docs details

Thanks for all the comments and for the corrections in the docs !

About all the points you mentioned:

> * the code samples assume the expected libraries have already been installed. Maybe add a section at start, or add it to every code sample. Something like `pip install datasets transformers torch  'datasets[streaming]'` (maybe just link to https://huggingface.co/docs/datasets/installation.html + a one-liner that installs all the requirements / alternatively a requirements.txt file)

Yes good idea

> * "If you’d like to play with the examples, you must install it from source." in https://huggingface.co/docs/datasets/installation.html: it's not clear to me what this means (what are these "examples"?)

It refers to examples scripts inside the git repository of the library, see the `examples` folder in the `transformers` repo.
We don't have examples yet in the git repo of `datasets` as in transformers. So currently there are no examples. Maybe we can just remove this sentence from the docs for now

> * in https://huggingface.co/docs/datasets/loading_datasets.html: "or AWS bucket if it’s not already stored in the library". It's the only place in the doc (aside from the docstring https://huggingface.co/docs/datasets/package_reference/loading_methods.html?highlight=aws bucket#datasets.list_datasets) where the "AWS bucket" is mentioned. It's not easy to understand what this means. Maybe explain more, and link to https://s3.amazonaws.com/datasets.huggingface.co and/or https://huggingface.co/docs/datasets/filesystems.html.

This is outdated and must be replaced by
```
or from the Hugging Face Hub if it’s not already stored in the library
```

> * example in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files is obsoleted by [Enable auto-download for PAN-X / Wikiann domain in XTREME #2326](https://github.com/huggingface/datasets/pull/2326). Also: see [xtreme / pan-x cannot be downloaded #2691](https://github.com/huggingface/datasets/issues/2691) for a bug on this specific dataset.

We can replace the `XTREME` `PANX` dataste by `matinf` instead for example

> * in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files the doc says "After you’ve downloaded the files, you can point to the folder hosting them locally with the data_dir argument as follows:", but the following example does not show how to use `data_dir`

Let's add `data_dir="path/to/your/downloaded/data"` for example

> * in https://huggingface.co/docs/datasets/loading_datasets.html#csv-files, it would be nice to have an URL to the csv loader reference (but I'm not sure there is one in the API reference). This comment applies in many places in the doc: I would want the API reference to contain doc for all the code/functions/classes... and I would want a lot more links inside the doc pointing to the API entries.

Currently there's no documentation for the CSV loader config. Maybe we can add the docstrings to the `CsvConfig` class to explain the parameters and how it works, and then redirect to the doc of this class in this section of the documentation.

> * in the API reference (docstrings) I would prefer "SOURCE" to link to github instead of a copy of the code inside the docs site (eg. https://github.com/huggingface/datasets/blob/master/src/datasets/load.py#L711 instead of https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset)

This is the same as in `transformers`, not sure if this is a big issue

> * it seems like not all the API is exposed in the doc. For example, there is no doc for [`disable_progress_bar`](https://github.com/huggingface/datasets/search?q=disable_progress_bar), see https://huggingface.co/docs/datasets/search.html?q=disable_progress_bar, even if the code contains docstrings. Does it mean that the function is not officially supported? (otherwise, maybe it also deserves a mention in https://huggingface.co/docs/datasets/package_reference/logging_methods.html)

The function `disable_progress_bar` should definitely be in the docs, thanks. We can add it to the logging methods

> * in https://huggingface.co/docs/datasets/loading_datasets.html?highlight=most%20efficient%20format%20have%20json%20files%20consisting%20multiple%20json%20objects#json-files, "The most efficient format is to have JSON files consisting of multiple JSON objects, one per line, representing individual data rows:", maybe link to https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON and give it a name ("line-delimited JSON"? "JSON Lines" as in https://huggingface.co/docs/datasets/processing.html#exporting-a-dataset-to-csv-json-parquet-or-to-python-objects ?)

Yes good idea !

> * in https://huggingface.co/docs/datasets/loading_datasets.html, for the local files sections, it would be nice to provide sample csv / json / text files to download, so that it's easier for the reader to try to load them (instead: they won't try)

Sure why not. Moreover the csv loader now supports remote files so you could just run the code pass an an URL to the sample csv file.

> * the doc explains how to shard a dataset, but does not explain why and when a dataset should be sharded (I have no idea... for [parallelizing](https://huggingface.co/docs/datasets/processing.html#multiprocessing)?). It does neither give an idea of the number of shards a dataset typically should have and why.

This can be used for distributed processing or just to use a percentage of the data. We can definitely give example of use cases

> * the code example in https://huggingface.co/docs/datasets/processing.html#mapping-in-a-distributed-setting does not work, because `training_args` has not been defined before in the doc.

`training_args` comes from `transformers`, it's a practical way to define all your arguments to train a model. Maybe we can just import it from `transformers` and use it with the default values

Some comments here:

- the code samples assume the expected libraries have already been installed. Maybe add a section at start, or add it to every code sample. Something like `pip install datasets transformers torch  'datasets[streaming]'` (maybe just link to https://huggingface.co/docs/datasets/installation.html + a one-liner that installs all the requirements / alternatively a requirements.txt file)
- "If you’d like to play with the examples, you must install it from source." in https://huggingface.co/docs/datasets/installation.html: it's not clear to me what this means (what are these "examples"?)
- in https://huggingface.co/docs/datasets/loading_datasets.html: "or AWS bucket if it’s not already stored in the library". It's the only place in the doc (aside from the docstring https://huggingface.co/docs/datasets/package_reference/loading_methods.html?highlight=aws bucket#datasets.list_datasets) where the "AWS bucket" is mentioned. It's not easy to understand what this means. Maybe explain more, and link to https://s3.amazonaws.com/datasets.huggingface.co and/or https://huggingface.co/docs/datasets/filesystems.html.
- example in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files is obsoleted by https://github.com/huggingface/datasets/pull/2326. Also: see https://github.com/huggingface/datasets/issues/2691 for a bug on this specific dataset.
- in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files the doc says "After you’ve downloaded the files, you can point to the folder hosting them locally with the data_dir argument as follows:", but the following example does not show how to use `data_dir`
- in https://huggingface.co/docs/datasets/loading_datasets.html#csv-files, it would be nice to have an URL to the csv loader reference (but I'm not sure there is one in the API reference). This comment applies in many places in the doc: I would want the API reference to contain doc for all the code/functions/classes... and I would want a lot more links inside the doc pointing to the API entries.
- in the API reference (docstrings) I would prefer "SOURCE" to link to github instead of a copy of the code inside the docs site (eg. https://github.com/huggingface/datasets/blob/master/src/datasets/load.py#L711 instead of https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset)
- it seems like not all the API is exposed in the doc. For example, there is no doc for [`disable_progress_bar`](https://github.com/huggingface/datasets/search?q=disable_progress_bar), see https://huggingface.co/docs/datasets/search.html?q=disable_progress_bar, even if the code contains docstrings. Does it mean that the function is not officially supported? (otherwise, maybe it also deserves a mention in https://huggingface.co/docs/datasets/package_reference/logging_methods.html)
- in https://huggingface.co/docs/datasets/loading_datasets.html?highlight=most%20efficient%20format%20have%20json%20files%20consisting%20multiple%20json%20objects#json-files, "The most efficient format is to have JSON files consisting of multiple JSON objects, one per line, representing individual data rows:", maybe link to https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON and give it a name ("line-delimited JSON"? "JSON Lines" as in https://huggingface.co/docs/datasets/processing.html#exporting-a-dataset-to-csv-json-parquet-or-to-python-objects ?)
- in https://huggingface.co/docs/datasets/loading_datasets.html, for the local files sections, it would be nice to provide sample csv / json / text files to download, so that it's easier for the reader to try to load them (instead: they won't try)
- the doc explains how to shard a dataset, but does not explain why and when a dataset should be sharded (I have no idea... for [parallelizing](https://huggingface.co/docs/datasets/processing.html#multiprocessing)?). It does neither give an idea of the number of shards a dataset typically should have and why.
- the code example in https://huggingface.co/docs/datasets/processing.html#mapping-in-a-distributed-setting does not work, because `training_args` has not been defined before in the doc.


cannot save the dataset to disk after rename_column

Something isn't working

Hi ! That's because you are trying to overwrite a file that is already open and being used.
Indeed `foo/dataset.arrow` is open and used by your `dataset` object.

When you do `rename_column`, the resulting dataset reads the data from the same arrow file.
In other cases like when using `map` on the other hand, the resulting dataset reads the data from another arrow file that is the result of the map transform.

Therefore overwriting a dataset after `rename_column` is not possible, but it is possible after `map`, since `rename_column` doesn't switch to using another arrow file (the actual data stay the same).

## Describe the bug
If you use `rename_column` and do no other modification, you will be unable to save the dataset using `save_to_disk`

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
In [1]: from datasets import Dataset, load_from_disk
In [5]: dataset=Dataset.from_dict({'foo': [0]})
In [7]: dataset.save_to_disk('foo')
In [8]: dataset=load_from_disk('foo')
In [10]: dataset=dataset.rename_column('foo', 'bar')
In [11]: dataset.save_to_disk('foo')
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-11-a3bc0d4fc339> in <module>
----> 1 dataset.save_to_disk('foo')

/mnt/beegfs/projects/meerqat/anaconda3/envs/meerqat/lib/python3.7/site-packages/datasets/arrow_dataset.py in save_to_disk(self, dataset_path
, fs)
    597             if Path(dataset_path, config.DATASET_ARROW_FILENAME) in cache_files_paths:
    598                 raise PermissionError(
--> 599                     f"Tried to overwrite {Path(dataset_path, config.DATASET_ARROW_FILENAME)} but a dataset can't overwrite itself."
    600                 )
    601             if Path(dataset_path, config.DATASET_INDICES_FILENAME) in cache_files_paths:

PermissionError: Tried to overwrite foo/dataset.arrow but a dataset can't overwrite itself.
```

N. B. I created the dataset from dict to enable easy reproduction but the same happens if you load an existing dataset (e.g. starting from `In [8]`)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.10
- PyArrow version: 3.0.0


hebrew language codes he and iw should be treated as aliases

Something isn't working

Hi @eyaler, thanks for reporting.

While you are true with respect the Hebrew language tag ("iw" is deprecated and "he" is the preferred value), in the "mc4" dataset (which is a derived dataset) we have kept the language tags present in the original dataset: [Google C4](https://www.tensorflow.org/datasets/catalog/c4).

For discoverability on the website I updated the YAML tags at the top of the mC4 dataset card https://github.com/huggingface/datasets/commit/38288087b1b02f97586e0346e8f28f4960f1fd37

Once the website is updated, mC4 will be listed in https://huggingface.co/datasets?filter=languages:he

https://huggingface.co/datasets/mc4 not listed when searching for hebrew datasets (he) as it uses the older language code iw, preventing discoverability.


Minor documentation fix

Currently, [Writing a dataset loading script](https://huggingface.co/docs/datasets/add_dataset.html) page has a small error. A link to `matinf` dataset in [_Dataset scripts of reference_](https://huggingface.co/docs/datasets/add_dataset.html#dataset-scripts-of-reference) section actually leads to `xsquad`, instead. This PR fixes that.


Fix bad config ids that name cache directories

`data_dir=None` was considered a dataset config parameter, hence creating a special config_id for all dataset being loaded.
Since the config_id is used to name the cache directories, this leaded to datasets being regenerated for users.

I fixed this by ignoring the value of `data_dir` when it's `None` when computing the config_id.
I also added a test to make sure the cache directories are not unexpectedly renamed in the future.

Fix https://github.com/huggingface/datasets/issues/2683


Updated dataset description of DaNE

Thanks for finishing it @albertvillanova


How to sample every file in a list of files making up a split in a dataset when loading?

I am loading a dataset with multiple train, test, and validation files like this:

```
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
    split=['train[:8]', 'test[:8]', 'val[:8]']
)

```

However, this only selects the first 8 rows from train_file1, test_file1, val_file1, since they are the first files in the lists.

I'm trying to formulate a split argument that can sample from each file specified in my list of files that make up each split.

Is this type of splitting supported? If so, how can I do it?


Something isn't working

the bug code locate in ：
    if data_args.task_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset("glue", data_args.task_name, cache_dir=model_args.cache_dir)

Hi @jinec,

From time to time we get this kind of `ConnectionError` coming from the github.com website: https://raw.githubusercontent.com

Normally, it should work if you wait a little and then retry.

Could you please confirm if the problem persists?

cannot connect，even by Web browser，please check that  there is some  problems。

I can access https://raw.githubusercontent.com/huggingface/datasets/1.7.0/datasets/glue/glue.py without problem...

Hello,
I am trying to run run_glue.py and it gives me this error -

Traceback (most recent call last):
  File "E:/BERT/pytorch_hugging/transformers/examples/pytorch/text-classification/run_glue.py", line 546, in <module>
    main()
  File "E:/BERT/pytorch_hugging/transformers/examples/pytorch/text-classification/run_glue.py", line 250, in main
    datasets = load_dataset("glue", data_args.task_name, cache_dir=model_args.cache_dir)
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\load.py", line 718, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\load.py", line 320, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\utils\file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\utils\file_utils.py", line 623, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.7.0/datasets/glue/glue.py

Trying to do python run_glue.py  --model_name_or_path
bert-base-cased
--task_name
mrpc
--do_train
--do_eval
--max_seq_length
128
--per_device_train_batch_size
32
--learning_rate
2e-5
--num_train_epochs
3
--output_dir
./tmp/mrpc/

Is this something on my end? From what I can tell, this was re-fixeded by @fullyz a few months ago.
Thank you!


Support streaming compressed files

Add support to stream compressed files (current options in fsspec):
- bz2
- lz4
- xz
- zstd

cc: @lewtun


Add KS task to SUPERB

thanks a lot for implementing this @anton-l !!

i won't have time to review this while i'm away, so happy for @albertvillanova and @patrickvonplaten to decide when to merge :)

@albertvillanova thanks! Everything should be ready now :)

> The _background_noise_/_silence_ audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

@anton-l I was thinking that maybe we could give some hints in the dataset card (in a Usage section); something similar as for diarization: https://github.com/huggingface/datasets/blob/master/datasets/superb/README.md#example-of-usage
Note that for diarization it is not yet finished: we have to test it and then provide an end-to-end example: https://github.com/huggingface/datasets/pull/2661/files#r680224909

@albertvillanova yeah, I'm not sure how to best implement it in pure `datasets` yet. It's something like this, where `sample_noise()` needs to be called from a pytorch batch collator or other framework-specific variant:

```python
def map_to_array(example):
    import soundfile as sf

    speech_array, sample_rate = sf.read(example["file"])
    example["speech"] = speech_array
    example["sample_rate"] = sample_rate
    return example

def sample_noise(example):
    # Use a version of this function in a stateless way to extract random 1 sec slices of background noise
    # on each epoch
    from random import randint

    # _silence_ audios are longer than 1 sec
    if example["label"] == "_silence_":
        random_offset = randint(0, len(example["speech"]) - example["sample_rate"] - 1)
        example["speech"] = example["speech"][random_offset : random_offset + example["sample_rate"]]

    return example
```

I see... Yes, not trivial indeed. Maybe for the moment you could add those functions above to the README (as it is the case for now in diarization)? What do you think?

Add the KS (keyword spotting) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051).

- [s3prl instructions](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/README.md#ks-keyword-spotting)
- [s3prl implementation](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/speech_commands/dataset.py)
- [TFDS implementation](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/speech_commands.py)

Some notable quirks:
- The dataset is originally single-archive (train+val+test all in one), but the test set has a "canonical" distribution in a separate archive, which is also used here (see `_split_ks_files()`).
- The `_background_noise_`/`_silence_` audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

Related to #2619.


Fix renaming of corpus_bleu args

Last `sacrebleu` release (v2.0.0) has renamed `sacrebleu.corpus_bleu` args from `(sys_stream, ref_streams)` to `(hipotheses, references)`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR passes the args without parameter names, so that it is valid for all versions of `sacrebleu`.

This is a partial hotfix of #2781.

Close #2781.


Latest v2.0.0 release of sacrebleu has broken some metrics

Something isn't working

## Describe the bug
After `sacrebleu` v2.0.0 release (see changes here: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15), some of `datasets` metrics are broken:
- Default tokenizer `sacrebleu.DEFAULT_TOKENIZER` no longer exists:
  - #2739
  - #2778
- Bleu tokenizers are no longer accessible with `sacrebleu.TOKENIZERS`:
  - #2779
- `corpus_bleu` args have been renamed from `(sys_stream, ref_streams)` to `(hipotheses, references)`:
  - #2782


VIVOS dataset for Vietnamese ASR


Fix sacrebleu tokenizers

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.TOKENIZERS`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR makes a hot fix of the bug by using a private function in `sacrebleu`: `sacrebleu.metrics.bleu._get_tokenizer()`.

Eventually, this should be further fixed in order to use only public functions.

This is a partial hotfix of #2781.


Do not pass tokenize to sacrebleu

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.DEFAULT_TOKENIZER`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR does not pass `tokenize` to `sacrebleu` (note that the user cannot pass it anyway) and `sacrebleu` will use its default, no matter where it is and how it is called.

Related to #2739.

This is a partial hotfix of #2781.


Use packaging to handle versions

Use packaging module to handle/validate/check versions of Python packages.

Related to #2769.


document `config.HF_DATASETS_OFFLINE` and precedence

enhancement

New feature or request

https://github.com/huggingface/datasets/pull/1976 implemented `HF_DATASETS_OFFLINE`, but:
1. `config.HF_DATASETS_OFFLINE` is not documented
2. the precedence is not documented (env, config)

I'm thinking it probably should be similar to what it says https://huggingface.co/docs/datasets/loading_datasets.html#from-the-huggingface-hub about `datasets.config.IN_MEMORY_MAX_SIZE`:

Quote:
> The default in 🤗 Datasets is to memory-map the dataset on disk unless you set datasets.config.IN_MEMORY_MAX_SIZE different from 0 bytes (default). In that case, the dataset will be copied in-memory if its size is smaller than datasets.config.IN_MEMORY_MAX_SIZE bytes, and memory-mapped otherwise. This behavior can be enabled by setting either the configuration option datasets.config.IN_MEMORY_MAX_SIZE (higher precedence) or the environment variable HF_DATASETS_IN_MEMORY_MAX_SIZE (lower precedence) to nonzero.

Context: trying to use `config.HF_DATASETS_OFFLINE` here:
https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/48
but are uncertain if it's safe, since it's not documented as a public API.

Thank you!

@lhoestq, @albertvillanova


`generate_random_fingerprint()` deterministic with 🤗Transformers' `set_seed()`

bug

Something isn't working

I dug into what I believe is the root of this issue and added a repro in my comment. If this is better addressed as a cross-team issue, let me know and I can open an issue in the Transformers repo

## Describe the bug

**Update:** I dug into this to try to reproduce the underlying issue, and I believe it's that `set_seed()` from the `transformers` library makes the "random" fingerprint identical each time. I believe this is still a bug, because `datasets` is used exactly this way in `transformers` after `set_seed()` has been called, and I think that using `set_seed()` is a standard procedure to aid reproducibility. I've added more details to reproduce this below.

Hi there! I'm using my own local dataset and custom preprocessing function. My preprocessing function seems to be unpickle-able, perhaps because it is from a closure (will debug this separately). I get this warning, which is expected:

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L260-L265

However, what's not expected is that the `datasets` actually _does_ seem to cache and reuse this dataset between runs! After that line, the next thing that's logged looks like:

```text
 Loading cached processed dataset at /home/xxx/.cache/huggingface/datasets/csv/default-xxx/0.0.0/xxx/cache-xxx.arrow
```

The path is exactly the same each run (e.g., last 26 runs).

This becomes a problem because I'll pass in the `--max_eval_samples` flag to the HuggingFace example script I'm running off of ([run_swag.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/multiple-choice/run_swag.py)).  The fact that the cached dataset is reused means this flag gets ignored. I'll try to load 100 examples, and it will load the full cached 1,000,000.

I think that

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L248

... is actually consistent because randomness is being controlled in HuggingFace/Transformers for reproducibility. I've added a demo of this below.

## Steps to reproduce the bug

```python
# Contents of print_fingerprint.py
from transformers import set_seed
from datasets.fingerprint import generate_random_fingerprint
set_seed(42)
print(generate_random_fingerprint())
```

```bash
for i in {0..10}; do
    python print_fingerprint.py
done

1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
```

## Expected results
After the "random hash" warning is emitted, a random hash is generated, and no outdated cached datasets are reused.

## Actual results
After the "random hash" warning is emitted, an identical hash is generated each time, and an outdated cached dataset is reused each run.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Prevent .map from using multiprocessing when loading from cache

## Context

On our setup, we use different setup to train vs proprocessing datasets. Usually we are able to obtain a high number of cpus to preprocess, which allows us to use `num_proc` however we can't use as many during training phase. Currently if we use `num_proc={whatever the preprocessing value was}` we load from cache, but we get:

```
Traceback (most recent call last):
  File "lib/python3.8/site-packages/multiprocess/pool.py", line 131, in worker
    put((job, i, result))
  File "lib/python3.8/site-packages/multiprocess/queues.py", line 371, in put
    self._writer.send_bytes(obj)
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 203, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 414, in _send_bytes
    self._send(header + buf)
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 371, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
```

Our current guess, is that we're spawning too many workers compared to the umber of cpus available, and it's running OOM. (Currently we're testing if this fixes the issue) (**UPDATE**: sequentiality has fixed our issue)

Therefore something that might be interesting, would be the ability to load a multiprocessed dataset sequentially? Or at least agnostic of the num_proc (as they are only used to dertermine the number of shards), and allow users to choose how to load the shards?

## Current issues

~I'm having a hard time making fingerprints match. For some reason, the multiprocessing and the sequential version generate two different hash.~

**EDIT**: Turns out multiprocessing and sequential have different `transform` value for fingerprinting (check `fingerprint_transform`) when running `_map_single`:
 - sequential : `datasets.arrow_dataset.Dataset._map_single`
 - multiprocessing: `datasets.arrow_dataset._map_single`

 This discrepancy is caused by multiprocessing pickling the transformer function, it doesn't see to keep the `Dataset` hierarchy. I'm still unclear on why `func.__qual_name__` isn't handled correctly in multiprocessing. But replacing `__qualname__` by `__name__` fixes the issue.

## What was done

~We try to prevent the usage of multiprocessing when loading a dataset. Instead we load all cached shards sequentially.~

I couldn't find a nice way to obtain the cached_file_name and check they all exist before deciding to use the multiprocessing flow or not. Instead I expose an optional boolean `sequential` in `map` method.

## TODO
 - [x] Check that the multiprocessed version and the sequential version output the same output
 - [x] Check that sequential can load multiprocessed
 - [x] Check that multiprocessed can load sequential

 ## Test

```python
from datasets import load_dataset
from multiprocessing import Pool
import random

def process(batch, rng):
  length = len(batch["text"])
  return {**batch, "processed_text": [f"PROCESSED {rng.random()}" for _ in range(length)]}

dataset = load_dataset("stas/openwebtext-10k", split="train")
print(dataset.column_names)
print(type(dataset))

rng = random.Random(42)
dataset1 = dataset.map(process, batched=True, batch_size=50, num_proc=4, fn_kwargs={"rng": rng})

# This one should be loaded from cache
rng = random.Random(42)
dataset2 = dataset.map(process, batched=True, batch_size=50, num_proc=4, fn_kwargs={"rng": rng}, sequential=True)

# Just to check that the random generator was correct
print(dataset1[-1]["processed_text"])
print(dataset2[-1]["processed_text"])
```

 ## Other solutions

I chose to load everything sequentially, but we can probably find a way to load shards in parallel using another number of workers (essentially this would be an argument not used for fingerprinting, allowing to allow `m` shards using `n` processes, which would be very useful when same dataset have to be loaded on two different setup, and we still want to leverage cache).

cc @lhoestq (since I had asked you previously on `num_proc` being used for fingerprinting). Don't know if this is acceptable.


Remove dataset_infos.json

enhancement

New feature or request

generic discussion

Generic discussion on the library

**Is your feature request related to a problem? Please describe.**
As discussed, there are infos in the `dataset_infos.json` which are redundant and we could have them only in the README file.

Others could be migrated to the README, like: "dataset_size", "size_in_bytes", "download_size", "splits.split_name.[num_bytes, num_examples]",...

However, there are others that do not seem too meaningful in the README, like the checksums.

**Describe the solution you'd like**
Open a discussion to decide what to do with the `dataset_infos.json` files: which information to be migrated and/or which information to be kept.

cc: @julien-c @lhoestq


Remove returned feature constrain

enhancement

New feature or request

In the current version, the returned value of the map function has to be list or ndarray. However, this makes it unsuitable for many tasks. In NLP, many features are sparse like verb words, noun chunks, if we want to assign different values to different words, which will result in a large sparse matrix if we only score useful words like verb words.

Mostly, when using it on large scale, saving it as a whole takes a lot of disk storage and making it hard to read,  the normal method is saving it in sparse form. However, the NumPy does not support sparse, therefore I have to use PyTorch or scipy to transform a matrix into special sparse form, which is not a form that can be transformed into list or ndarry. This violates the feature constraints of the map function.

I do appreciate the convenience of Datasets package, but I do not think the compulsory datatype constrain is necessary, in some cases, we just cannot transform it into a list or ndarray due to some reasons.  Any way to fix this? Or what I can do to disable the compulsory datatype constrain?


[WIP][Common Voice 7] Add common voice 7.0

This PR allows to load the new common voice dataset manually as explained when doing:

```python
from datasets import load_dataset

ds = load_dataset("./datasets/datasets/common_voice_7", "ab")
```

=>

```
                   Please follow the manual download instructions:

                     You need to manually the dataset from `https://commonvoice.mozilla.org/en/datasets`.
                     Make sure you choose the version `Common Voice Corpus 7.0`.
                     Choose a language of your choice and find the corresponding language-id, *e.g.*, `Abkhaz` with language-id `ab`. The following language-ids are available:

                     ['ab', 'ar', 'as', 'az', 'ba', 'bas', 'be', 'bg', 'br', 'ca', 'cnh', 'cs', 'cv', 'cy', 'de', 'dv', 'el', 'en', 'eo', 'es', 'et', 'eu', 'fa', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gl', 'gn', 'ha', 'hi', 'hsb', 'hu', 'hy-AM', 'ia', 'id', 'it', 'ja', 'ka', 'kab', 'kk', 'kmr', 'ky', 'lg', 'lt', 'lv', 'mn', 'mt', 'nl', 'or', 'pa-IN', 'pl', 'pt', 'rm-sursilv', 'rm-vallader', 'ro', 'ru', 'rw', 'sah', 'sk', 'sl', 'sr', 'sv-SE', 'ta', 'th', 'tr', 'tt', 'ug', 'uk', 'ur', 'uz', 'vi', 'vot', 'zh-CN', 'zh-HK', 'zh-TW']

                     Next, you will have to enter your email address to download the dataset in the `tar.gz` format. Save the file under <path-to-file>.
                     The file should then be extracted with: ``tar -xvzf <path-to-file>`` which will extract a folder called ``cv-corpus-7.0-2021-07-21``.
                     The dataset can then be loaded with `datasets.load_dataset("common_voice", <language-id>, data_dir="<path-to-'cv-corpus-7.0-2021-07-21'-folder>", ignore_verifications=True).
```

Having followed those instructions one can then download the data as follows:

```python
from datasets import load_dataset

ds = load_dataset("./datasets/datasets/common_voice_7", "ab", data_dir="./cv-corpus-7.0-2021-07-21/", ignore_verifications=True)
```

## TODO
- [ ] Discuss naming. Is the name ok here "common_voice_7"? The dataset script differs only really in one point from `common_voice.py` in that all the metadata is different (more hours etc...) and that it has to use manual data dir for now
- [ ] Ideally we should get a bundled download link. For `common_voice.py` there is a bundled download link: `https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-6.1-2020-12-11/{}.tar.gz` that allows one to directly download the data. However such a link is missing for Common Voice 7. I guess we should try to contact common voice about it and ask whether we could host the data or help otherwise somehow. See: https://github.com/common-voice/common-voice-bundler/issues/15 cc @yjernite
- [ ] I did not compute the dataset.json and it would mean that I'd have to download 76 datasets totalling around 1TB manually before running the checksum command. This just takes too much time. For now the user will have to add a `ignore_verifications=True` to download the data. This step would also be much easier if we could get a bundled link
- [ ] Add dummy data


Add support for fast tokenizer in BertScore

This PR adds support for a fast tokenizer in BertScore, which has been added recently to the lib.
Fixes #2765


Allow PyArrow from source

When installing pyarrow from source the version is:

```python
>>> import pyarrow; pyarrow.__version__
'2.1.0.dev612'
```

-> however this breaks the install check at init of `datasets`. This PR makes sure that everything coming after the last `'.'` is removed.


`ArrowInvalid: Added column's length must match table's length.` after using `select`

bug

Something isn't working

Hi,

the `select` method creates an indices mapping and doesn't modify the underlying PyArrow table by default for better performance. To modify the underlying table after the `select` call, call `flatten_indices` on the dataset object as follows:
```python
from datasets import load_dataset

ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
ds = ds.flatten_indices()
ds = ds.add_column('ones', [1]*128)
```

Thanks for the question @lvwerra. And thanks for the answer @mariosasko. ^^

## Describe the bug
I would like to add a column to a downsampled dataset. However I get an error message saying the length don't match with the length of the unsampled dataset indicated. I suspect that the dataset size is not updated when calling `select`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
ds = ds.add_column('ones', [1]*128)
```

## Expected results
I would expect a new column named `ones` filled with `1`. When I check the length of `ds` it says `128`. Interestingly, it works when calling `ds = ds.map(lambda x: x)` before adding the column.

## Actual results
Specify the actual results or traceback.
```python
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
/var/folders/l4/2905jygx4tx5jv8_kn03vxsw0000gn/T/ipykernel_6301/868709636.py in <module>
      1 from datasets import load_dataset
      2 ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
----> 3 ds = ds.add_column('ones', [0]*128)

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    183         }
    184         # apply actual function
--> 185         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    186         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    187         # re-apply format to the output

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/arrow_dataset.py in add_column(self, name, column, new_fingerprint)
   2965         column_table = InMemoryTable.from_pydict({name: column})
   2966         # Concatenate tables horizontally
-> 2967         table = ConcatenationTable.from_tables([self._data, column_table], axis=1)
   2968         # Update features
   2969         info = self.info.copy()

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in from_tables(cls, tables, axis)
    715             table_blocks = to_blocks(table)
    716             blocks = _extend_blocks(blocks, table_blocks, axis=axis)
--> 717         return cls.from_blocks(blocks)
    718
    719     @property

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in from_blocks(cls, blocks)
    663             return cls(table, blocks)
    664         else:
--> 665             table = cls._concat_blocks_horizontally_and_vertically(blocks)
    666             return cls(table, blocks)
    667

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in _concat_blocks_horizontally_and_vertically(cls, blocks)
    623             if not tables:
    624                 continue
--> 625             pa_table_horizontally_concatenated = cls._concat_blocks(tables, axis=1)
    626             pa_tables_to_concat_vertically.append(pa_table_horizontally_concatenated)
    627         return cls._concat_blocks(pa_tables_to_concat_vertically, axis=0)

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in _concat_blocks(blocks, axis)
    612                 else:
    613                     for name, col in zip(table.column_names, table.columns):
--> 614                         pa_table = pa_table.append_column(name, col)
    615             return pa_table
    616         else:

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.append_column()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.add_column()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Added column's length must match table's length. Expected length 31962 but got length 128
```

## Environment info
- `datasets` version: 1.11.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 5.0.0


equal operation to perform unbatch for huggingface datasets

Something isn't working

Hi @lhoestq
Maybe this is clearer to explain like this, currently map function, map one example to "one" modified one, lets assume we want to map one example to "multiple" examples, in which we do not know in advance how many examples they would be per each entry. I greatly appreciate telling me how I can handle this operation, thanks a lot

Hi,
this is also my question on how to perform similar operation as "unbatch" in tensorflow in great huggingface dataset library.
thanks.

Hi,

`Dataset.map` in the batched mode allows you to map a single row to multiple rows. So to perform "unbatch", you can do the following:
```python
import collections

def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {ex['passage']}"
        new_batch["inputs"].extend([inputs] * len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"])
    return new_batch

dset = dset.map(unbatch, batched=True, remove_columns=dset.column_names)
```

Dear @mariosasko
First, thank you very much for coming back to me on this, I appreciate it a lot. I tried this solution,  I am getting errors, do you mind
giving me one test example to be able to run your code, to understand better the format of the inputs to your function?
in this function https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L952 they copy each example to the number of "answers", do you mean one should not do the copying part and use directly your function?

thank you very much for your help and time.

Hi @mariosasko
I think finally I got this, I think you mean to do things in one step, here is the full example for completeness:

```
def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        # updates the passage.
        passage = ex['passage']
        passage = re.sub(r'(\.|\?|\!|\"|\')\n@highlight\n', r'\1 ', passage)
        passage = re.sub(r'\n@highlight\n', '. ', passage)
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {passage}"
        # duplicates the samples based on  number of answers.
        num_answers = len(ex["answers"])
        num_duplicates = np.maximum(1, num_answers)
        new_batch["inputs"].extend([inputs] * num_duplicates) #len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"] if num_answers > 0 else ["<unk>"])
    return new_batch

data = datasets.load_dataset('super_glue', 'record', split="train", script_version="master")
data = data.map(unbatch, batched=True, remove_columns=data.column_names)
```

Thanks a lot again, this was a super great way to do it.

Hi
I need to use "unbatch" operation in tensorflow on a huggingface dataset, I could not find this operation, could you kindly direct me how I can do it, here is the problem I am trying to solve:

I am considering "record" dataset in SuperGlue and I need to replicate each entery of the dataset for each answer, to make it similar to what T5 originally did:

https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925

Here please find an example:

  For example, a typical example from ReCoRD might look like
  {
      'passsage': 'This is the passage.',
      'query': 'A @placeholder is a bird.',
      'entities': ['penguin', 'potato', 'pigeon'],
      'answers': ['penguin', 'pigeon'],
  }
  and I need a prosessor which would turn this example into the following two examples:
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'penguin',
  }
  and
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'pigeon',
  }

For doing this, one need unbatch, as each entry can map to multiple samples depending on the number of answers, I am not sure how to perform this operation with  huggingface datasets library and greatly appreciate your help

@lhoestq

Thank you very much.


fix typo (ShuffingConfig -> ShufflingConfig)

pretty straightforward, it should be Shuffling instead of Shuffing


BERTScore Error

Something isn't working

Hi,

The `use_fast_tokenizer` argument has been recently added to the bert-score lib. I've opened a PR with the fix. In the meantime, you can try to downgrade the version of bert-score with the following command to make the code work:
```
pip uninstall bert-score
pip install "bert-score<0.3.10"
```

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
predictions = ["hello there", "general kenobi"]
references = ["hello there", "general kenobi"]
bert = load_metric('bertscore')
bert.compute(predictions=predictions, references=references,lang='en')
```

# Bug
`TypeError: get_hash() missing 1 required positional argument: 'use_fast_tokenizer'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Colab
- Python version:
- PyArrow version:


Add DER metric for SUPERB speaker diarization task


English wikipedia datasets is not clean

Something isn't working

## Describe the bug
Wikipedia english dumps contain many wikipedia paragraphs like "References", "Category:" and "See Also" that should not be used for training.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
w = load_dataset('wikipedia', '20200501.en')
print(w['train'][0]['text'])
```

> 'Yangliuqing () is a market town in Xiqing District, in the western suburbs of Tianjin, People\'s Republic of China. Despite its relatively small size, it has been named since 2006 in the "famous historical and cultural market towns in China".\n\nIt is best known in China for creating nianhua or Yangliuqing nianhua. For more than 400 years, Yangliuqing has in effect specialised in the creation of these woodcuts for the New Year.  wood block prints using vivid colourschemes to portray traditional scenes of children\'s games often interwoven with auspiciouse objects.\n\n, it had 27 residential communities () and 25 villages under its administration.\n\nShi Family Grand Courtyard\n\nShi Family Grand Courtyard (Tiānjīn Shí Jiā Dà Yuàn, 天津石家大院) is situated in Yangliuqing Town of Xiqing District, which is the former residence of wealthy merchant Shi Yuanshi - the 4th son of Shi Wancheng, one of the eight great masters in Tianjin. First built in 1875, it covers over 6,000 square meters, including large and small yards and over 200 folk houses, a theater and over 275 rooms that served as apartments and places of business and worship for this powerful family. Shifu Garden, which finished its expansion in October 2003, covers 1,200 square meters, incorporates the elegance of imperial garden and delicacy of south garden. Now the courtyard of Shi family covers about 10,000 square meters, which is called the first mansion in North China. Now it serves as the folk custom museum in Yangliuqing, which has a large collection of folk custom museum in Yanliuqing, which has a large collection of folk art pieces like Yanliuqing New Year pictures, brick sculpture.\n\nShi\'s ancestor came from Dong\'e County in Shandong Province, engaged in water transport of grain. As the wealth gradually accumulated, the Shi Family moved to Yangliuqing and bought large tracts of land and set up their residence. Shi Yuanshi came from the fourth generation of the family, who was a successful businessman and a good household manager, and the residence was thus enlarged for several times until it acquired the present scale. It is believed to be the first mansion in the west of Tianjin.\n\nThe residence is symmetric based on the axis formed by a passageway in the middle, on which there are four archways. On the east side of the courtyard, there are traditional single-story houses with rows of rooms around the four sides, which was once the living area for the Shi Family. The rooms on north side were the accountants\' office. On the west are the major constructions including the family hall for worshipping Buddha, theater and the south reception room. On both sides of the residence are side yard rooms for maids and servants.\n\nToday, the Shi mansion, located in the township of Yangliuqing to the west of central Tianjin, stands as a surprisingly well-preserved monument to China\'s pre-revolution mercantile spirit. It also serves as an on-location shoot for many of China\'s popular historical dramas. Many of the rooms feature period furniture, paintings and calligraphy, and the extensive Shifu Garden.\n\nPart of the complex has been turned into the Yangliuqing Museum, which includes displays focused on symbolic aspects of the courtyards\'  construction, local folk art and customs, and traditional period furnishings and crafts.\n\n**See also \n\nList of township-level divisions of Tianjin\n\nReferences \n\n http://arts.cultural-china.com/en/65Arts4795.html\n\nCategory:Towns in Tianjin'**

## Expected results
I expect no junk in the data.

## Actual results
Specify the actual results or traceback.

## Environment info
- `datasets` version: 1.10.2
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 3.0.0


Add RVL-CDIP dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** RVL-CDIP
- **Description:** The RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) dataset consists of 400,000 grayscale images in 16 classes, with 25,000 images per class. There are 320,000 training images, 40,000 validation images, and 40,000 test images. The images are sized so their largest dimension does not exceed 1000 pixels.
- **Paper:** https://www.cs.cmu.edu/~aharley/icdar15/
- **Data:** https://www.cs.cmu.edu/~aharley/rvl-cdip/
- **Motivation:** I'm currently adding LayoutLMv2 and LayoutXLM to HuggingFace Transformers. LayoutLM (v1) already exists in the library. This dataset has a large value for document image classification (i.e. classifying scanned documents). LayoutLM models obtain SOTA on this dataset, so would be great to directly use it in notebooks.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Error loading C4 realnewslike dataset

bug

Something isn't working

Hi @danshirron,
`c4` was updated few days back by @lhoestq. The new configs are `['en', 'en.noclean', 'en.realnewslike', 'en.webtextlike'].` You'll need to remove any older version of this dataset you previously downloaded and then run `load_dataset` again with new configuration.

@bhavitvyamalik @lhoestq , just tried the above and got:
>>> a=datasets.load_dataset('c4','en.realnewslike')
Downloading: 3.29kB [00:00, 1.66MB/s]
Downloading: 2.40MB [00:00, 12.6MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 819, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 701, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 1049, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 268, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 360, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig en.realnewslike not found. Available: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']
>>>

datasets version is 1.11.0

I think I had an older version of datasets installed and that's why I commented the old configurations in my last comment, my bad! I re-checked and updated it to latest version (`datasets==1.11.0`) and it's showing `available configs: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']`.

I tried `raw_datasets = load_dataset('c4', 'realnewslike')` and the download started. Make sure you don't have any old copy of this dataset and you download it fresh using the latest version of datasets. Sorry for the mix up!

It works. I probably had some issue with the cache. after cleaning it im able to download the dataset. Thanks

## Describe the bug
Error loading C4 realnewslike dataset. Validation part mismatch

## Steps to reproduce the bug
```python
 raw_datasets = load_dataset('c4', 'realnewslike', cache_dir=model_args.cache_dir)
## Expected results
success on data loading
## Actual results
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.3M/15.3M [00:00<00:00, 28.1MB/s]Traceback (most recent call last):
  File "run_mlm_tf.py", line 794, in <module>
    main()
  File "run_mlm_tf.py", line 425, in main
    raw_datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 843, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 608, in download_and_prepare
    self._download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 698, in _download_and_prepare                                                                                verify_splits(self.info.splits, split_dict)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=38165657946, num_examples=13799838, dataset_name='c4'), 'recorded': SplitInfo(name='validation', num_bytes=37875873, num_examples=13863, dataset_name='c4')}]

## Environment info
- `datasets` version: 1.10.2
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1


Add Nuswide dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *NUSWIDE*
- **Description:** *[A Real-World Web Image Dataset from National University of Singapore](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html)*
- **Paper:** *[here](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/nuswide-civr2009.pdf)*
- **Data:** *[here](https://github.com/wenting-zhao/nuswide)*
- **Motivation:** *This dataset is a benchmark in the Text Retrieval task.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


the meteor metric seems not consist with the official version

bug

Something isn't working

the issue is caused by the differences between varied meteor versions:
meteor1.0 is for https://aclanthology.org/W07-0734.pdf
meteor1.5 is for https://aclanthology.org/W14-3348.pdf

here is a very similar issue in NLTK
https://github.com/nltk/nltk/issues/2655

Hi @jianguda, thanks for reporting.

Currently, at 🤗 `datasets` we are using METEOR 1.0 (indeed using NLTK: `from nltk.translate import meteor_score`): See the [citation here](https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py#L23-L35).

If there is some open source implementation of METEOR 1.5, that could be an interesting contribution! 😉

## Describe the bug
The computed meteor score seems strange because the value is very different from the scores computed by other tools. For example, I use the meteor score computed by [NLGeval](https://github.com/Maluuba/nlg-eval) as the reference (which reuses the official jar file for the computation)

## Steps to reproduce the bug
```python
from datasets import load_metric
from nlgeval import NLGEval, compute_individual_metrics

meteor = load_metric('meteor')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = ["It is a guide to action that ensures that the military will forever heed Party commands"]
results = meteor.compute(predictions=predictions, references=references)
# print the actual result
print(round(results["meteor"], 4))
metrics_dict = compute_individual_metrics(references, predictions[0])
# print the expected result
print(round(metrics_dict["METEOR"], 4))
```
By the way, you need to install the `nlg-eval` library first. Please check the installation guide [here](https://github.com/Maluuba/nlg-eval#setup), thanks!

## Expected results
`0.4474`

## Actual results
`0.7398`

## Environment info
- `datasets` version: 1.10.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 4.0.1


Raise ManualDownloadError when loading a dataset that requires previous manual download

This PR implements the raising of a `ManualDownloadError` when loading a dataset that requires previous manual download, and this is missing.

The `ManualDownloadError` is raised whether the dataset is loaded in normal or streaming mode.

Close #2749.

cc: @severo


Unexpected type after `concatenate_datasets`

Something isn't working

Hi @JulesBelveze, thanks for your question.

Note that 🤗  `datasets` internally store their data in Apache Arrow format.

However, when accessing dataset columns, by default they are returned as native Python objects (lists in this case).

If you would like their columns to be returned in a more suitable format for your use case (torch arrays), you can use the method `set_format()`:
```python
concat_dataset.set_format(type="torch")
```

You have detailed information in our docs:
- [Using a Dataset with PyTorch/Tensorflow](https://huggingface.co/docs/datasets/torch_tensorflow.html)
- [Dataset.set_format()](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format)

Thanks @albertvillanova it indeed did the job 😃
Thanks for your answer!

## Describe the bug
I am trying to concatenate two `Dataset` using `concatenate_datasets` but it turns out that after concatenation the features are casted from `torch.Tensor` to `list`.
It then leads to a weird tensors when trying to convert it to a `DataLoader`. However, if I use each `Dataset` separately everything behave as expected.

## Steps to reproduce the bug
```python
>>> featurized_teacher
Dataset({
    features: ['t_labels', 't_input_ids', 't_token_type_ids', 't_attention_mask'],
    num_rows: 502
})
>>> for f in featurized_teacher.features:
     print(featurized_teacher[f].shape)
torch.Size([502])
torch.Size([502, 300])
torch.Size([502, 300])
torch.Size([502, 300])

>>> featurized_student
Dataset({
    features: ['s_features', 's_labels'],
    num_rows: 502
})
>>> for f in featurized_student.features:
     print(featurized_student[f].shape)
torch.Size([502, 64])
torch.Size([502])
```
The shapes seem alright to me. Then the results after concatenation are as follow:
```python
>>> concat_dataset = datasets.concatenate_datasets([featurized_student, featurized_teacher], axis=1)
>>> type(concat_dataset["t_labels"])
<class 'list'>
```
One would expect to obtain the same type as the one before concatenation.

Am I doing something wrong here? Any idea on how to fix this unexpected behavior?

## Environment info
- `datasets` version: 1.9.0
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.9.5
- PyArrow version: 3.0.0


Fix metadata JSON for ubuntu_dialogs_corpus dataset

Related to #2743.


Fix metadata JSON for turkish_movie_sentiment dataset

Related to #2743.


Generate metadata JSON for telugu_books dataset

Related to #2743.


Generate metadata JSON for reclor dataset

Related to #2743.


Generate metadata JSON for lm1b dataset

Related to #2743.


Update metadata for wikihow dataset

Update metadata for wikihow dataset:
- Remove leading new line character in description and citation
- Update metadata JSON
- Remove no longer necessary `urls_checksums/checksums.txt` file

Related to #2748.


Second concatenation of datasets produces errors

Something isn't working

Hi @Aktsvigun, thanks for reporting.

I'm investigating this.

Hi @albertvillanova ,
any update on this? Can I probably help in some way?

Hi @Aktsvigun! We are planning to address this issue before our next release, in a couple of weeks at most. 😅

In the meantime, if you would like to contribute, feel free to open a Pull Request. You are welcome. Here you can find more information: [How to contribute to Datasets?](CONTRIBUTING.md)

Hi,

I am need to concatenate my dataset with others several times, and after I concatenate it for the second time, the features of features (e.g. tags names) are collapsed. This hinders, for instance, the usage of tokenize function with `data.map`.

```
from datasets import load_dataset, concatenate_datasets

data = load_dataset('trec')['train']
concatenated = concatenate_datasets([data, data])
concatenated_2 = concatenate_datasets([concatenated, concatenated])
print('True features of features:', concatenated.features)
print('\nProduced features of features:', concatenated_2.features)
```
outputs

```
True features of features: {'label-coarse': ClassLabel(num_classes=6, names=['DESC', 'ENTY', 'ABBR', 'HUM', 'NUM', 'LOC'], names_file=None, id=None), 'label-fine': ClassLabel(num_classes=47, names=['manner', 'cremat', 'animal', 'exp', 'ind', 'gr', 'title', 'def', 'date', 'reason', 'event', 'state', 'desc', 'count', 'other', 'letter', 'religion', 'food', 'country', 'color', 'termeq', 'city', 'body', 'dismed', 'mount', 'money', 'product', 'period', 'substance', 'sport', 'plant', 'techmeth', 'volsize', 'instru', 'abb', 'speed', 'word', 'lang', 'perc', 'code', 'dist', 'temp', 'symbol', 'ord', 'veh', 'weight', 'currency'], names_file=None, id=None), 'text': Value(dtype='string', id=None)}

Produced features of features: {'label-coarse': Value(dtype='int64', id=None), 'label-fine': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None)}
```

I am using `datasets` v.1.11.0


Raise a proper exception when trying to stream a dataset that requires to manually download files

Something isn't working

Hi @severo, thanks for reporting.

As discussed, datasets requiring manual download should be:
- programmatically identifiable
- properly handled with more clear error message when trying to load them with streaming

In relation with programmatically identifiability, note that for datasets requiring manual download, their builder have a property `manual_download_instructions` which is not None:
```python
# Dataset requiring manual download:
builder.manual_download_instructions is not None
```

## Describe the bug

At least for 'reclor', 'telugu_books', 'turkish_movie_sentiment', 'ubuntu_dialogs_corpus', 'wikihow', trying to `load_dataset` in streaming mode raises a `TypeError` without any detail about why it fails.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("reclor", streaming=True)
```

## Expected results

Ideally: raise a specific exception, something like `ManualDownloadError`.

Or at least give the reason in the message, as when we load in normal mode:

```python
from datasets import load_dataset
dataset = load_dataset("reclor")
```

```
AssertionError: The dataset reclor with config default requires manual data.
 Please follow the manual download instructions:   to use ReClor you need to download it manually. Please go to its homepage (http://whyu.me/reclor/) fill the google
  form and you will receive a download link and a password to extract it.Please extract all files in one folder and use the path folder in datasets.load_dataset('reclor', data_dir='path/to/folder/folder_name')
  .
 Manual data can be loaded with `datasets.load_dataset(reclor, data_dir='<path/to/manual/data>')
```

## Actual results

```
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Environment info

- `datasets` version: 1.11.0
- Platform: macOS-11.5-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Generate metadata JSON for wikihow dataset

Related to #2743.