Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()

To reproduce:

```python
>>> import datasets
>>> datasets.get_dataset_split_names('MalakhovIlya/RuREBus', config_name='raw_txt')
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 101, in _split_generators
    decode_file_names(folder)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 26, in decode_file_names
    for root, dirs, files in os.walk(folder, topdown=False):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 66, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
TypeError: xwalk() got an unexpected keyword argument 'topdown'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

It's not related to the dataset viewer. Maybe @albertvillanova or @lhoestq could help more on this issue.

Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.

@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed)

@mariosasko thank you for your reply. I couldn't reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced
```
def decode_file_names(folder):
    for root, dirs, files in os.walk(folder, topdown=False):
        root = Path(root)
        for file in files:
            old_name = root / Path(file)
            new_name = root / Path(
                file.encode('cp437').decode('cp866'))
            old_name.rename(new_name)
        for dir in dirs:
            old_name = root / Path(dir)
            new_name = root / Path(dir.encode('cp437').decode('cp866'))
            old_name.rename(new_name)

folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
decode_file_names(folder)
```
by
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
if not is_url(zip_file):
    folder = extract(zip_file)
else:
    folder = None
```
and now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn't find information about how it works. I would be very grateful, if you could tell me how to fix this)

This is what I get when I try to stream the `raw_txt` subset:
```python
>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)
>>> next(iter(dset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
```
So there is a bug in your script.

streaming=True helped me to find solution. I fixed
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
folder = extract(zip_file)
```
by
```
folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
path = os.path.join(folder, 'MED_txt/unparsed_txt')
for root, dirs, files in os.walk(path):
    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')
```
@mariosasko thank you for your help :)

## Dataset viewer issue for 'MalakhovIlya/RuREBus'

**Link:** https://huggingface.co/datasets/MalakhovIlya/RuREBus

**Description**
Using os.walk(topdown=False) in DatasetBuilder causes following error:
Status code:   400
Exception:     TypeError
Message:       xwalk() got an unexpected keyword argument 'topdown'
Couldn't find where "xwalk" come from. How can I fix this?

Am I the one who added this dataset ? Yes


Add single dataset citations for TweetEval

_The documentation is not available anymore as the PR was closed or merged._

The `test_dataset_cards` method is failing with the error:

```
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           The following typing errors are found: {'annotations_creators': "(Expected `typing.List` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)\nOR\n(Expected `typing.Dict` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)"}
```

Adding `found` as annotation creators.

This PR adds single data citations as per request of the original creators of the TweetEval dataset.

This is a recent email from the creator:

> Could I ask you a favor? Would you be able to add at the end of the README the citations of the single datasets as well? You can just copy our readme maybe? https://github.com/cardiffnlp/tweeteval#citing-tweeteval
(just to be sure that the creator of the single datasets also get credits when tweeteval is used)

Please let me know if this looks okay or if any changes are needed.

Thanks,
Gunjan


Support streaming xtreme dataset for PAN-X config.


ELI5 supporting documents

question

Further information is requested

Hi ! Please post your question on the [forum](https://discuss.huggingface.co/), more people will be able to help you there ;)

if i am using dense search to create supporting documents for eli5 how much time it will take bcz i read somewhere that it takes about 18 hrs??


HANS dataset preview broken

The dataset cannot be loaded, be it in normal or streaming mode.

```python
>>> import datasets
>>> dataset=datasets.load_dataset("hans", split="train", streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1595, in __next__
    out = self.readline()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1592, in readline
    return self.readuntil(b"\n")
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1581, in readuntil
    self.seek(start + found + len(char))
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 676, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file
>>> dataset=datasets.load_dataset("hans", split="train", streaming=False)
Downloading and preparing dataset hans/plain_text (download: 29.51 MiB, generated: 30.34 MiB, post-processed: Unknown size, total: 59.85 MiB) to /home/slesage/.cache/huggingface/datasets/hans/plain_text/1.0.0/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1087, in _prepare_split
    for key, record in logging.tqdm(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/hans/1bbcb735c482acd54f2e118074b59cfd2bf5f7a5a285d4d540d1e632216672ac/hans.py", line 121, in _generate_examples
    for idx, line in enumerate(open(filepath, "rb")):
ValueError: readline of closed file
```

Hi! I've opened a PR that should make this dataset stremable. You can test it as follows:
```python
from datasets import load_dataset
dset = load_dataset("hans", split="train", streaming=True, revision="49decd29839c792ecc24ac88f861cbdec30c1c40")
```

@severo The current script doesn't throw an error in normal mode (only in streaming mode) on my local machine or in Colab. Can you update your installation of `datasets` and see if that fixes the issue?

Thanks for this. It works well, thanks! The dataset viewer is using https://github.com/huggingface/datasets/releases/tag/2.0.0, I'm eager to upgrade to 2.0.1 😉

## Dataset viewer issue for '*hans*'

**Link:** [https://huggingface.co/datasets/hans](https://huggingface.co/datasets/hans)

HANS dataset preview is broken with error 400

Am I the one who added this dataset ? No


Support streaming xtreme dataset for PAWS-X config.


Support streaming xtreme dataset for udpos config.


Add SBU Captions Photo Dataset


dataset metadata for reproducibility

enhancement

New feature or request

When pulling a dataset from the hub, it would be useful to have some metadata about the specific dataset and version that is used. The metadata could then be passed to the `Trainer` which could then be saved to a model card. This is useful for people who run many experiments on different versions (commits/branches) of the same dataset.

The dataset could have a list of “source datasets” metadata and ignore what happens to them before arriving in the Trainer (i.e. ignore mapping, filtering, etc.).

Here is a basic representation (made by @lhoestq )
```python
>>> from datasets import load_dataset
>>>
>>> my_dataset = load_dataset(...)["train"]
>>> my_dataset = my_dataset.map(...)
>>>
>>> my_dataset.sources
[HFHubDataset(repo_id=..., revision=..., arguments={...})]
```


More robust `cast_to_python_objects` in `TypedSequence`

Adds a fallback to run an expensive version of `cast_to_python_objects` which exhaustively checks entire lists to avoid the `ArrowInvalid: Could not convert` error in `TypedSequence`. Currently, this error can happen in situations where only some images are decoded in `map`, in which case `cast_to_python_objects` fails to recognize that it needs to cast `PIL.Image` objects if they are not at the beginning of the sequence and stops after the first image dictionary (e.g., if `data` is `[{"bytes": None, "path": "some path"}, PIL.Image(), ...]`)

Fix #4124


Add configs with processed data in medical_dialog dataset

There exist processed data files that do not require parsing the raw data files (which can take long time).

Fix #4122.


dataset viewer issue for common_voice

dataset-viewer

Related to the dataset viewer on huggingface.co

Yes, it's a known issue, and we expect to fix it soon.

Fixed.

<img width="1393" alt="Capture d’écran 2022-04-25 à 15 42 05" src="https://user-images.githubusercontent.com/1676121/165101176-d729d85b-efff-45a8-bad1-b69223edba5f.png">

## Dataset viewer issue for 'common_voice'

**Link:** https://huggingface.co/datasets/common_voice

Server Error
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'

Am I the one who added this dataset ? No


BIG-bench

> It looks like the CI is failing on windows because our windows CI is unable to clone the bigbench repository (maybe it has to do with filenames that are longer than 256 characters, which windows don't like). Could the smaller installation of bigbench via pip solve this issue ?
> Otherwise we can see how to remove this limitation in our windows CI.

I'm not sure.
If it's git's fault that it can't handle the long filenames, it will possibly be resolved by the pip install. If it's an issue with windows not liking long filenames after it's installed, then it will not be resolved.
I don't have a windows computer to try it on, but I might be able to tweek this PR and do an experiment to find out.
We're waiting for a quota increase for the pip install (https://github.com/pypa/pypi-support/issues/1782). It's been pending for 2-3 weeks, and I don't have an estimate for when it will be resolved.

>Regarding the dummy data zip files, I think we can just keep datasets/bigbench/dummy/abstract_narrative_understanding/1.0.0/dummy_data.zip and remove all the other ones. We just require to have at least one dummy_data.zip file.

Sounds great. I will trim that down.

Do you know what are the other tests dependencies that have conflicts with bigbench ? I can try to split the CI to end up with a compatible list of test dependencies

Hi @lhoestq,

I haven't played with eliminating requirements form the test dependencies, and I've been trying to resolve this by modifying the bigbench repo to become compatible.
In the original bigbench repo, the version requirements were strict, and specifically it had a datasets==1.17.0 requirement which was causing trouble.
I'm working on PR https://github.com/google/BIG-bench/pull/766 to get some more flexible versions that might be compatible with the test dependencies in  HF/datasets.
We're somewhat flexible in modifying these version numbers if we can figure out what the exact conflict is.

I've spent some time experimenting with different versions, but I don't have a very efficient way of doing this debugging on my work computer (which for some reason doesn't produce the same sets of errors running python 3.9 instead of 3.6 or 3.7 in the tests).
It currently fails at
> The conflict is caused by:
>    bert-score 0.3.6 depends on matplotlib
>    big-bench 0.0.1 depends on matplotlib<4.0 and >=3.5.1

which doesn't seem like it can be the real issue.

If you have any advice for how to resolve these conflicts, that would be greatly appreciated!

Hi again @lhoestq,
After some more or less random guessing of conflicting packages, I've managed to find a configuration that seems to be compatible with HF/datasets.

The errors went away after removing version limits on matplotlib and scipy, and loosening numpy from 1.19 -> 1.17 in the bigbench requirements.

I might do some more tweaking to see if it lets me set some minimal limits on matplotlib and scipy, but I think we at least can move forward.

The WIN tests are still failing, now because of

> Did not find path entry C:\tools\miniconda3\bin
>C:\tools\miniconda3\envs\py37\python.exe: No module named pytest

I have no way of debugging this locally, and unless there's some way to get more verbose logs, I don't know why it's not finding pytest. Would you be able to take a quick look?

Update: Actually, I see it's still failing because of the long filenames. So perhaps the pytest error is just because the previous steps failed.

One more update on the WIN errors.
I think all the long filenames are in files in the github repo that does not need to be included.
We will try to remove them .

Hi ! The remaining error seems to be a `UnicodeDecodeError` from `setup.py`. I think you can fix your setup.py:
```diff
-     with open(os.path.join(os.path.dirname(__file__), fname)) as f:
+     with open(os.path.join(os.path.dirname(__file__), fname), encoding="utf-8") as f:
```
Indeed on windows, when you `open` a file it doesn't always use "utf-8" by default

Hi @lhoestq,
The dependency issues seems to now be resolved 🎉

Now, the WIN tests are failing at
> ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - botocore...
> ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - botocore...

Is this testing the dummy dataset that's added in bigbench? If so, I might need some help getting the right format in.

The error message I'm seeing is
> raise EndpointConnectionError(endpoint_url=request.url, error=e)
> E           botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://127.0.0.1:5555/test"

Which seems unrelated, but perhaps the real issue is somewhere I'm not seeing?

Woohoo awesome !

Let me check the CI error

Can you try to re-run the CI, just in case CircleCI messed up ?

Hi @lhoestq,
Rerunning did not seem to solve the problem.
The `test_dummy_dataset_serialize_s3` error still seems to remain.

Hi again @lhoestq,
I'm not sure if this is informative or not in terms of debugging, but I deleted the dummy data and the errors for windows still fail and the others still pass.
Do you have any idea what could be causing this error on windows?

_The documentation is not available anymore as the PR was closed or merged._

Now the last question: let's have the dataset under`google/bigbench` @andersjohanandreassen ?

I think it would be nicer, this way you and anyone in your team can update the dataset card whevener you want without going through a github PR. You just need to join the https://huggingface.co/google page using your google email :)

Hi @lhoestq,

Thank you so much for the help! I really appreciate it!!!

After some discussion with the other bigbench organizers, I think there is a slight preference for bigbench to not be under google/bigbench since this is a collaboration with researchers from many different institutions/organizations beyond Google.

I see the drawback with the updates to the dataset card having to go through a PR, but hopefully that won't be very frequent.

We're finalizing putting the bigbench api on pip, so once that's finalized I just need to update the setup.py with the correct dependency and I think we are ready to merge.

I noticed that in the latest windows CI run it takes forever to install the dependencies, was there any change in the bigbench dependencies recently ?

oh, sorry! I just did a double check on the dependencies, and it seems like there is at least one left that should have been removed.  There's also one new one added.
Let me get those removed again. Will ping you here when it's updated.

It looks like there is a circular dependency in `bigbench` at https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz

```python
>>> import bigbench.api.util as bb_utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/api/util.py", line 29, in <module>
    import bigbench.models.query_logging_model as query_logging_model
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/models/query_logging_model.py", line 23, in <module>
    import bigbench.api.util as util
AttributeError: module 'bigbench.api' has no attribute 'util'
```

Hi @lhoestq ,
I think we are ready to merge!

I have one minor question that I haven't been able to figure out:
Is there a way to bypass the `verify_infos` from triggering? I have `max_examples` as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have *very* many examples). But this is a variable that's not specified by the configs, so it raises an `NonMatchingSplitsSizesError`.
I wasn't able to work my way around this, but perhaps there is a way to bypass this that I'm not seeing?
If this cannot be done, I'm happy to ignore this for now.

Regarding pypi, we are working on a release there, but I'm told there is some issue that there is a problem regarding the upload, and we are not sure when it will be resolved, and it's not in my control.
I think merging this PR with the GCS is a great idea, and I will open a new PR when the pypi version is ready.

Cool ! Merging then :D

> Is there a way to bypass the verify_infos from triggering? I have max_examples as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have very many examples). But this is a variable that's not specified by the configs, so it raises an NonMatchingSplitsSizesError.

This is a bug, I opened an issue [here](https://github.com/huggingface/datasets/issues/4462). It should be easy to fix :)

The bigbench page is available here ! https://huggingface.co/datasets/bigbench

I think we can update the dataset viewer to install bigbench on it, but since this is production code I'd rather use the version on pypi for bigbench when it comes out

This PR adds all BIG-bench json tasks to huggingface/datasets.


Image decoding often fails when  transforming Image datasets

bug

Something isn't working

A quick hack I have found is that we can call the image first before running the transforms and it makes sure the image is decoded before being passed on.

For this I just needed to add `example['img'] = example['img']` to the top of my `generate_flipped_data` function, defined above, so that image decode in invoked.

After this minor change this function works:
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    example['img'] = example['img']  # <<< This is the only change
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example
```

Hi @RafayAK, thanks for reporting.

Current implementation of the Image feature performs the decoding only if the "img" field is accessed by the mapped function.

In your original `generate_flipped_data` function:
- it only accesses the "img" field (and thus performs decoding) if `rng.random() > p`;
- on the other hand, for the cases where `rng.random() <= p`, the "img" field is not accessed and thus no decoding is performed for those examples

By adding the code line `example['img'] = example['img']`, you make sure the "img" field is accessed in all cases, and the decoding is done for all examples.

Also note that there is a little bug in your implementation: `p` is not the probability of flipping, but the probability of not-flipping; the larger is `p`, the smaller is the probability of flipping.

Some refactoring (fixing also `p`):
```python
def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down.

    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    do_flip = rng.random() < p  # Note the "<" sign here instead of ">"
    example['img'] = example['img'].transpose(1) if do_flip else example['img']  # Note "img" is always accessed
    example['is_flipped'] = 1 if do_flip else 0
    return example

@albertvillanova Thanks for letting me know this is intended behavior. The docs are severely lacking on this, if I hadn't posted this here I would have never found out how I'm actually supposed to modify images in a Dataset object.

@albertvillanova Secondly if you check the error message it shows that around 1999 images were successfully created, I'm pretty sure some of them were also flipped during the process. Back to my main contention, sometimes the decoding takes place other times it fails.

I suppose to run `map` on any dataset all the examples should be invoked even if on some of them we end up doing nothing, is that right?

Hi @RafayAK! I've opened a PR with the fix, which adds a fallback to reattempt casting to PyArrow format with a more robust (but more expensive) procedure if the first attempt fails. Feel free to test it by installing `datasets` from the PR branch with the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-4124
```

@mariosasko I'll try this right away and report back.

@mariosasko Thanks a lot for looking into this, now the `map` function at least behaves as one would expect a function to behave.

Looking forward to exploring Hugging Face more and even contributing 😃.

```bash
 $ conda list | grep datasets
datasets                  2.0.1.dev0               pypi_0    pypi

```

```python
def preprocess_data(dataset):
    """
    Helper funtion to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping functions that transforms some of the image up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)
```

The output now show the function was applied successfully:
``` bash
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Downloading builder script: 5.61kB [00:00, 3.16MB/s]
Downloading metadata: 4.21kB [00:00, 2.56MB/s]
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
100%|██████████| 10000/10000 [00:01<00:00, 5149.15ex/s]
```

## Describe the bug
When transforming/modifying images in an image dataset using the `map` function the PIL images often fail to decode in time for the image transforms, causing errors.

Using a debugger it is easy to see what the problem is, the Image decode invocation does not take place and the resulting image passed around is still raw bytes:
```
[{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00 \x00\x00\x00 \x08\x02\x00\x00\x00\xfc\x18\xed\xa3\x00\x00\x08\x02IDATx\x9cEVIs[\xc7\x11\xeemf\xde\x82\x8d\x80\x08\x89"\xb5V\\\xb6\x94(\xe5\x9f\x90\xca5\x7f$\xa7T\xe5\x9f&9\xd9\x8a\\.\xdb\xa4$J\xa4\x00\x02x\xc0{\xb3t\xe7\x00\xca\x99\xd3\\f\xba\xba\xbf\xa5?|\xfa\xf4\xa2\xeb\xba\xedv\xa3f^\xf8\xd5\x0bY\xb6\x10\xb3\xaaDq\xcd\x83\x87\xdf5\xf3gZ\x1a\x04\x0f\xa0fp\xfa\xe0\xd4\x07?\x9dN\xc4\xb1\x99\xfd\xf2\xcb/\x97\x97\x97H\xa2\xaaf\x16\x82\xaf\xeb\xca{\xbf\xd9l.\xdf\x7f\xfa\xcb_\xff&\x88\x08\x00\x80H\xc0\x80@.;\x0f\x8c@#v\xe3\xe5\xfc\xd1\x9f\xee6q\xbf\xdf\xa6\x14\'\x93\xf1\xc3\xe5\xe3\xd1x\x14c\x8c1\xa5\x1c\x9dsM\xd3\xb4\xed\x08\x89SJ)\xa5\xedv\xbb^\xafNO\x97D\x84Hf ....
```

## Steps to reproduce the bug
```python
from datasets import load_dataset, Dataset
import numpy as np
# seeded NumPy random number generator for reprodducinble results.
rng = np.random.default_rng(seed=0)

test_dataset = load_dataset('cifar100', split="test")

def preprocess_data(dataset):
    """
    Helper function to pre-process HuggingFace Cifar-100 Dataset to remove fine_label and coarse_label columns and
    add is_flipped column
    Args:
        dataset: HuggingFace CIFAR-100 Dataset Object

    Returns:
        new_dataset: A Dataset object with "img" and "is_flipped" columns only

    """
    # remove fine_label and coarse_label columns
    new_dataset = dataset.remove_columns(['fine_label', 'coarse_label'])
    # add the column for is_flipped
    new_dataset = new_dataset.add_column(name="is_flipped", column=np.zeros((len(new_dataset)), dtype=np.uint8))

    return new_dataset

def generate_flipped_data(example, p=0.5):
    """
    A Dataset mapping function that transforms some of the images up-side-down.
    If the probability value (p) is 0.5 approximately half the images will be flipped upside-down
    Args:
        example: An example from the dataset containing a Python dictionary with "img" and "is_flipped" key-value pair
        p: the probability of flipping the image up-side-down, Default 0.5

    Returns:
        example: A Dataset object

    """
    # example['img'] = example['img']
    if rng.random() > p:  # the flip the image and set is_flipped column to 1
        example['img'] = example['img'].transpose(
            1)  # ImageOps.flip(example['img'])  #example['img'].transpose(Image.FLIP_TOP_BOTTOM)
        example['is_flipped'] = 1

    return example

my_test = preprocess_data(test_dataset)
my_test = my_test.map(generate_flipped_data)

```

## Expected results
The dataset should be transformed without problems.

## Actual results
```
/home/rafay/anaconda3/envs/pytorch_new/bin/python /home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
Reusing dataset cifar100 (/home/rafay/.cache/huggingface/datasets/cifar100/cifar100/1.0.0/f365c8b725c23e8f0f8d725c3641234d9331cd2f62919d1381d1baa5b3ba3142)
 20%|█▉        | 1999/10000 [00:00<00:01, 5560.44ex/s]
Traceback (most recent call last):
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2326, in _map_single
    writer.write(example)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 441, in write
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rafay/Documents/you_only_live_once/upside_down_detector/create_dataset.py", line 55, in <module>
    my_test = my_test.map(generate_flipped_data)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1953, in map
    return self._map_single(
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 519, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 486, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2360, in _map_single
    writer.finalize()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 522, in finalize
    self.write_examples_on_file()
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 399, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/rafay/anaconda3/envs/pytorch_new/lib/python3.10/site-packages/datasets/arrow_writer.py", line 185, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=32x32 at 0x7F56AEE61DE0> with type Image: did not recognize Python value type when inferring an Arrow data type

Process finished with exit code 1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux(Fedora 35)
- Python version: 3.10
- PyArrow version: 7.0.0


Building C4 takes forever

Something isn't working

Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...:
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


medical_dialog zh has very slow _generate_examples

Something isn't working

Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance.

Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this.
@albertvillanova please let me know if I am doing something unnecessary or time consuming.

Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url

from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten


datasets.load_metric can not load a local metirc

Something isn't working

## Describe the bug
No matter how I hard try to tell load_metric that I want to load a local metric file, it still continues to fetch things on the Internet. And unfortunately it says 'ConnectionError: Couldn't reach'. However I can download this file without connectionerror and tell load_metric its local directory. And it comes back where it begins...

## Steps to reproduce the bug
```python
metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

metric = load_metric(path='bleu')
    ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.12.1/metrics/bleu/bleu.py

metric = load_metric(path='./blue/bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py
```

## Expected results
I do read the docs [here](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_metric). There are no other parameters that help function to distinguish from local and online file but path. As what I code above, it should load from local.

## Actual results

> metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')

> ~\AppData\Local\Temp\ipykernel_19636\1855752034.py in <module>
----> 1 metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    817         if data_files is None and data_dir is not None:
    818             data_files = os.path.join(data_dir, "**")
--> 819
    820         self.name = name
    821         self.revision = revision
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    639         self,
    640         path: str,
--> 641         download_config: Optional[DownloadConfig] = None,
    642         download_mode: Optional[DownloadMode] = None,
    643         dynamic_modules_path: Optional[str] = None,
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    297             token = hf_api.HfFolder.get_token()
    298         if token:
--> 299             headers["authorization"] = f"Bearer {token}"
    300     return headers
    301
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    604             def _resumable_file_manager():
    605                 with open(incomplete_path, "a+b") as f:
--> 606                     yield f
    607
    608             temp_file_manager = _resumable_file_manager
ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.3.4

Any advice would be appreciated.


Representing dictionaries (json) objects as features

enhancement

New feature or request

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.

It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.


Hotfix failing CI tests on Windows

_The documentation is not available anymore as the PR was closed or merged._

This PR makes a hotfix for our CI Windows tests: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414

Fix #4118

I guess this issue is related to this PR:
- huggingface/huggingface_hub#815


Failing CI tests on Windows

bug

Something isn't working

## Describe the bug
Our CI Windows tests are failing from yesterday: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414


Something isn't working

Hi @arymbe, thanks for reporting.

Unfortunately, I'm not able to reproduce your problem.

Could you please write the complete stack trace? That way we will be able to see which package originates the exception.

Hello, thank you for your fast replied. this is the complete error that I got

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

Input In [27], in <module>
----> 1 from datasets import load_dataset

venv/lib/python3.8/site-packages/datasets/__init__.py:39, in <module>
     37 from .arrow_dataset import Dataset, concatenate_datasets
     38 from .arrow_reader import ReadInstruction
---> 39 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     40 from .combine import interleave_datasets
     41 from .dataset_dict import DatasetDict, IterableDatasetDict

venv/lib/python3.8/site-packages/datasets/builder.py:40, in <module>
     32 from .arrow_reader import (
     33     HF_GCP_BASE_URL,
     34     ArrowReader,
   (...)
     37     ReadInstruction,
     38 )
     39 from .arrow_writer import ArrowWriter, BeamWriter
---> 40 from .data_files import DataFilesDict, sanitize_patterns
     41 from .dataset_dict import DatasetDict, IterableDatasetDict
     42 from .features import Features

venv/lib/python3.8/site-packages/datasets/data_files.py:297, in <module>
    292     except FileNotFoundError:
    293         raise FileNotFoundError(f"The directory at {base_path} doesn't contain any data file") from None
    296 def _resolve_single_pattern_in_dataset_repository(
--> 297     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    298     pattern: str,
    299     allowed_extensions: Optional[list] = None,
    300 ) -> List[PurePath]:
    301     data_files_ignore = FILES_TO_IGNORE
    302     fs = HfFileSystem(repo_info=dataset_info)

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

This is weird... It is long ago that the package `huggingface_hub` has a submodule called `hf_api`.

Maybe you have a problem with your installed `huggingface_hub`...

Could you please try to update it?
```shell
pip install -U huggingface_hub
```

Yap, I've updated several times. Then, I've tried numeral combination of datasets and huggingface_hub versions. However, I think your point is right that there is a problem with my huggingface_hub installation. I'll try another way to find the solution. I'll update it later when I get the solution. Thank you :)

I'm sorry I can't reproduce your problem.

Maybe you could try to create a new Python virtual environment and install all dependencies there from scratch. You can use either:
- Python venv: https://docs.python.org/3/library/venv.html
- or conda venv (if you are using conda): https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Facing the same issue.

Response from `pip show datasets`
```
Name: datasets
Version: 1.15.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: aiohttp, dill, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, requests, tqdm, xxhash
Required-by: lm-eval
```

Response from `pip show huggingface_hub`

```
Name: huggingface-hub
Version: 0.8.1
Summary: Client library to download and publish models, datasets and other repos on the huggingface.co hub
Home-page: https://github.com/huggingface/huggingface_hub
Author: Hugging Face, Inc.
Author-email: julien@huggingface.co
License: Apache
Location: /usr/local/lib/python3.8/dist-packages
Requires: filelock, packaging, pyyaml, requests, tqdm, typing-extensions
Required-by: datasets
```

response from `datasets-cli env`

```
Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 5, in <module>
    from datasets.commands.datasets_cli import main
  File "/usr/local/lib/python3.8/dist-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/usr/local/lib/python3.8/dist-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/__init__.py", line 105, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```

A workaround:
I changed lines around Line 125 in `__init__.py` of `huggingface_hub` to something like
```
__getattr__, __dir__, __all__ = _attach(
    __name__,
    submodules=['hf_api'],
```
and it works ( which gives `datasets` direct access to `huggingface_hub.hf_api` ).

I was getting the same issue. After trying a few versions, following combination worked for me.
dataset==2.3.2
huggingface_hub==0.7.0

In another environment, I just installed latest repos from pip through `pip install -U transformers datasets tokenizers evaluate`, resulting in following versions. This also worked. Hope it helps someone.

datasets-2.3.2 evaluate-0.1.2 huggingface-hub-0.8.1 responses-0.18.0 tokenizers-0.12.1 transformers-4.20.1

For layoutlm_v3 finetune
datasets-2.3.2 evaluate-0.1.2 huggingface-hub-0.8.1 responses-0.18.0 tokenizers-0.12.1 transformers-4.12.5

## Describe the bug
Could you help me please. I got this following error.

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

## Steps to reproduce the bug
when I imported the datasets

# Sample code to reproduce the bug
from datasets import list_datasets, load_dataset, list_metrics, load_metric

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: macOS-12.3-x86_64-i386-64bit
- Python version: 3.8.9
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Huggingface-hub: 0.5.0
- Transformers: 4.18.0

Thank you in advance.


Pretty print dataset info files

maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small?

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

enhancement

New feature or request

Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default.

CC @mariosasko

Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova?

I think they should always ignore them actually ! Not sure if adding a flag would be helpful

@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually


Allow downloading just some columns of a dataset

New feature or request

In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess

Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.

**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.


Multiprocessing with FileLock fails in python 3.9

bug

Something isn't working

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock

def run(i):
    print(f"got the lock in multi process [{i}]")

with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c


ImageFolder with Grayscale images dataset

Hi! Replacing:
```python
transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")
```

with:
```python
def transform_func(examples):
    examples["image"] = [transforms(img).to("cuda") for img in examples["image"]]
    return examples

transformed_dataset = dataset.with_transform(transform_func)
```
should fix the issue. `datasets` doesn't support chaining of transforms (you can think of `set_format`/`with_format` as a predefined transform func for `set_transform`/`with_transforms`), so the last transform (in your case, `set_format`) takes precedence over the previous ones (in your case `with_format`). And the PyTorch formatter is not supported by the Image feature, hence the error (adding support for that is on our short-term roadmap).

Ok thanks a lot for the code snippet!

I love the way `datasets` is easy to use but it made it really long to pre-process all the images (400.000 in my case) before training anything. `ImageFolder` from pytorch is faster in my case but force me to have the images on my local machine.

I don't know how to speed up the process without switching to `ImageFolder` :smile:

You can pass `ignore_verifications=True` in `load_dataset` to skip checksum verification, which takes a lot of time if the number of files is large. We will consider making this the default behavior.

Hi, I'm facing a problem with a grayscale images dataset I have uploaded [here](https://huggingface.co/datasets/ChainYo/rvl-cdip) (RVL-CDIP)

I'm getting an error while I want to use images for training a model with PyTorch DataLoader. Here is the full traceback:

```bash
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1765, in __getitem__
    return self._getitem(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1750, in _getitem
    formatted_output = format_table(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 314, in map_nested
    mapped = [
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 315, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 251, in _single_map_nested
    return function(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
```

I don't really understand why the image is still a bytes object while I used transformations on it. Here the code I used to upload the dataset (and it worked well):

```python
train_dataset = load_dataset("imagefolder", data_dir="data/train")
train_dataset = train_dataset["train"]
test_dataset = load_dataset("imagefolder", data_dir="data/test")
test_dataset = test_dataset["train"]
val_dataset = load_dataset("imagefolder", data_dir="data/val")
val_dataset = val_dataset["train"]

dataset = DatasetDict({
    "train": train_dataset,
    "val": val_dataset,
    "test": test_dataset
})
dataset.push_to_hub("ChainYo/rvl-cdip")
```

Now here is the code I am using to get the dataset and prepare it for training:

```python
img_size = 512
batch_size = 128
normalize = [(0.5), (0.5)]
data_dir = "ChainYo/rvl-cdip"

dataset = load_dataset(data_dir, split="train")

transforms = transforms.Compose([
        transforms.Resize(img_size),
        transforms.CenterCrop(img_size),
        transforms.ToTensor(),
        transforms.Normalize(*normalize)
])

transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")

train_dataloader = torch.utils.data.DataLoader(
    transformed_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory=True
)
```

But this get me the error above. I don't understand why it's doing this kind of weird thing?
Do I need to map something on the dataset? Something like this:

```python
labels = dataset.features["label"].names
num_labels = dataset.features["label"].num_classes

def preprocess_data(examples):
    images = [ex.convert("RGB") for ex in examples["image"]]
    labels = [ex for ex in examples["label"]]
    return {"images": images, "labels": labels}

features = Features({
    "images": Image(decode=True, id=None),
    "labels": ClassLabel(num_classes=num_labels, names=labels)
})

decoded_dataset = dataset.map(preprocess_data, remove_columns=dataset.column_names, features=features, batched=True, batch_size=100)
```


Update security policy

_The documentation is not available anymore as the PR was closed or merged._


Matthews Correlation Metric Card


Add Spearmanr Metric Card

changes made! @lhoestq let me know what you think

The CI fail is unrelated to this PR and fixed on master, feel free to merge :)


Perplexity Speedup

WRT the high values, can you add some unit tests with some [string, model] pairs and their resulting perplexity code, and @TristanThrush can run the same pairs through his version of the code?

> I thought that the perplexity metric should output the average perplexity value of all the strings that it gets as input (not a perplexity value per string, as the new version does). @lhoestq , @TristanThrush thoughts?

I support this change from Emi. If we have a perplexity function that loads GPT2 and then returns an average over all of the strings, then it is impossible to get multiple perplexities of a batch of strings efficiently. If we have this new perplexity function that is built for batching, then it is possible to get a batch of perplexities efficiently and you can still compute the average efficiently afterwards.

Thanks a lot for working on this @emibaylor @TristanThrush :)

For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
```python
return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
```
we're also doing this for the COMET metric.

> Thanks a lot for working on this @emibaylor @TristanThrush :)
>
> For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
>
> ```python
> return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
> ```
>
> we're also doing this for the COMET metric.

Thanks! Sounds great to me.

The CI fail is unrelated to your PR and has been fixed on master, feel free to merge the master branch into your PR to fix the CI ;)

This PR makes necessary changes to perplexity such that:
- it runs much faster (via batching)
- it throws an error when input is empty, or when input is one word without <BOS> token
- it adds the option to add a <BOS> token

Issues:
- The values returned are extremely high, and I'm worried they aren't correct. Even if they are correct, they are sometimes returned as `inf`, which is not very useful (see [comment below](https://github.com/huggingface/datasets/pull/4108#discussion_r843931094) for some of the output values).
  - If the values are not correct, can you help me find the error?
  - If the values are correct, it might be worth it to measure something like perplexity per word, which would allow us to get actual values for the larger perplexities, instead of just `inf`

Future:
- `stride` is not currently implemented here. I have some thoughts on how to make it happen with batching, but I think it would be better to get another set of eyes to look at any possible errors causing such large values now rather than later.


Unable to view the dataset and loading the same dataset throws the error - ArrowInvalid: Exceeded maximum rows

Something isn't working

  It's not related to the dataset viewer in itself. I can replicate the error with:

```
>>> import datasets as ds
>>> d = ds.load_dataset('Pavithree/explainLikeImFive')
Using custom data configuration Pavithree--explainLikeImFive-b68b6d8112cd8a51
Downloading and preparing dataset json/Pavithree--explainLikeImFive to /home/slesage/.cache/huggingface/datasets/json/Pavithree--explainLikeImFive-b68b6d8112cd8a51/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 305M/305M [00:03<00:00, 98.6MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.9M/17.9M [00:00<00:00, 75.7MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [00:00<00:00, 70.6MB/s]
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.92s/it]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1948.42it/s]
Failed to read file '/home/slesage/.cache/huggingface/datasets/downloads/5fee9c8819754df277aee6f252e4db6897d785231c21938407b8862ca871d246' with error <class 'pyarrow.lib.ArrowInvalid'>: Exceeded maximum rows
Traceback (most recent call last):
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 144, in _generate_tables
    dataset = json.load(f)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 916 (char 915)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/load.py", line 1691, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 1151, in _prepare_split
    for key, table in logging.tqdm(
  File "/home/slesage/.pyenv/versions/datasets/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise e
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 122, in _generate_tables
    pa_table = paj.read_json(
  File "pyarrow/_json.pyx", line 246, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```

cc @lhoestq @albertvillanova @mariosasko

It seems that train.json is not a valid JSON Lines file: it has several JSON objects in the first line (the 915th character in the first line starts a new object, and there's no "\n")

You need to have one JSON object per line

I'm closing this issue.

@Pavithree, please, feel free to re-open it if fixing the JSON file does not solve it.

## Dataset viewer issue - -ArrowInvalid: Exceeded maximum rows

**Link:** *https://huggingface.co/datasets/Pavithree/explainLikeImFive*

*This is the subset of original eli5 dataset  https://huggingface.co/datasets/vblagoje/lfqa. I just filtered the data samples which belongs to one particular subreddit thread. However, the dataset preview for train split  returns  the below mentioned error:
Status code:   400
Exception:     ArrowInvalid
Message:       Exceeded maximum rows
When I try to load the same dataset it returns ArrowInvalid: Exceeded maximum rows error*

Am I the one who added this dataset ? Yes


Support huggingface_hub 0.5

Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`:
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6?

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, <s>and I set the `hugginface_hub` requirement to `>=0.5.0`</s>

cc @adrinjalali @LysandreJik


push to hub fails with huggingface-hub 0.5.0

Something isn't working

Hi ! Indeed there was a breaking change in `huggingface_hub` 0.5.0 in `HfApi.create_repo`, which is called here in `datasets` by passing the org name in both the `repo_id` and the `organization` arguments:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

I think we should fix that in `huggingface_hub`, will keep you posted. In the meantime please use `huggingface_hub` 0.4.0

I'll be sending a fix for this later today on the `huggingface_hub` side.

The error would be converted to a `FutureWarning` if `datasets` uses kwargs instead of positional, for example here:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

to be:

``` python
            api.create_repo(
                name=dataset_name,
                token=token,
                repo_type="dataset",
                organization=organization,
                private=private,
            )
```

But `name` and `organization` are deprecated in `huggingface_hub=0.5`, and people should pass `repo_id='org/name` instead. Note that `repo_id` was introduced in 0.5 and if `datasets` wants to support older `huggingface_hub` versions (which I encourage it to do), there needs to be a helper function to do that. It can be something like:

```python
def create_repo(
    client,
    name: str,
    token: Optional[str] = None,
    organization: Optional[str] = None,
    private: Optional[bool] = None,
    repo_type: Optional[str] = None,
    exist_ok: Optional[bool] = False,
    space_sdk: Optional[str] = None,
) -> str:
    try:
        return client.create_repo(
            repo_id=f"{organization}/{name}",
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
    except TypeError:
        return client.create_repo(
            name=name,
            organization=organization,
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
```

in a `utils/_fixes.py` kinda file and and be used internally.

I'll be sending a patch to `huggingface_hub` to convert the error reported in this issue to a `FutureWarning`.

PR with the hotfix on the `huggingface_hub` side: https://github.com/huggingface/huggingface_hub/pull/822

We can definitely change `push_to_hub` to use `repo_id` in `datasets` and require `huggingface_hub>=0.5.0`.

Let me open a PR :)

`huggingface_hub` 0.5.1 just got released with a fix, feel free to update `huggingface_hub` ;)

## Describe the bug
`ds.push_to_hub` is failing when updating a dataset in the form "org_id/repo_id"

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("rubrix/news_test")
ds.push_to_hub("<your-user>/news_test", token="<your-token>")
```

## Expected results
The dataset is successfully uploaded

## Actual results
An error validation is raised:

```bash
if repo_id and (name or organization):
>           raise ValueError(
                "Only pass `repo_id` and leave deprecated `name` and "
                "`organization` to be None."
E               ValueError: Only pass `repo_id` and leave deprecated `name` and `organization` to be None.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.1
- `huggingface-hub`: 0.5
- Platform: macOS
- Python version: 3.8.12
- PyArrow version: 6.0.0

cc @adrinjalali


Add time series data - stock market

dataset request

Requesting to add a new dataset

Can I use instructions present in below link for time series dataset as well?
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md

cc'ing @kashif and @NielsRogge for visibility!

@INF800 happy to add this dataset! I will try to set a PR by the end of the day... if you can kindly point me to the dataset? Also, note we have a bunch of time series datasets checked in e.g. `electricity_load_diagrams` or `monash_tsf`, and ideally this dataset could also be in a similar format.

Thankyou. This is how raw data looks like before cleaning for an individual stocks:

1. https://github.com/INF800/marktech/tree/raw-data/f/data/raw
2. https://github.com/INF800/marktech/tree/raw-data/t/data/raw
3. https://github.com/INF800/marktech/tree/raw-data/rdfn/data/raw
4. https://github.com/INF800/marktech/tree/raw-data/irbt/data/raw
5. https://github.com/INF800/marktech/tree/raw-data/hll/data/raw
6. https://github.com/INF800/marktech/tree/raw-data/infy/data/raw
7. https://github.com/INF800/marktech/tree/raw-data/reli/data/raw
8. https://github.com/INF800/marktech/tree/raw-data/hdbk/data/raw

> Scraping is automated using GitHub Actions. So, everyday we will see a new file added in the above links.

I can rewrite the cleaning scripts to make sure it fits HF dataset standards. (P.S I am very much new to HF dataset)

The data set above can be converted into univariate regression / multivariate regression / sequence to sequence generation dataset etc. So, do we have some kind of transformation modules that will read the dataset as some type of dataset (`GenericTimeData`) and convert it to other possible dataset relating to a specific ML task. **By having this kind of transformation module, I only have to add data once** and use transformation module whenever necessary

Additionally, having some kind of versioning for the dataset will be really helpful because it will keep on updating - especially time series datasets

thanks @INF800 I'll have a look. I believe it should be possible to incorporate this into the time-series format.

@INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.

In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with...

Do you think you can make a version with just numerical data?

> @INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.
>
> In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with...
>
> Do you think you can make a version with just numerical data?

Will share the numeric data and conversion script within end of this week.

I am on a business trip currently - it is in my desktop.

## Adding a Time Series Dataset
- **Name:** 2min ticker data for stock market
- **Description:** 8 stocks' data collected for 1month post ukraine-russia war. 4 NSE stocks and 4 NASDAQ stocks. Along with technical indicators (additional features) as shown in below image
- **Data:** Collected by myself from investing.com
- **Motivation:** Test applicability of transformer based model on stock market / time series problem

![image](https://user-images.githubusercontent.com/45640029/161904077-52fe97cb-3720-4e3f-98ee-7f6720a056e2.png)


Add the `GSM8K` dataset

_The documentation is not available anymore as the PR was closed or merged._

The CI is failing because it's outdated, but the task tags are updated on `master`, merging :)


[hub] Fix `api.create_repo` call?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4102). All of your documentation changes will be reflected on that endpoint.

Closing in favor of https://github.com/huggingface/datasets/pull/4106


enhancement

New feature or request

Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.

How can I download only the train and test split for full numbers using load_dataset()?

I do not need the extra split and it will take 40 mins just to download in Colab. I have very short time in hand. Please help.


Improve RedCaps dataset card

I find this preprocessing a bit too specific to add it as a method to `datasets` as it's only useful in the context of CV (and we support multiple modalities). However, I agree it would be great to move this code to another lib to avoid code duplication. Maybe we should create a package with preprocessing functions/transforms for this purpose?

This PR modifies the RedCaps card to:
* fix the formatting of the Point of Contact fields on the Hub
* speed up the image fetching logic (aligns it with the [img2dataset](https://github.com/rom1504/img2dataset) tool) and make it more robust (return None if **any** exception is thrown)


bug

Something isn't working

Hi @andreybond, thanks for reporting.

Unfortunately, I'm not able to able to reproduce your issue:
```python
In [4]: from datasets import load_dataset
   ...: datasets = load_dataset("nielsr/XFUN", "xfun.ja")

In [5]: datasets
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 194
    })
    validation: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 71
    })
})
```

The only reason I can imagine this issue may arise is if your default encoding is not "UTF-8" (and it is ASCII instead). This is usually the case on Windows machines; but you say your environment is a Linux machine. Maybe you change your machine default encoding?

Could you please check this?
```python
In [6]: import sys

In [7]: sys.getdefaultencoding()
Out[7]: 'utf-8'
```

I opened a PR in the original dataset loading script:
-  microsoft/unilm#677

and fixed the corresponding dataset script on the Hub:
- https://huggingface.co/datasets/nielsr/XFUN/commit/73ba5e026621e05fb756ae0f267eb49971f70ebd

import sys
sys.getdefaultencoding()

returned: 'utf-8'

---------------------

I've just cloned master branch - your fix works! Thank you!

## Describe the bug
Error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)" is thrown when downloading dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("nielsr/XFUN", "xfun.ja")
```

## Expected results
Dataset should be downloaded without exceptions

## Actual results
Stack trace (for the second-time execution):
Downloading and preparing dataset xfun/xfun.ja to /root/.cache/huggingface/datasets/nielsr___xfun/xfun.ja/0.0.0/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477...
Downloading data files: 100%
2/2 [00:00<00:00, 88.48it/s]
Extracting data files: 100%
2/2 [00:00<00:00, 79.60it/s]

UnicodeDecodeErrorTraceback (most recent call last)
<ipython-input-31-79c26bd1109c> in <module>
      1 from datasets import load_dataset
      2
----> 3 datasets = load_dataset("nielsr/XFUN", "xfun.ja")

/usr/local/lib/python3.6/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    604         )
    605
--> 606         # By default, return all splits
    607         if split is None:
    608             split = {s: s for s in self.info.splits}

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    692         Args:
    693             split: `datasets.Split` which subset of the data to read.
--> 694
    695         Returns:
    696             `Dataset`

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)

/usr/local/lib/python3.6/dist-packages/tqdm/notebook.py in __iter__(self)
    252         if not self.disable:
    253             self.display(check_delay=False)
--> 254
    255     def __iter__(self):
    256         try:

/usr/local/lib/python3.6/dist-packages/tqdm/std.py in __iter__(self)
   1183             for obj in iterable:
   1184                 yield obj
-> 1185             return
   1186
   1187         mininterval = self.mininterval

~/.cache/huggingface/modules/datasets_modules/datasets/nielsr--XFUN/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477/XFUN.py in _generate_examples(self, filepaths)
    140             logger.info("Generating examples from = %s", filepath)
    141             with open(filepath[0], "r") as f:
--> 142                 data = json.load(f)
    143
    144             for doc in data["documents"]:

/usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    294
    295     """
--> 296     return loads(fp.read(),
    297         cls=cls, object_hook=object_hook,
    298         parse_float=parse_float, parse_int=parse_int,

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0 (but reproduced with many previous versions)
- Platform: Docker: Linux da5b74136d6b 5.3.0-1031-azure #32~18.04.1-Ubuntu SMP Mon Jun 22 15:27:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ; Base docker image is : huggingface/transformers-pytorch-cpu
- Python version: 3.6.9
- PyArrow version:   6.0.1


Proposing WikiSplit metric card

A quick Github tip ;) To avoid running N times the CI, you can push all the changes at once: go to Files Changed tab, and on each suggestion there's a "add to commit batch" and then you can do one commit for all the suggestions you want to approve ;)

Pinging @lhoestq to ensure that my distinction between the dataset and the metric are clear :sweat_smile:


Updating FrugalScore metric card

removing duplicate paragraph