Dataset Viewer issue for rajistics/indian_food_images

Hi, thanks for reporting. I triggered a refresh of the preview for this dataset, and it works now. I'm not sure what occurred.
<img width="1019" alt="Capture d’écran 2022-07-18 à 11 01 52" src="https://user-images.githubusercontent.com/1676121/179541327-f62ecd5e-a18a-4d91-b316-9e2ebde77a28.png">

### Link

https://huggingface.co/datasets/rajistics/indian_food_images/viewer/rajistics--indian_food_images/train

### Description

I have a train/test split in my dataset
<img width="410" alt="Screen Shot 2022-07-15 at 11 44 42 AM" src="https://user-images.githubusercontent.com/6808012/179293215-7b419ec3-3527-46f2-8dad-adbc5568cfa0.png">
t
The dataset viewer works for the test split (images of indian food), but does not show my train split.  My guess is maybe there is some corrupt image file that is guessing this.  But I have no idea.

The original dataset was pulled from here: https://www.kaggle.com/datasets/l33tc0d3r/indian-food-classification?resource=download-directory

### Owner

Yes


Refactor base extractors

_The documentation is not available anymore as the PR was closed or merged._

This PR:
- Refactors base extractors as subclasses of `BaseExtractor`:
  - this is an abstract class defining the interface with:
    - `is_extractable`: abstract class method
    - `extract`: abstract static method
- Implements abstract `MagicNumberBaseExtractor` (as subclass of `BaseExtractor`):
  - this has a default implementation of `is_extractable`
  - this improves performance (reducing the number of file reads) by allowing passing already read `magic_number`
- Refactors `Extractor`:
  - reads magic number from file only once

This PR deprecates:
```python
is_extractable, extractor = self.extractor.is_extractable(input_path, return_extractor=True)
self.extractor.extract(input_path, output_path, extractor=extractor)
```
and uses more Pythonic instead:
```python
extractor_format = self.extractor.infer_extractor_format(input_path)
self.extractor.extract(input_path, output_path, extractor_format)
```


Test extractors for all compression formats

This PR:
- Adds all compression formats to `test_extractor`
- Tests each base extractor for all compression formats

Note that all compression formats are tested except "rar".


Skip test_extractor only for zstd param if zstandard not installed

Currently, if `zstandard` is not installed, `test_extractor` is skipped for all compression format parameters.

This PR fixes `test_extractor` so that if `zstandard` is not installed, `test_extractor` is skipped only for the `zstd` compression parameter, that is, it is not skipped for all the other compression parameters (`gzip`, `xz`,...).


Trigger CI also on push to main

Currently, new CI (on GitHub Actions) is only triggered on pull requests branches when the base branch is main.

This PR also triggers the CI when a PR is merged to main branch.


Align logging with Transformers (again)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4686). All of your documentation changes will be reflected on that endpoint.

I wasn't aware of https://github.com/huggingface/datasets/pull/1845 before opening this PR. This issue seems much more complex now ...

Fix #2832


Fix mock fsspec

This PR:
- Removes an unused method from `DummyTestFS`
- Refactors `mock_fsspec` to make it simpler


How to assign new values to Dataset?

enhancement

New feature or request

Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.

![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```


Update create dataset card docs

documentation

Improvements or additions to documentation

This PR proposes removing the [online dataset card creator](https://huggingface.co/datasets/card-creator/) in favor of simply copy/pasting a template and using the [Datasets Tagger app](https://huggingface.co/spaces/huggingface/datasets-tagging) to generate the tags. The Tagger app provides more guidance by showing all possible values a user can select in the dropdown menus, whereas the online dataset card creator doesn't, which can make it difficult to know what tag values to input.

Let me know what you think! :)


weird issue/bug with columns (dataset iterable/stream mode)

I have a dataset online (CloverSearch/cc-news-mutlilingual) that has a bunch of columns, two of which are "score_title_maintext" and "score_title_description". the original files are jsonl formatted. I was trying to iterate through via streaming mode and grab all "score_title_description" values, but I kept getting key not found after a certain point of iteration. I found that some json objects in the file don't have "score_title_description". And in SOME cases, this returns a NONE and in others it just gets a key error. Why is there an inconsistency here and how can I fix it?


IndexError when loading ImageFolder

bug

Something isn't working

## Describe the bug
Loading an image dataset with `imagefolder` throws `IndexError: list index out of range` when the given folder contains a non-image file (like a csv).

## Steps to reproduce the bug
Put a csv file in a folder with images and load it:
```python
import datasets
 datasets.load_dataset("imagefolder", data_dir=path/to/folder)
```

## Expected results
I would expect a better error message, like `Unsupported file` or even the dataset loader just ignoring every file that is not an image in that case.

## Actual results
Here is the whole traceback:

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.11.0-051100-generic-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset Viewer issue for codeparrot/xlcost-text-to-code

There seems to be an issue with the `C++-snippet-level` config:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("codeparrot/xlcost-text-to-code", "C++-snippet-level")
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 352, in get_dataset_config_info
    info.splits = {
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

I remove the dataset-viewer tag since it's not directly related.

Pinging @huggingface/datasets

Thanks I found that this subset wasn't properly defined the the config, I fixed it. Now I can see the subsets but I get this error for the viewer
````
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

Yes, the cache is being refreshed, hopefully, it will work in some minutes for all the splits. Some are already here:

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code/viewer/Python-snippet-level/train

<img width="1533" alt="Capture d’écran 2022-07-18 à 12 04 06" src="https://user-images.githubusercontent.com/1676121/179553933-64d874fa-ada9-4b82-900e-082619523c20.png">

I think all the splits are working as expected now

### Link

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code

### Description

Error
```
Server Error
Status code:   400
Exception:     TypeError
Message:       'NoneType' object is not iterable
```
Before I did a minor change in the dataset script (removing some comments), the viewer was working but not properely, it wasn't showing the dataset subsets. But the data can be loaded successfully.

Thanks!

### Owner

Yes


Added method to remove excess nesting in a DatasetDict

Hi ! I think the issue you linked is closed and suggests to use `remove_columns`.

Moreover if you end up with a dataset with an unnecessarily nested data, please modify your processing functions to not output nested data, or use `map(..., batched=True)` if you function take batches as input

Hi @lhoestq , you are right about the issues this pull has steered beyond that issue. I created this [colab notebook](https://colab.research.google.com/drive/16aLu6QrDSV_aUYRdpufl5E4iS08qkUGj?usp=sharing) to present the error. I tried using batch and that won't resolve it either. I'm looking into that error right now.

I think you just need to pass one example at a time to your tokenizer, this way you don't end up with nested data:
```python

def preprocessFunction(row):
  collatedContext = tokenizer.eos_token.join([row["context"+str(i+1)] for i in range(int(AMT_OF_CONTEXT))])
  response = row["response"]
  tokenizedContext = tokenizer(
    collatedContext, max_length=max_context_length, truncation=True  # don't pass as a list here
  )
  with tokenizer.as_target_tokenizer():
    tokenized_response = tokenizer(
      response, max_length=max_response_length, truncation=True  # don't pass a a list here
    )
  tokenizedContext["labels"] = tokenized_response["input_ids"]
  return tokenizedContext
```

Yes that is correct, the purpose of this pull is to advise of a more general solution like with `def remove_excess_nesting(self)` or maybe automate the solution (stas00 advised not to automate it as it could "not be backwards compatible").

I'm not sure I understand how having `remove_excess_nesting` would make more sense than just fixing the preprocessFunction to simply not return nested samples, can you elaborate ?

Figuring out the issue can be a bit difficult to figure out. Only until I added batch does it make a little more sense with the error

> sequence item 0: expected str instance, list found

but batch was never intended.

When you run the colab you will notice that only until collating do you learn there is this error. So i figured it would be better to address it during at the `DatasetDict` level.
I think it would be ideal if the user could be notified at the preprocess function.

I'm not arguing that `remove_excess_nesting` is the right solution but what I aim to address is dealing with unnecessary nesting as early as possible.

> When you run the colab you will notice that only until collating do you learn there is this error.

I think users can just check the `dataset.features` and they would notice that the data are nested
```python
{
   'input_ids': Sequence(Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None)
   ...
}
```

Sometime nested data are intentional, so you can't know in advance if it's a user's mistake or something planned.

Added the ability for a DatasetDict to remove additional nested layers within its features to avoid conflicts when collating. It is meant to accompany [this PR](https://github.com/huggingface/transformers/pull/18119) to resolve the same issue [#15505](https://github.com/huggingface/transformers/issues/15505).

@stas00 @lhoestq


Cant pass streaming dataset to dataloader after take()

Something isn't working

Hi! Calling `take` on an iterable/streamable dataset makes it not possible to shard the dataset, which in turn disables multi-process loading (attempts to split the workload over the shards), so to go past this limitation, you can either use single-process loading in `DataLoader` (`num_workers=None`) or fetch the first `50_000/batch_size` batches in the loop.

## Describe the bug
I am trying to pass a streaming version of c4 to a dataloader, but it can't be passed after I call `dataset.take(n)`. Some functions such as `shuffle()` can be applied without breaking the dataloader but not take.

## Steps to reproduce the bug
```python
import datasets
import torch

dset = datasets.load_dataset(path='c4', name='en', split="train", streaming=True)
dset = dset.take(50_000)
dset = dset.with_format("torch")

num_workers = 8
batch_size = 512

loader = torch.utils.data.DataLoader(dataset=dset,
                                     batch_size=batch_size,
                                     num_workers=num_workers)
for batch in loader:
    ...
```

## Expected results
No error thrown when iterating over the dataloader

## Actual results
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/root/.local/lib/python3.9/site-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py", line 48, in __iter__
    for key, example in self._iter_shard(shard_idx):
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 586, in _iter_shard
    yield from ex_iterable.shard_data_sources(shard_idx)
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 60, in shard_data_sources
    raise NotImplementedError(f"{type(self)} doesn't implement shard_data_sources yet")
NotImplementedError: <class 'datasets.iterable_dataset.TakeExamplesIterable'> doesn't implement shard_data_sources yet

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Random 400 Client Error when pushing dataset

Something isn't working

## Describe the bug
When pushing a dataset, the client errors randomly with `Bad Request for url:...`.
At the next call, a new parquet file is created for each shard.
The client may fail at any random shard.

## Steps to reproduce the bug
```python
dataset.push_to_hub("ORG/DATASET", private=True, branch="main")
```

## Expected results
Push all the dataset to the Hub with no duplicates.
If it fails, it should retry or fail, but continue from the last failed shard.

## Actual results
```
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
testing.ipynb Cell 29 in <cell line: 1>()
----> [1](testing.ipynb?line=0) dataset.push_to_hub("ORG/DATASET", private=True, branch="main")

File ~/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py:4297, in Dataset.push_to_hub(self, repo_id, split, private, token, branch, max_shard_size, shard_size, embed_external_files)
   4291     warnings.warn(
   4292         "'shard_size' was renamed to 'max_shard_size' in version 2.1.1 and will be removed in 2.4.0.",
   4293         FutureWarning,
   4294     )
   4295     max_shard_size = shard_size
-> 4297 repo_id, split, uploaded_size, dataset_nbytes, repo_files, deleted_size = self._push_parquet_shards_to_hub(
   4298     repo_id=repo_id,
   4299     split=split,
   4300     private=private,
   4301     token=token,
   4302     branch=branch,
   4303     max_shard_size=max_shard_size,
   4304     embed_external_files=embed_external_files,
   4305 )
   4306 organization, dataset_name = repo_id.split("/")
   4307 info_to_dump = self.info.copy()

File ~/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py:4195, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, embed_external_files)
   4193         shard.to_parquet(buffer)
   4194         uploaded_size += buffer.tell()
-> 4195         _retry(
   4196             api.upload_file,
   4197             func_kwargs=dict(
   4198                 path_or_fileobj=buffer.getvalue(),
   4199                 path_in_repo=shard_path_in_repo,
   4200                 repo_id=repo_id,
   4201                 token=token,
   4202                 repo_type="dataset",
   4203                 revision=branch,
   4204                 identical_ok=False,
   4205             ),
   4206             exceptions=HTTPError,
   4207             status_codes=[504],
   4208             base_wait_time=2.0,
   4209             max_retries=5,
   4210             max_wait_time=20.0,
   4211         )
   4212     shards_path_in_repo.append(shard_path_in_repo)
   4214 # Cleanup to remove unused files

File ~/.local/lib/python3.9/site-packages/datasets/utils/file_utils.py:284, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    282 except exceptions as err:
    283     if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):
--> 284         raise err
    285     else:
    286         sleep_time = min(max_wait_time, base_wait_time * 2**retry)  # Exponential backoff

File ~/.local/lib/python3.9/site-packages/datasets/utils/file_utils.py:281, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    279 while True:
    280     try:
--> 281         return func(*func_args, **func_kwargs)
    282     except exceptions as err:
    283         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File ~/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py:1967, in HfApi.upload_file(self, path_or_fileobj, path_in_repo, repo_id, token, repo_type, revision, identical_ok, commit_message, commit_description, create_pr)
   1957 commit_message = (
   1958     commit_message
   1959     if commit_message is not None
   1960     else f"Upload {path_in_repo} with huggingface_hub"
   1961 )
   1962 operation = CommitOperationAdd(
   1963     path_or_fileobj=path_or_fileobj,
   1964     path_in_repo=path_in_repo,
   1965 )
-> 1967 pr_url = self.create_commit(
   1968     repo_id=repo_id,
   1969     repo_type=repo_type,
   1970     operations=[operation],
   1971     commit_message=commit_message,
   1972     commit_description=commit_description,
   1973     token=token,
   1974     revision=revision,
   1975     create_pr=create_pr,
   1976 )
   1977 if pr_url is not None:
   1978     re_match = re.match(REGEX_DISCUSSION_URL, pr_url)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py:1844, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads)
   1836 commit_url = f"{self.endpoint}/api/{repo_type}s/{repo_id}/commit/{revision}"
   1838 commit_resp = requests.post(
   1839     url=commit_url,
   1840     headers={"Authorization": f"Bearer {token}"},
   1841     json=commit_payload,
   1842     params={"create_pr": 1} if create_pr else None,
   1843 )
-> 1844 _raise_for_status(commit_resp)
   1845 return commit_resp.json().get("pullRequestUrl", None)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:84, in _raise_for_status(request)
     76 if request.status_code == 401:
     77     # The repo was not found and the user is not Authenticated
     78     raise RepositoryNotFoundError(
     79         f"401 Client Error: Repository Not Found for url: {request.url}. If the"
     80         " repo is private, make sure you are authenticated. (Request ID:"
     81         f" {request_id})"
     82     )
---> 84 _raise_with_request_id(request)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:95, in _raise_with_request_id(request)
     92 if request_id is not None and len(e.args) > 0 and isinstance(e.args[0], str):
     93     e.args = (e.args[0] + f" (Request ID: {request_id})",) + e.args[1:]
---> 95 raise e

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:90, in _raise_with_request_id(request)
     88 request_id = request.headers.get("X-Request-Id")
     89 try:
---> 90     request.raise_for_status()
     91 except Exception as e:
     92     if request_id is not None and len(e.args) > 0 and isinstance(e.args[0], str):

File ~/.local/lib/python3.9/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/ORG/DATASET/commit/main (Request ID: a_F0IQAHJdxGKVRYyu1cF)
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.31
- Python version: 3.9.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset.map gets stuck on _cast_to_python_objects

Something isn't working

Are you able to reproduce this? My example is small enough that it should be easy to try.

Hi! Thanks for reporting and providing a reproducible example. Indeed, by default, `datasets` performs an expensive cast on the values returned by `map` to convert them to one of the types supported by PyArrow (the underlying storage format used by `datasets`). This cast is not needed on NumPy arrays as PyArrow supports them natively, so one way to make this transform faster is to add `return_tensors="np"` to the tokenizer call.

I think we should mention this in the docs (cc @stevhliu)

I tested this tokenize function and indeed noticed a casting. However it seems to only concerns the `offset_mapping` field, which contains a list of tuples, that is converted to a list of lists. Since `pyarrow` also supports tuples, we actually don't need to convert the tuples to lists.

I think this can be changed here:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L382-L383

```diff
- if isinstance(obj, list):
+ if isinstance(obj, (list, tuple)):
```

and here:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L386-L387

```diff
- return obj if isinstance(obj, list) else [], isinstance(obj, tuple)
+ return obj, False
```

@srobertjames can you try applying these changes and let us know if it helps ? If so, feel free to open a Pull Request to contribute this improvement if you want :)

Wow, adding `return_tensors="np"` sped up my example by a **factor 17x** of and completely eliminated the casting! I'd recommend not only to document it, but to make that the default.

The code at https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb does not specify `return_tensors="np"` but yet avoids the casting penalty.  How does it do that?  (The ntbk seems to do `return_overflowing_tokens=True, return_offsets_mapping=True,`).

Also, surprisingly enough, using `return_tensors="pt"` (which is my eventual application) yields this error:
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types
[<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>].
When using `batched=True`, make sure provided `function` returns a `dict` of types like
`(<class 'list'>, <class 'numpy.ndarray'>)`.
```

Setting the output to `"np"` makes the whole pipeline fast because it moves the data buffers from rust to python to arrow using zero-copy, and also because it does eliminate the casting completely ;)

Have you had a chance to try eliminating the tuple casting using the trick above ?

@lhoestq I just benchmarked the two edits to `features.py` above, and they appear to solve the problem, bringing my original example to within 20% the speed of the output `"np"` example.  Nice!

For a pull request, do you suggest simply following https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md ?

Cool ! Sure feel free to follow these instructions to open a PR :) thanks !

## Describe the bug
`Dataset.map`, when fed a Huggingface Tokenizer as its map func, can sometimes spend huge amounts of time doing casts.  A minimal example follows.

Not all usages suffer from this.  For example, I profiled the preprocessor at https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb , and it did _not_ have this problem.  However, I'm at a loss to figure out how it avoids it, as the example below is simple and minimal and still has this problem.

This casting, where it occurs, causes the `Dataset.map` to run approximately 7x slower than it runs for code which does not cause this casting.

This may be related to https://github.com/huggingface/datasets/issues/1046 .  However, the tokenizer is _not_ set to return Tensors.

## Steps to reproduce the bug
A minimal, self-contained example to reproduce is below:
```python
import transformers
from transformers import AutoTokenizer
from datasets import load_dataset
import torch
import cProfile

pretrained = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(pretrained)

squad = load_dataset('squad')
squad_train = squad['train']
squad_tiny = squad_train.select(range(5000))

assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)

def tokenize(ds):
        tokens = tokenizer(text=ds['question'],
                                text_pair=ds['context'],
                                add_special_tokens=True,
                                padding='max_length',
                                truncation='only_second',
                                max_length=160,
                                stride=32,
                                return_overflowing_tokens=True,
                                return_offsets_mapping=True,
                                )
        return tokens

cmd = 'squad_tiny.map(tokenize, batched=True, remove_columns=squad_tiny.column_names)'
cProfile.run(cmd, sort='tottime')
```

## Actual results
The code works, but takes 10-25 sec per batch (about 7x slower than non-casting code), with the following profile.  Note that `_cast_to_python_objects` is the culprit.
```
    63524075 function calls (58206482 primitive calls) in 121.836 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
5274034/40   68.751    0.000  111.060    2.776 features.py:262(_cast_to_python_objects)
 42223832   24.077    0.000   33.310    0.000 {built-in method builtins.isinstance}
 16338/20    5.121    0.000  111.053    5.553 features.py:361(<listcomp>)
  5274135    4.747    0.000    4.749    0.000 {built-in method _abc._abc_instancecheck}
    80/40    4.731    0.059  116.292    2.907 {pyarrow.lib.array}
  5274135    4.485    0.000    9.234    0.000 abc.py:96(__instancecheck__)
2661564/2645196    2.959    0.000    4.298    0.000 features.py:1081(_check_non_null_non_empty_recursive)
        5    2.786    0.557    2.786    0.557 {method 'encode_batch' of 'tokenizers.Tokenizer' objects}
  2668052    0.930    0.000    0.930    0.000 {built-in method builtins.len}
     5000    0.930    0.000    0.938    0.000 tokenization_utils_fast.py:187(_convert_encoding)
        5    0.750    0.150    0.808    0.162 {method 'to_pydict' of 'pyarrow.lib.Table' objects}
        1    0.444    0.444  121.749  121.749 arrow_dataset.py:2501(_map_single)
       40    0.375    0.009  116.291    2.907 arrow_writer.py:151(__arrow_array__)
       10    0.066    0.007    0.066    0.007 {method 'write_batch' of 'pyarrow.lib._CRecordBatchWriter' objects}
        1    0.060    0.060  121.835  121.835 fingerprint.py:409(wrapper)
11387/5715    0.049    0.000    0.175    0.000 {built-in method builtins.getattr}
       36    0.049    0.001    0.049    0.001 {pyarrow._compute.call_function}
    15000    0.040    0.000    0.040    0.000 _collections_abc.py:719(__iter__)
        3    0.023    0.008    0.023    0.008 {built-in method _imp.create_dynamic}
       77    0.020    0.000    0.020    0.000 {built-in method builtins.dir}
       37    0.019    0.001    0.019    0.001 socket.py:543(send)
       15    0.017    0.001    0.017    0.001 tokenization_utils_fast.py:460(<listcomp>)
  432/421    0.015    0.000    0.024    0.000 traitlets.py:1388(_notify_observers)
     5000    0.015    0.000    0.018    0.000 _collections_abc.py:672(keys)
       51    0.014    0.000    0.042    0.001 traitlets.py:276(getmembers)
        5    0.014    0.003    3.775    0.755 tokenization_utils_fast.py:392(_batch_encode_plus)
      3/1    0.014    0.005    0.035    0.035 {built-in method _imp.exec_dynamic}
        5    0.012    0.002    0.950    0.190 tokenization_utils_fast.py:438(<listcomp>)
    31626    0.012    0.000    0.012    0.000 {method 'append' of 'list' objects}
1532/1001    0.011    0.000    0.189    0.000 traitlets.py:643(get)
        5    0.009    0.002    3.796    0.759 arrow_dataset.py:2631(apply_function_on_filtered_inputs)
       51    0.009    0.000    0.062    0.001 traitlets.py:1766(traits)
        5    0.008    0.002    3.784    0.757 tokenization_utils_base.py:2632(batch_encode_plus)
      368    0.007    0.000    0.044    0.000 traitlets.py:1715(_get_trait_default_generator)
       26    0.007    0.000    0.022    0.001 traitlets.py:1186(setup_instance)
       51    0.006    0.000    0.010    0.000 traitlets.py:1781(<listcomp>)
    80/32    0.006    0.000    0.052    0.002 table.py:1758(cast_array_to_feature)
      684    0.006    0.000    0.007    0.000 {method 'items' of 'dict' objects}
4344/1794    0.006    0.000    0.192    0.000 traitlets.py:675(__get__)
...
```
## Environment info
I observed this on both Google colab and my local workstation:

### Google colab
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

### Local
- `datasets` version: 2.3.2
- Platform: Windows-7-6.1.7601-SP1
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Unable to use dataset with PyTorch dataloader

Something isn't working

Hi! `para_crawl` has a single column of type `Translation`, which stores translation dictionaries. These dictionaries can be stored in a NumPy array but not in a PyTorch tensor since PyTorch only supports numeric types. In `datasets`, the conversion to `torch` works as follows:
1. convert PyArrow table to NumPy arrays
2. convert NumPy arrays to Torch tensors.

The 2nd step is problematic for your case as `datasets` attempts to convert the array of dictionaries to a PyTorch tensor. One way to fix this is to use the [preprocessing logic](https://github.com/huggingface/transformers/blob/8581a798c0a48fca07b29ce2ca2ef55adcae8c7e/examples/pytorch/translation/run_translation.py#L440-L458) from the Transformers translation script. And on our side, I think we can replace a NumPy array of dicts with a dict of NumPy array if the feature type is `Translation`/`TranslationVariableLanguages` (one array for each language) to get the official PyTorch error message for strings in such case.

## Describe the bug

When using `.with_format("torch")`, an arrow table is returned and I am unable to use it by passing it to a PyTorch DataLoader: please see the code below.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset(
    "para_crawl",
    name="enfr",
    cache_dir="/tmp/test/",
    split="train",
    keep_in_memory=True,
)

dataloader = DataLoader(ds.with_format("torch"), num_workers=32)
print(next(iter(dataloader)))
```

Is there something I am doing wrong? The documentation does not say much about the behavior of `.with_format()` so I feel like I am a bit stuck here :-/

Thanks in advance for your help!

## Expected results

The code should run with no error

## Actual results

```
AttributeError: 'str' object has no attribute 'dtype'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Issue loading datasets -- pyarrow.lib has no attribute

Something isn't working

Hi @margotwagner, thanks for reporting.

Unfortunately, I'm not able to reproduce your bug: in an environment with datasets-2.3.2 and pyarrow-8.0.0, I can load the datasets without any problem:
```python
>>> ds = load_dataset("glue", "cola")
>>> ds
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 8551
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1043
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1063
    })
})

>>> import pyarrow
>>> pyarrow.__version__
8.0.0
>>> from pyarrow.lib import IpcReadOptions
>>> IpcReadOptions
pyarrow.lib.IpcReadOptions
```

I think you may have a problem in your Python environment: maybe you have also an old version of pyarrow that has precedence when importing it.

Could you please check this (just after you tried to load the dataset and got the error)?
```python
>>> import pyarrow
>>> pyarrow.__version__
```

## Describe the bug
I am trying to load sentiment analysis datasets from huggingface, but any dataset I try to use via load_dataset, I get the same error:
`AttributeError: module 'pyarrow.lib' has no attribute 'IpcReadOptions'`

## Steps to reproduce the bug
```python
dataset = load_dataset("glue", "cola")
```

## Expected results
Download datasets without issue.

## Actual results
`AttributeError: module 'pyarrow.lib' has no attribute 'IpcReadOptions'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 8.0.0
- Pandas version: 1.1.0


load_datasets on csv returns everything as a string

Something isn't working

Hi @courtneysprouse, thanks for reporting.

Yes, you are right: by default the "csv" loader loads all columns as strings.

You could tweak this behavior by passing the `feature` argument to `load_dataset`, but it is also true that currently it is not possible to perform some kind of casts, due to lacking of implementation in PyArrow. For example:
```python
import datasets

features = datasets.Features(
    {
        "tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(datasets.Value("int32")),
    }
)

new_conll = datasets.load_dataset("csv", data_files="ner_conll.csv", features=features)
```
gives `ArrowNotImplementedError` error:
```
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from string to list using function cast_list
```

On the other hand, if you just would like to save and afterwards load your dataset, you could use `save_to_disk` and `load_from_disk` instead. These functions preserve all data types.
```python
>>> orig_conll.save_to_disk("ner_conll")

>>> from datasets import load_from_disk

>>> new_conll = load_from_disk("ner_conll")
>>> new_conll
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 14042
    })
    validation: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3251
    })
    test: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3454
    })
})
>>> new_conll["train"][0]
{'chunk_tags': [11, 21, 11, 12, 21, 22, 11, 12, 0],
 'id': '0',
 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0],
 'pos_tags': [22, 42, 16, 21, 35, 37, 16, 21, 7],
 'tokens': ['EU',
  'rejects',
  'German',
  'call',
  'to',
  'boycott',
  'British',
  'lamb',
  '.']}
>>> new_conll["train"].features
{'chunk_tags': Sequence(feature=ClassLabel(num_classes=23, names=['O', 'B-ADJP', 'I-ADJP', 'B-ADVP', 'I-ADVP', 'B-CONJP', 'I-CONJP', 'B-INTJ', 'I-INTJ', 'B-LST', 'I-LST', 'B-NP', 'I-NP', 'B-PP', 'I-PP', 'B-PRT', 'I-PRT', 'B-SBAR', 'I-SBAR', 'B-UCP', 'I-UCP', 'B-VP', 'I-VP'], id=None), length=-1, id=None),
 'id': Value(dtype='string', id=None),
 'ner_tags': Sequence(feature=ClassLabel(num_classes=9, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], id=None), length=-1, id=None),
 'pos_tags': Sequence(feature=ClassLabel(num_classes=47, names=['"', "''", '#', '$', '(', ')', ',', '.', ':', '``', 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB'], id=None), length=-1, id=None),
 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```

Hi @albertvillanova!

Thanks so much for your suggestions! That worked!

## Describe the bug
If you use:

`conll_dataset.to_csv("ner_conll.csv")`

It will create a csv file with all of your data as expected, however when you load it with:

`conll_dataset = load_dataset("csv", data_files="ner_conll.csv")`

everything is read in as a string. For example if I look at everything in 'ner_tags' I get back `['[3 0 7 0 0 0 7 0 0]', '[1 2]', '[5 0]']` instead of what I originally saved which was `[[3, 0, 7, 0, 0, 0, 7, 0, 0], [1, 2], [5, 0]]`

I think maybe there is something funky going on with the csv delimiter

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
#load original conll dataset
orig_conll = load_dataset("conll2003")

#save original conll as a csv
orig_conll.to_csv("ner_conll.csv")

#reload conll data as a csv
new_conll = load_dataset("csv", data_files="ner_conll.csv")`
```

## Expected results
A clear and concise description of the expected results.
I would expect the data be returned as the data type I saved it as. I.e. if I save a list of ints
[[3, 0, 7, 0, 0, 0, 7, 0, 0]], I shouldnt get back a string ['[3 0 7 0 0 0 7 0 0]']
I also get back a string when I pass a list of strings ['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.']

## Actual results
A list of strings `['[3 0 7 0 0 0 7 0 0]', '[1 2]', '[5 0]']`
A string "['EU' 'rejects' 'German' 'call' 'to' 'boycott' 'British' 'lamb' '.']"

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0


Support extract 7-zip compressed data files

_The documentation is not available anymore as the PR was closed or merged._

Cool! Can you please remove `Fix #3541` from the description as this PR doesn't add support for streaming/`iter_archive`, so it only partially addresses the issue?

Side note:
I think we can use `libarchive` (`libarchive-c` is a Python package with the bindings) for streaming 7z archives. The only issue with this lib is that it's tricky to install on Windows/Mac.

Fix partially #3541, fix #4670.


Dataset Viewer issue for wmt16

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting, @lewtun.

~We can't load the dataset locally, so I think this is an issue with the loading script (not the viewer).~

 We are investigating...

Recently, there was a merged PR related to this dataset:
- #4554

We are looking at this...

Indeed, the above mentioned PR fixed the loading script (it was not working before).

I'm forcing the refresh of the Viewer.

Please note that the above mentioned PR also made an enhancement in the `datasets` library, required by this loading script. This enhancement will only be available to the Viewer once we make our next release.

### Link

https://huggingface.co/datasets/wmt16

### Description

[Reported](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/12#62cb83f14c7f35284e796f9c) by a user of AutoTrain Evaluate. AFAIK this dataset was working 1-2 weeks ago, and I'm not sure how to interpret this error.

```
Status code:   400
Exception:     NotImplementedError
Message:       This is a abstract method
```

Thanks!

### Owner

No


Can't extract files from `.7z` zipfile using `download_and_extract`

bug

Something isn't working

Hi @bhavitvyamalik, thanks for reporting.

Yes, currently we do not support 7zip archive compression: I think we should.

As a workaround, you could uncompress it explicitly, like done in e.g. `samsum` dataset:

https://github.com/huggingface/datasets/blob/fedf891a08bfc77041d575fad6c26091bc0fce52/datasets/samsum/samsum.py#L106-L110

Related to this issue: https://github.com/huggingface/datasets/issues/3541

Sure, let me look into and check what can be done. Will keep you guys updated here!

Initially, I thought of solving this without any external dependency. Almost everywhere I saw `lzma` can be used for this but there is a caveat that lzma doesn’t work with 7z archives but only single files. In my case the 7z archive has multiple files so it didn't work. Is it fine to use external library here?

Hi @bhavitvyamalik, thanks for your investigation.

On Monday, I started a PR that will eventually close this issue as well: I'm linking it to this.
- #4672

Let me know what you think.

## Describe the bug
I'm adding a new dataset which is a `.7z` zip file in Google drive and contains 3 json files inside. I'm able to download the data files using `download_and_extract` but after downloading it throws this error:
```
>>> dataset = load_dataset("./datasets/mantis/")
Using custom data configuration default
Downloading and preparing dataset mantis/default to /Users/bhavitvyamalik/.cache/huggingface/datasets/mantis/default/1.1.0/611affa804ec53e2055a335cc1b8b213bb5a0b5142d919967729d5ee23c6bab4...
Downloading data: 100%|█████████████████████████████████████████████████████████| 77.2M/77.2M [00:23<00:00, 3.28MB/s]
/Users/bhavitvyamalik/.cache/huggingface/datasets/downloads/fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/load.py", line 1745, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 20] Not a directory: '/Users/bhavitvyamalik/.cache/huggingface/datasets/downloads/fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6/merged_train.json'
```
just before generating the splits. I checked `fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6` file and it's `7z` zip file (similar to downloaded Google drive file) which means it didn't get unzip. Do I need to unzip it separately and then pass the paths for train,dev,test files in `SplitGenerator`?

## Environment info
- `datasets` version: 1.18.4.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.8
- PyArrow version: 5.0.0


loading oscar-corpus/OSCAR-2201  raises an error

Something isn't working

I had to use the appropriate token for use_auth_token. Thank you.

## Describe the bug
load_dataset('oscar-2201', 'af')

raises an error:
Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "..python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1189, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at .../oscar-2201/oscar-2201.py or any data file in the same directory. Couldn't find 'oscar-2201' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/oscar-2201/oscar-2201.py

I've tried other permutations such as :
oscar_22 = load_dataset('oscar-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-2201', 'af')
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201')

with the same unfortunate result.

## Steps to reproduce the bug
oscar_22 = load_dataset('oscar-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-2201', 'af')
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201')
# Sample code to reproduce the bug
```

## Expected results
loaded data

## Actual results
Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "..python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1189, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at .../oscar-2201/oscar-2201.py or any data file in the same directory. Couldn't find 'oscar-2201' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/oscar-2201/oscar-2201.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.13.0-37-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset Viewer issue for hungnm/multilingual-amazon-review-sentiment-processed

It seems like a private dataset. The viewer is currently not supported on the private datasets.

### Link

https://huggingface.co/hungnm/multilingual-amazon-review-sentiment

### Description

_No response_

### Owner

Yes


duplicate

This issue or pull request already exists

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Issues with concatenating datasets

bug

Something isn't working

Hi! I agree we should improve the features equality checks to account for this particular case. However, your code fails due to `answer_start` having the dtype `int64` instead of `int32` after loading from JSON (it's not possible to embed type precision info into a JSON file; `save_to_disk` does that for arrow files), which would lead to the concatenation error as PyArrow does not support this sort of type promotion. This can be fixed as follows:
```python
temp = load_dataset("json", data_files={"train": "output.jsonl"}, features=squad["train"].features)
```

That makes sense. I totally missed the `int64` and `int32` part. Thanks for pointing it out! Will close this issue for now.

## Describe the bug
It is impossible to concatenate datasets if a feature is sequence of dict in one dataset and a dict of sequence in another. But based on the document, it should be automatically converted.

> A [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence) with a internal dictionary feature will be automatically converted into a dictionary of lists. This behavior is implemented to have a compatilbity layer with the TensorFlow Datasets library but may be un-wanted in some cases. If you don’t want this behavior, you can use a python list instead of the [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence).

## Steps to reproduce the bug
```python
from datasets import concatenate_datasets, load_dataset

squad = load_dataset("squad_v2")
squad["train"].to_json("output.jsonl", lines=True)

temp = load_dataset("json", data_files={"train": "output.jsonl"})
concatenate_datasets([temp["train"], squad["train"]])
```

## Expected results
No error executing that code

## Actual results
```
ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'text': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)} or Value("null").
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.8.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Unable to create dataset having Python dataset script only

Something isn't working

Hi @aleSuglia, thanks for reporting.

We are having a look at it.

We transfer this issue to the Community tab of the corresponding Hub dataset: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/discussions

## Describe the bug
Hi there,

I'm trying to add the following dataset to Huggingface datasets: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/blob/

I'm trying to do so using the CLI commands but seems that this command generates the wrong `dataset_info.json` file (you can find it in the repo already):
```
datasets-cli test Heriot-WattUniversity/dialog-babi/dialog_babi.py --save_infos --all-configs
```
while it errors when I remove the python script:
```
datasets-cli test Heriot-WattUniversity/dialog-babi/ --save_infos --all-configs
```
The error message is the following:
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /Users/as2180/workspace/Heriot-WattUniversity/dialog-babi with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add stanford dog dataset

Hi @khushmeeet, thanks for your contribution.

But wouldn't it be better to add this dataset to the Hub?
- https://huggingface.co/docs/datasets/share
- https://huggingface.co/docs/datasets/dataset_script

Hi @albertvillanova

Dataset is added to Hub - https://huggingface.co/datasets/dgrnd4/stanford_dog_dataset

Great, so I guess we can close this issue, as the dataset is already available on the Hub.

OK I read the discussion on:
- #4504

Currently, priority is adding datasets to the Hub, not here on GitHub.

If you would like to contribute the loading script and all the metadata you generated (README + JSON files), you could:
- Either make a PR to the existing dataset on the Hub
- Create a new dataset on the Hub:
  - Either under your personal namespace
  - or even more professionally, under the namespace `stanfordSVL` (Stanford Vision and Learning Lab: https://svl.stanford.edu/)

You can use the Community tab to ping us if you need help or have any questions.

This PR is for adding dataset, related to issue #4504.

We are adding Stanford dog breed dataset. It is a multi class image classification dataset.
Details can be found here - http://vision.stanford.edu/aditya86/ImageNetDogs/

Tests on dummy data is failing currently, which I am looking into.


Add text decorators

_The documentation is not available anymore as the PR was closed or merged._

This PR adds some decoration to text about different modalities to make it more obvious separate guides exist for audio, vision, and text. The goal is to make it easier for users to discover these guides!

![underline](https://user-images.githubusercontent.com/59462357/178044392-9596693e-9a4a-479a-a282-f1edbd90be1a.png)

TODO:

- [x] Open PR to support new Tailwind classes


Fix: conll2003 - fix empty example

As reported in https://huggingface.co/datasets/conll2003/discussions/2#62c45a14f93fc97e8260532f, there was an extra empty example at the end of the dataset


Concurrency bug when using same cache among several jobs

Something isn't working

I can confirm that if I run one job first that processes the dataset, then I can run any jobs in parallel with no problem (no write-concurrency anymore...).

Hi! That's weird. It seems like the error points to the `mkstemp` function, but the official docs state the following:
```
There are no race conditions in the file’s creation, assuming that the platform properly implements the [os.O_EXCL](https://docs.python.org/3/library/os.html#os.O_EXCL) flag for [os.open()](https://docs.python.org/3/library/os.html#os.open)
```
So this could mean your platform doesn't support that flag.

~~Can you please check if wrapping the temp file creation (the line `tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)` in `_map_single`) with the `multiprocess.Lock` fixes the issue?~~
Perhaps wrapping the temp file creation in `_map_single`  with `filelock` could work:
```python
with FileLock(lock_path):
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
```
Can you please check if that helps?

## Describe the bug
I used to see this bug with an older version of the datasets. It seems to persist.

This is my concrete scenario: I launch several evaluation jobs on a cluster in which I share the file system and I share the cache directory used by huggingface libraries. The evaluation jobs read the same *.csv files. If my jobs get all scheduled pretty much at the same time, there are all kinds of weird concurrency errors. Sometime it crashes silently. This time I got lucky that it crashed with a stack trace that I can share and maybe you get to the bottom of this. If you don't have a similar setup available, it may be hard to reproduce as you really need two jobs accessing the same file at the same time to see this type of bug.

## Steps to reproduce the bug
I'm running a modified version of `run_glue.py` script adapted to my use case. I've seen the same problem when running some glue datasets as well (so it's not specific to loading the datasets from csv files).

## Expected results
No crash, concurrent access to the (intermediate) files just fine.

## Actual results
Crashes due to races/concurrency bugs.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 8.0.0
- Pandas version: 1.1.0

Stack trace that I just got with the crash (I've obfuscated some names, it should still be quite informative):

```
Running tokenizer on dataset:   0%|          | 0/3 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "../../src/models//run_*******.py", line 600, in <module>
    main()
  File "../../src/models//run_*******.py", line 444, in main
    raw_datasets = raw_datasets.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 770, in map
    {
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 771, in <dictcomp>
    k: dataset.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2376, in map
    return self._map_single(
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 551, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2776, in _map_single
    buf_writer, writer, tmp_file = init_buffer_and_writer()
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2696, in init_buffer_and_writer
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 541, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 250, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '/*******/cache-transformers//transformers/csv/default-ef9cd184210742a7/0.0.0/51cce309a08df9c4d82ffd9363bbe090bf173197fc01a71b034e8594995a1a58/tmps8l6j5yc'
```

As I ran 100s of experiments last year for an empirical paper, I ran into this type of bugs several times. I found several bandaid/work-arounds, e.g., run one job first that caches the dataset => eliminate concurrency; OR use unique caches => eliminate concurrency (but increase storage space), etc. and it all works fine.

I'd like to help you fixing this bug as it's really annoying to always apply the work arounds. Let me know what other info from my side could help you figure out the issue.

Thanks for your help!


Fix _resolve_single_pattern_locally on Windows with multiple drives

Good catch ! Sorry I forgot (again) about windows paths when writing this x)

Currently, when `_resolve_single_pattern_locally` is called from a different drive than the one in `pattern`, it raises an exception:
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\io\parquet.py:35: in __init__
    **kwargs,
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\builder.py:287: in __init__
    sanitize_patterns(data_files), base_path=base_path, use_auth_token=use_auth_token
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:761: in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:723: in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:321: in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:239: in _resolve_single_pattern_locally
    for filepath in glob_iter
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:242: in <listcomp>
    os.path.relpath(filepath, base_path), os.path.relpath(pattern, base_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\popen-gw0\\data6\\dataset.parquet'
start = '/'

...

E                   ValueError: path is on mount 'C:', start on mount 'D:'
```

This PR makes sure that `base_path` is in the same drive as `pattern`.


Transfer CI to GitHub Actions

Thanks a lot @albertvillanova ! I hope we're finally done with flakiness on windows ^^

Also thanks for paying extra attention to billing and avoiding running unnecessary jobs. Though for certain aspects (see my comments), I think it's worth having the extra jobs to make our life easier

~@lhoestq I think you forgot to add your comments?~

I had missed it among all the other comments...

@lhoestq, I'm specially enthusiastic with the fail-fast policy: it was in my TODO list for a long time. I really think it will have a positive impact (I would love to know the spent time saving it will enable, besides the carbon footprint reduction). :wink:

So yes, as you said above, let's give it a try at least. If we encounter any inconvenience, we can easily disable it.

Question: I guess I have to disable CircleCI CI before merging this PR?

This PR transfers CI from CircleCI to GitHub Actions. The implementation in GitHub Actions tries to be as faithful as possible to the implementation in CircleCI and get the same output results (exceptions below).

**IMPORTANT NOTE**: The fast-fail policy (described below) is not finally implemented, so that:
- we can continue merging PRs with CI in red because of some random error returned by the Hub
- it is not annoying for maintainers to have to relaunch failed CI jobs

See comments here: https://github.com/huggingface/datasets/pull/4659#discussion_r918802348

Differences in the implementation in GitHub Actions compared to the CircleCI one:
- This PR introduces some *fail-fast* mechanisms to significantly reduce the total time CI is running, both because of environmental impact and because CI in GitHub Actions billing depends on the minutes per month running time (see [About billing for GitHub Actions](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions)):
  - All tests *depend* on `check_code_quality` job: only if `check_code_quality` passes, then the other test jobs are launched
  - The tests are implemented with a matrix strategy (cross-product: OS and PyArrow versions) and fail-fast: if any of the 4 processes fails, the others are cancelled
- OS dependencies for Linux (see table below)

   | OS dependencies | Passed tests | Skipped tests |
   | --- | ---: | ---: |
   | libsndfile1-dev | 4786 | 3119 |
   | libsndfile1 | 4786 | 3119 |
   | libsndfile1, sox | 4788 | 3117 |

  - This PR replaces `libsndfile1-dev` with `libsndfile1`: the same number of passing tests but less packages installed
  - This PR adds `sox`: required by MP3 tests (2 more tests are passed: 4788 instead of 4786)
- For tests using PyArrow 6, this PR uses 6.0.1 instead of 6.0.0

TO DO:
- [ ] Remove old CircleCI CI: kept for the moment to compare stability and performance

Close #4658.

## Comparison between CircleCI and GitHub Actions

| | | CircleCI | GitHub Actions |
| --- | --- | ---: | ---: |
| Ubuntu, pyarrow-latest ||||
|| Passed tests | 4786 | 4788 |
|| Duration | 11m 0s | 10m 10s |
| Windows, pyarrow-latest ||||
|| Passed tests | 4783 | 4783 |
|| Duration | 29m 59s | 22m 56s |


Transfer CI tests to GitHub Actions

Let's try CI tests using GitHub Actions to see if they are more stable than on CircleCI.


Add SQuAD2.0 Dataset

dataset request

Requesting to add a new dataset

Hey, It's already present [here](https://huggingface.co/datasets/squad_v2)

Hi! This dataset is indeed already available on the Hub. Closing.

## Adding a Dataset
- **Name:** *SQuAD2.0*
- **Description:** *Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.*
- **Paper:** *https://aclanthology.org/P18-2124.pdf*
- **Data:** *https://rajpurkar.github.io/SQuAD-explorer/*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Amazon-QA Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Amazon-QA*
- **Description:** *The dataset is .jsonl format, where each line in the file is a json string that corresponds to a question, existing answers to the question and the extracted review snippets (relevant to the question).*
- **Paper:** *https://github.com/amazonqa/amazonqa/tree/master/paper*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/amazon-qa.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Simple Wikipedia

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Simple Wikipedia*
- **Description:** *Two different versions of the data set now exist. Both were generated by aligning Simple English Wikipedia and English Wikipedia. A complete description of the extraction process can be found in "Simple English Wikipedia: A New Simplification Task", William Coster and David Kauchak (2011).*
- **Paper:** *https://aclanthology.org/P11-2117/*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/SimpleWiki.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Quora Question Triplets Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Quora Question Triplets*
- **Description:** *This dataset consists of over 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair.*
- **Paper:**
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/quora_duplicates_triplets.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Altlex dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Altlex*
- **Description:** *Git repository for software associated with the 2016 ACL paper "Identifying Causal Relations Using Parallel Wikipedia Articles.”*
- **Paper:** *https://aclanthology.org/P16-1135.pdf*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/altlex.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Sentence Compression Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Sentence Compression*
- **Description:** *Large corpus of uncompressed and compressed sentences from news articles.*
- **Paper:** *https://www.aclweb.org/anthology/D13-1155/*
- **Data:** *https://github.com/google-research-datasets/sentence-compression/tree/master/data*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Flickr 30k Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Flickr 30k*
- **Description:** *To produce the denotation graph, we have created an image caption corpus consisting of 158,915 crowd-sourced captions describing 31,783 images. This is an extension of our previous Flickr 8k Dataset. The new images and captions focus on people involved in everyday activities and events.*
- **Paper:** *https://transacl.org/ojs/index.php/tacl/article/view/229/33*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/flickr30k_captions.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add SPECTER dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *SPECTER*
- **Description:** *SPECTER: Document-level Representation Learning using Citation-informed Transformers*
- **Paper:** *https://doi.org/10.18653/v1/2020.acl-main.207*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/specter_train_triples.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add PAQ dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *PAQ*
- **Description:** *This repository contains code and models to support the research paper PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them*
- **Paper:** *https://arxiv.org/abs/2102.07033*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/PAQ_pairs.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add WikiAnswers dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *WikiAnswers*
- **Description:** *The WikiAnswers corpus contains clusters of questions tagged by WikiAnswers users as paraphrases. Each cluster optionally contains an answer provided by WikiAnswers users.*
- **Paper:** *https://dl.acm.org/doi/10.1145/2623330.2623677*
- **Data:** *https://github.com/afader/oqa#wikianswers-corpus*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Reddit dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Reddit comments (2015-2018)*
- **Description:** *Reddit is an American social news aggregation website, where users can post links, and take part in discussions on these posts. These threaded discussions provide a large corpus, which is converted into a conversational dataset using the tools in this directory.*
- **Paper:** *https://arxiv.org/abs/1904.06472*
- **Data:** *https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Set HF_SCRIPTS_VERSION to main

After renaming "master" to "main", the CI fails with
```
AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/main/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at /home/circleci/datasets/_dummy/_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"
```

This is because in the CI we were still using `HF_SCRIPTS_VERSION=master`. I changed it to "main"


[Minor fix] Typo correction

recieve -> receive


Rename master to main

All the mentions I found on google were simple URLs that will be redirected, so it's fine. I also checked the spaces and we should be good:
- dalle-mini used to install the master branch but [it's no longer the case](https://huggingface.co/spaces/flax-community/dalle-mini/commit/b78c972afd5c2d2bed087be6479fe5c9c6cfa741)
- same for [logo generator](https://huggingface.co/spaces/tom-doerr/logo_generator/commit/a9ea330e518870d0ca8f65abb56f71d86750d8e4)
- I opened a PR to fix [vision-datasets-viewer](https://huggingface.co/spaces/nateraw/vision-datasets-viewer/discussions/1)

Ok let's rename the branch, and then we can merge this PR

This PR renames mentions of "master" by "main" in the code base for several cases:
- set the default dataset script version to "main" if the local installation of `datasets` is a dev installation
- update URLs to this github repository to use "main"
- update the DVC benchmark
- update the github workflows
- update docstrings
- update tests to compare the changes in dataset cards against "main"


Streaming issue for ccdv/pubmed-summarization

Thanks for reporting @lewtun.

I confirm there is an issue with streaming: it does not stream locally.

Oh, after investigation, the source of the issue is in the Hub dataset loading script.

I'm opening a PR on the Hub dataset.

I've opened a PR on their Hub dataset to support streaming: https://huggingface.co/datasets/ccdv/pubmed-summarization/discussions/2

### Link

https://huggingface.co/datasets/ccdv/pubmed-summarization

### Description

This was reported by a [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/7). It seems like streaming doesn't work due to the way the dataset loading script is defined?

```
Status code:   400
Exception:     FileNotFoundError
Message:       https://huggingface.co/datasets/ccdv/pubmed-summarization/resolve/main/train.zip/train.txt
```

### Owner

No


Dataset Viewer issue for kmfoda/booksum

Thanks for reporting, @lewtun.

It works locally in streaming mode:
```
{'bid': 27681,
 'is_aggregate': True,
 'source': 'cliffnotes',
 'chapter_path': 'all_chapterized_books/27681-chapters/chapters_1_to_2.txt',
 'summary_path': 'finished_summaries/cliffnotes/The Last of the Mohicans/section_1_part_0.txt',
 'book_id': 'The Last of the Mohicans.chapters 1-2',
 'summary_id': 'chapters 1-2',
 'content': None,
 'summary': '{"name": "Chapters 1-2", "url": "https://web.archive.org/web/20201101053205/https://www.cliffsnotes.com/literature/l/the-last-of-the-mohicans/summary-and-analysis/chapters-12", "summary": "Before any characters appear, the time and geography are made clear. Though it is the last war that England and France waged for a country that neither would retain, the wilderness between the forces still has to be...
```

I'm forcing the refresh of the preview.

The preview appears as expected once the refresh forced.

### Link

https://huggingface.co/datasets/kmfoda/booksum

### Description

A [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/9) discovered this dataset cannot be streamed due to:

```
Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/kmfoda/booksum/resolve/47953f583d6967f086cb16a2f4d2346e9834024d/test.csv')
```

I'm not sure why it says "Unauthorized" since it's just a bunch of CSV files in a repo

### Owner

No


Support all split in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4640). All of your documentation changes will be reflected on that endpoint.

Fix #4637.


Add HaGRID -- HAnd Gesture Recognition Image Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** HaGRID -- HAnd Gesture Recognition Image Dataset
- **Description:** We introduce a large image dataset HaGRID (HAnd Gesture Recognition Image Dataset) for hand gesture recognition (HGR) systems. You can use it for image classification or image detection tasks. Proposed dataset allows to build HGR systems, which can be used in video conferencing services (Zoom, Skype, Discord, Jazz etc.), home automation systems, the automotive sector, etc.
- **Paper:** https://arxiv.org/abs/2206.08219
- **Data:** https://github.com/hukenovs/hagrid

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


The speechocean762 dataset

CircleCL reported two errors, but I didn't find the reason. The error message:
```
_________________ ERROR collecting tests/test_dataset_cards.py _________________
tests/test_dataset_cards.py:53: in <module>
    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
tests/test_dataset_cards.py:35: in get_changed_datasets
    diff_output = check_output(["git", "diff", "--name-only", "origin/master...HEAD"], cwd=repo_path)
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:356: in check_output
    **kwargs).stdout
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:438: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '['git', 'diff', '--name-only', 'origin/master...HEAD']' returned non-zero exit status 128.

=========================== short test summary info ============================
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
= 4011 passed, 2357 skipped, 2 xfailed, 1 xpassed, 116 warnings, 2 errors in 284.32s (0:04:44) =

Exited with code exit status 1
```
I'm not sure if it was caused by this PR ...

I ran `tests/test_dataset_cards.py` in my local environment, and it passed:
```
(venv)$ pytest tests/test_dataset_cards.py
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/zhangjunbo/src/datasets
plugins: forked-1.4.0, datadir-1.3.1, xdist-2.5.0
collected 1531 items

tests/test_dataset_cards.py ..... [100%]
======================= 766 passed, 765 skipped in 2.55s ========================
```

[speechocean762](https://www.openslr.org/101/) is a non-native English corpus for pronunciation scoring tasks. It is free for both commercial and non-commercial use.

I believe it will be easier to use if it could be available on Hugging Face.


The "all" split breaks streaming

bug

Something isn't working

Thanks for reporting @cakiki.

Yes, this is a bug. We are investigating it.

@albertvillanova Nice! Let me know if it's something I can fix my self; would love to contribtue!

@cakiki I was working on this but if you would like to contribute, go ahead. I will close my PR. ;)

For the moment I just pushed the test (to see if it impacts other tests).

It impacted the test `test_generator_based_download_and_prepare` and I have fixed this.

So that you can copy the test I implemented in my PR and then implement a fix for this issue that passes the test `tests/test_builder.py::test_builder_as_streaming_dataset`.

Hi @cakiki are you still interested in working on this? Are you planning to open a PR?

Hi @albertvillanova ! Sorry it took so long; I wanted to spend this weekend working on it.

## Describe the bug

Not sure if this is a bug or just the way streaming works, but setting `streaming=True` did not work when setting `split="all"`

## Steps to reproduce the bug
The following works:
```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all')
```
The following throws `ValueError: Bad split: all. Available splits: ['train', 'validation', 'test']`:

```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all', streaming=True)
```

## Expected results
An iterator over all splits.

## Actual results
I had to do the following to achieve the desired result:
```python
from itertools import chain
ds = load_dataset('super_glue', 'wsc.fixed', streaming=True)
it =  chain.from_iterable(ds.values())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add info in docs about behavior of download_config.num_proc

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

I went to override `download_config.num_proc` and was confused about what was happening under the hood. It would be nice to have the behavior documented a bit better so folks know what's happening when they use it.

**Describe the solution you'd like**

- Add note about how the default number of workers is 16. Related code:

https://github.com/huggingface/datasets/blob/7bcac0a6a0fc367cc068f184fa132b8de8dfa11d/src/datasets/download/download_manager.py#L299-L302

- Add note that if the number of workers is higher than the number of files to download, it won't use multiprocessing.

**Describe alternatives you've considered**

maybe it would also be nice to set `num_proc` = `num_files` when `num_proc` > `num_files`.

**Additional context**

...


Dataset Viewer issue for vadis/sv-ident

Thanks for reporting, @e-tornike

Some context:
- #4527

The dataset loads locally in streaming mode:
```python
In [2]: from datasets import load_dataset; ds = load_dataset("vadis/sv-ident", split="validation", streaming=True); item = next(iter(ds)); item
Using custom data configuration default
Out[2]:
{'sentence': 'Im Falle von Umweltbelastungen kann selten eindeutig entschieden werden, ob Unbedenklichkeitswerte bereits erreicht oder überschritten sind, die die menschliche Gesundheit oder andere Wohlfahrts»güter« beeinträchtigen.',
 'is_variable': 0,
 'variable': [],
 'research_data': [],
 'doc_id': '51971',
 'uuid': 'ee3d7f88-1a3e-4a59-997f-e986b544a604',
 'lang': 'de'}
```

~~I have forced the refresh of the split in the preview without success.~~

I have forced the refresh of the split in the preview, and now it works.

Preview seems to work now.

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

OK, thank you @e-tornike.

Apparently, after forcing the refresh, we just had to wait a little until it is effectively refreshed.

I'm closing this issue as it was solved after forcing the refresh of the split in the preview.

### Link

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

### Description

Error message when loading validation split in the viewer:

```
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

### Owner

_No response_


Can't load the Hausa audio dataset

Could you provide the error details. It is difficult to debug otherwise. Also try other config. `ha` is not a valid.

common_voice_train = load_dataset("common_voice", "ha", split="train+validation")


[data_files] Only match separated split names

_The documentation is not available anymore as the PR was closed or merged._

I ran a script to find affected datasets (just did it on non-private non-gated). Adding "testing" and "evaluation" fixes all of of them except one:
- projecte-aina/cat_manynames:	human_annotated_testset.tsv

Let me open a PR on their repository to fix it
EDIT: pr [here](https://huggingface.co/datasets/projecte-aina/cat_manynames/discussions/2)

Feel free to merge @albertvillanova if it's all good to you :)

Thanks for the feedback @albertvillanova  I took your comments into account :)
- added numbers as supported delimiters
- used list comprehension to create the patterns list
- updated the docs and the tests according to your comments

Let me know what you think !

I ended up removing the patching and the context manager :) merging

As reported in https://github.com/huggingface/datasets/issues/4477, the current pattern matching to infer which file goes into which split is too permissive. For example a file "contest.py" would be considered part of a test split (it contains "test") and "seqeval.py" as well (it contains "eval").

In this PR I made the pattern matching more robust by only matching split names **between separators**. The supported separators are dots, dashes, spaces and underscores.

I updated the docs accordingly.

One detail about the tests: I had to update one test because it was using `PurePath.match` as a reference for globbing, but it doesn't support the `[..]` glob pattern. Therefore I added a `mock_fs` context manager that can be used to easily define a dummy filesystem with certain files in it and run pattern matching tests. Its code comes mostly from test_streaming_download_manager.py

Close https://github.com/huggingface/datasets/issues/4477


'sort' method sorts one column only

Hi ! `ds.sort()` does sort the full dataset, not just one column:
```python
from datasets import *

ds = Dataset.from_dict({"foo": [3, 2, 1], "bar": ["c", "b", "a"]})
print(d.sort("foo").to_pandas()
#    foo bar
# 0    1   a
# 1    2   b
# 2    3   c
```

What made you think it was not the case ? Did you experience a situation where it was only sorting one column ?

Hi! thank you for your quick reply!
I wanted to sort the `cnn_dailymail` dataset by the length of the labels (num of characters). I added a new column to the dataset (`ds.add_column`) with the lengths and then sorted by this new column. Only the new length column was sorted, the reset left in their original order.

That's unexpected, can you share the code you used to get this ?

The 'sort' method changes the order of one column only (the one defined by the argument 'column'), thus creating a mismatch between a sample fields. I would expect it to change the order of the samples as a whole, based on the 'column' order.


Update WinoBias README

I'm adding some information about Winobias that I got from the paper :smile:

I think this makes it a bit clearer!


fix(dataset_wrappers): Fixes access to fsspec.asyn in torch_iterable_dataset.py.

Fix #4612.

Apparently, newest `fsspec` versions do not allow access to attribute-based modules if they are not imported, such as `fsspec.async`.

Thus, @mariosasko suggested to add the missing part to the module import to allow for its access.


Rename repo default branch to main

maintenance

Maintenance tasks

Rename repository default branch to `main` (instead of current `master`).

Once renamed, users will have to manually update their local repos:

- [ ] Upstream:
  ```
  git branch -m master main
  git fetch upstream main
  git branch -u upstream/main main
  git remote set-head upstream -a
  ```

- [ ] Origin:
Rename fork default branch as well at: https://github.com/USERNAME/lam/settings/branches
Then:
  ```
  git fetch origin main
  git remote set-head origin -a
  ```

CC: @sgugger


Fix time type `_arrow_to_datasets_dtype` conversion

Fix #4620

The issue stems from the fact that `pa.array([time_data]).type` returns `DataType(time64[unit])`, which doesn't expose the `unit` attribute, instead of `Time64Type(time64[unit])`. I believe this is a bug in PyArrow. Luckily, the both types have the same `str()`, so in this PR I call `pa.type_for_alias(str(type))` to convert them both to the `Time64Type(time64[unit])` format.

cc @severo


fixed duplicate calculation of spearmanr function in metrics wrapper.

Great, can open a PR in `evaluate` as well to optimize this.

Relatedly, I wanted to add a new metric, Kendall Tau (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html). If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the `datasets` or `evaluate` repo (or both)?

Thanks!

PR opened in`evaluate` library with same minor adjustment: https://github.com/huggingface/evaluate/pull/176

> If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the datasets or evaluate repo (or both)?

I think you could just add it to `evaluate`, we're not adding new metrics in this repo anymore

During _compute, the scipy.stats spearmanr function was called twice, redundantly, once for calculating the score and once for calculating the p-value, under the conditional branch where return_pvalue=True. I adjusted the _compute function to execute the spearmanr function once, store the results tuple in a temporary variable, and then pass the indexed contents to the expected keys of the returned dictionary.


Add non-commercial licensing info for datasets for which we removed tags

yep plus `license_details` also makes sense for this IMO

We removed several YAML tags saying that certain datasets can't be used for commercial purposes: https://github.com/huggingface/datasets/pull/4613#discussion_r911919753

Reason for this is that we only allow tags that are part of our [supported list of licenses](https://github.com/huggingface/datasets/blob/84fc3ad73c85de4eda5d152dfede7671491449cb/src/datasets/utils/resources/standard_licenses.tsv)

We should update the Licensing Information section of the concerned dataset cards, now that the non-commercial tag doesn't exist anymore for certain datasets


Unpack `dl_manager.iter_files` to allow parallization

Cool thanks ! Yup it sounds like the right solution.

It looks like `_generate_tables` needs to be updated as well to fix the CI

Iterate over data files outside `dl_manager.iter_files` to allow parallelization in streaming mode.

(The issue reported [here](https://discuss.huggingface.co/t/dataset-only-have-n-shard-1-when-has-multiple-shards-in-repo/19887))

PS: Another option would be to override `FilesIterable.__getitem__` to make it indexable and check for that type in `_shard_kwargs` and `n_shards,` but IMO this solution adds too much unnecessary complexity.


Remove all paperswithcode_id: null

> We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

@lhoestq maybe it's better to accept it on the Hub side then? Let me know if you want us to do it Hub-side

Yup it's maybe better to support it on the Hub side then indeed, thanks ! Closing this one

On the Hub there is a validation error on the `paperswithcode_id` tag when the value is `null`:

<img width="686" alt="image" src="https://user-images.githubusercontent.com/42851186/177151825-93d341c5-25bd-41ab-96c2-c0b516d51c68.png">

We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

To have the validation working again we can simply remove all the `paperswithcode_id: null`.

cc @julien-c


Loading MNIST as Pytorch Dataset

bug

Something isn't working

Hi ! We haven't implemented the conversion from images data to PyTorch tensors yet I think

cc @mariosasko

So I understand:

set_format() does not properly do the conversion to pytorch tensors from PIL images.

So that someone who stumbles on this can use the package:

```python
dataset = load_dataset("mnist", split="train")
def transform_func(examples):
    examples["image"] = [np.array(img) for img in examples["image"]]
    return examples
dataset = dataset.with_transform(transform_func)
dataset[0]
```

This then appears to work with pytorch dataloaders as:
```
dataloader=torch.utils.data.DataLoader(dataset,batch_size=1)
```

and tensorflow as:
```
dataset=dataset.to_tf_dataset(batch_size=1)
```

Hi! `set_transform`/`with_transform` is indeed the correct solution for the conversion. Improving this part of the API is one of the things I'm working on currently, so stay tuned!

## Describe the bug
Conversion of MNIST dataset to pytorch fails with bug

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("mnist", split="train")
dataset.set_format('torch')
dataset[0]
print()
```

## Expected results
Expect to see torch tensors image and label

## Actual results
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/chapm/PycharmProjects/multiviewdata/multiviewdata/huggingface/mnist.py", line 13, in <module>
    dataset[0]
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2154, in __getitem__
    return self._getitem(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2139, in _getitem
    formatted_output = format_table(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 356, in map_nested
    mapped = [
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 357, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 293, in _single_map_nested
    return function(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
python-BaseException

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Windows-10-10.0.22579-SP0
- Python version: 3.9.2
- PyArrow version: 8.0.0
- Pandas version: 1.4.1


Fix ImageFolder with parameters drop_metadata=True and drop_labels=False (when metadata.jsonl is present)

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq @mariosasko pls take a look at https://github.com/huggingface/datasets/pull/4622/commits/769e4c046a5bd5e3a4dbd09cfad1f4cf60677869. I modified `_generate_examples()` according to the same logic too: removed checking if `metadata_files` are not empty for the case when `self.config.drop_metadata=True` because I think we should be aligned with the config and preserve labels if `self.config.drop_labels=False` (the default value) and `self.config.drop_metadata=True` but `metadata_files` are passed. This is an extremely unlikely use case (when `self.config.drop_metadata=True`, but `metadata_files` are passed to `_generate_examples()`) since users usually do not use `_generate_examples()` alone but I believe it would be consistent to have the same behavior as in `_splits_generators()`. This change requires change in tests too if we suppose that we want to preserve labels (default value of `self.config.drop_labels` is False) when `self.config.drop_metadata=True`, even if `metadata_files` are for some reason provided (as it is done in tests).

wdyt about this change?

@lhoestq it wouldn't raise an error if we check `example.keys() == {"image", "label"}` as test checks only `_generate_examples`, not `encode_example`. and in the implementation of this PR `_generate_examples` would return both `image` and `label` key in the case when `drop_metadata=True` and `drop_labels=False` (default) as it seems that we agreed on that :)

and on the other hand it would raise an error if `label` column is missing in _generate_examples when `drop_metadata=True` and `drop_labels=False`

by "it" i mean tests :D (`test_generate_examples_with_metadata_that_misses_one_image`, `test_generate_examples_with_metadata_in_wrong_location` and `test_generate_examples_drop_metadata`)

Perhaps we could make `self.config.drop_metadata = None` and `self.config.drop_labels = None` the defaults to see explicitly what the user wants. This would then turn into  `self.config.drop_metadata = False` and `self.config.drop_labels = True` if metadata files are present and `self.config.drop_metadata = True` and `self.config.drop_labels = False` if not. And if the user wants to have the `label` column alongside metadata columns, it can do so by passing `drop_labels = False` explicitely (in that scenario we have to check that the `label` column is not already present in metadata files). And maybe we can also improve the logging messages.

I find it problematic that the current implementation drops labels in some scenarios even if `self.config.drop_labels = False`, and the user doesn't have control over this behavior.

Let me know what you think.

Will fix #4621

ImageFolder raises `KeyError: 'label'` with params `drop_metadata=True` and `drop_labels=False` (if there is at least one metadata.jsonl file a data directory). This happens because metadata files are collected inside `analyze()` function regardless of `drop_metadata` value. And then the following condition doesn't pass: https://github.com/huggingface/datasets/blob/master/src/datasets/packaged_modules/imagefolder/imagefolder.py#L167

So I suggest to double check it inside `analyze()` not to collect metadata files if they are not needed. (and labels too, to be consistent)

---
Also, I added a test to check if labels are inferred correctly from directories names in general (because we didn't have it) :)