kardwalker/my_dataset huggingface dataset:
GitHub issue #6185: Error in saving the PIL image into *.arrow files using datasets.arrow_writer
State: open | Created: 2023-08-26T12:15:57Z

### Describe the bug

I am using the ArrowWriter from datasets.arrow_writer to save a json-style file as arrow files. Within the dictionary, it contains a feature called "image" which is a list of PIL.Image objects.
I am saving the json using the following script:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
```
However, when I attempt to restore the dataset and use the ```Dataset.from_file(path)``` function to load the arrow file, there seems to be an issue with the PIL.Image object in the dataset. The list of PIL.Images appears as follows rather than a normal PIL.Image object:
![1693051705440](https://github.com/huggingface/datasets/assets/14247682/03b204c2-d0fa-4d19-beff-6f4d7b83c848)


### Steps to reproduce the bug

1. Storing the data json into arrow files:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
save_to_arrow( path, json_file )
```
2.  try to load the arrow file into the Dataset object using the ```Dataset.from_file(path)```


### Expected behavior

Except to saving the contained "image" feature as a list PIL.Image objects as the arrow file. And I can restore the dataset from the file.

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.4.4
GitHub issue #6184: Map cache does not detect function changes in another module
State: closed | Created: 2023-08-25T22:59:14Z

```python
# dataset.py
import os
import datasets

if not os.path.exists('/tmp/test.json'):
  with open('/tmp/test.json', 'w') as file:
    file.write('[{"text": "hello"}]')

def transform(example):
  text = example['text']
  # text += ' world'
  return {'text': text}

data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
```

```python
# test.py
import dataset
print(next(iter(dataset.data)))
```

Initialize cache
```
python3 test.py
# {'text': 'hello'}
```

Edit dataset.py and uncomment the commented line, run again
```
python3 test.py
# {'text': 'hello'}
# expected: {'text': 'hello world'}
```

Clear cache and run again
```
rm -rf ~/.cache/huggingface/datasets/*
python3 test.py
# {'text': 'hello world'}
```


If instead the two files are combined, then changes to the function are detected correctly. But it's expected when working on any realistic codebase that things will be modularized into separate files.
GitHub issue #6182: Loading Meteor metric in HF evaluate module crashes due to datasets import issue 
State: closed | Created: 2023-08-25T14:54:06Z

### Describe the bug

When using python3.9 and ```evaluate``` module loading Meteor metric crashes at a non-existent import from ```datasets.config``` in ```datasets v2.14```

### Steps to reproduce the bug

```
from evaluate import load
meteor = load("meteor")
```

produces the following error:
  ```
    from datasets.config import importlib_metadata, version
ImportError: cannot import name 'importlib_metadata' from 'datasets.config' (<path_to_project>/venv/lib/python3.9/site-packages/datasets/config.py)
```

### Expected behavior

```datasets``` of v2.10 has the following workaround in ```config.py```:

```
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```

However, it's absent in v2.14 which might be the cause of the issue.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.6
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Evaluate version: 0.4.0
GitHub issue #6179: Map cache with tokenizer
State: open | Created: 2023-08-25T12:55:18Z

Similar issue to https://github.com/huggingface/datasets/issues/5985, but across different sessions rather than two calls in the same session.

Unlike that issue, explicitly calling tokenizer(my_args) before the map() doesn't help, because the tokenizer was created with a different hash to begin with...

setup
```
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained('bert-base-uncased').save_pretrained("tok")
```

this prints different value each time
```
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets
print(hash(dumps(AutoTokenizer.from_pretrained("tok"))))
```
GitHub issue #6178: 'import datasets' throws "invalid syntax error"
State: closed | Created: 2023-08-25T08:35:14Z

### Describe the bug

Hi,

I have been trying to import the datasets library but I keep gtting this error.

`Traceback (most recent call last):

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/IPython/core/interactiveshell.py:3508 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[2], line 1
    import datasets

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/__init__.py:22
    from .arrow_dataset import Dataset

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_dataset.py:67
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_writer.py:27
    from .features import Features, Image, Value

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/__init__.py:17
    from .audio import Audio

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/audio.py:11
    from ..download.streaming_download_manager import xopen, xsplitext

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/__init__.py:10
    from .streaming_download_manager import StreamingDownloadManager

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/streaming_download_manager.py:18
    from aiohttp.client_exceptions import ClientError

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/__init__.py:7
    from .connector import *  # noqa

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/connector.py:12
    from .client import ClientRequest

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/client.py:144
    yield from asyncio.async(resp.release(), loop=loop)
                       ^
SyntaxError: invalid syntax`

I have simply used these commands:
`import datasets`
and
`from datasets import load_dataset`

### Environment info

The library has been installed a virtual machine on JupyterHub. Although I have used this library multiple times (on the same VM) before, to train/test an ASR or other ML models, I had never encountered this error.
GitHub issue #6176: how to limit the size of memory mapped file?
State: open | Created: 2023-08-24T05:33:45Z

### Describe the bug

Huggingface datasets use memory-mapped file to map large datasets in memory for fast access.
However, it seems like huggingface will occupy all the memory for memory-mapped files, which makes a troublesome situation since we cluster will distribute a small portion of memory to me (once it's over the limit, memory cannot be allocated), however, when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 
So is there a way to explicitly limit the size of memory mapped file?

### Steps to reproduce the bug

python
>>> from datasets import load_dataset
>>> dataset = load_dataset("c4", "en", streaming=True)

### Expected behavior

In a normal environment, this will not have any problem.
However, when the system allocates a portion of the memory to the program and when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 

### Environment info

linux cluster with SGE（Sun Grid Engine）
GitHub issue #6175: PyArrow 13 CI fixes
State: closed | Created: 2023-08-23T15:45:53Z

Fixes:
* bumps the PyArrow version check in the `cast_array_to_feature` to avoid the offset bug (still not fixed)
* aligns the Pandas formatting tests with the Numpy ones (the current test fails due to https://github.com/apache/arrow/pull/35656, which requires `.to_pandas(coerce_temporal_nanoseconds=True)` to always return `datetime [ns]` objects)

Fix #6173
GitHub issue #6173: Fix CI for pyarrow 13.0.0
State: closed | Created: 2023-08-23T14:11:20Z

pyarrow 13.0.0 just came out

```
FAILED tests/test_formatting.py::ArrowExtractorTest::test_pandas_extractor - AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[us, UTC]
[right]: datetime64[ns, UTC]
```

```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```

e.g. in https://github.com/huggingface/datasets/actions/runs/5952253963/job/16143847230

first error may be related to https://github.com/apache/arrow/issues/33321

second one maybe because `feature.length * len(array) == len(array_values)` is not satisfied anymore somehow ?
GitHub issue #6172: Make Dataset streaming queries retryable
State: open | Created: 2023-08-23T13:15:38Z

### Feature request

Streaming datasets, as intended, do not load the entire dataset in memory or disk. However, while querying the next data chunk from the remote, sometimes it is possible that the service is down or there might be other issues that may cause the query to fail. In such a scenario, it would be nice to make these queries retryable (perhaps with a backoff strategy).

### Motivation

I was working on a model and the model checkpoints after every 1000 steps. At step 1800 I got a 504 HTTP status code error from Huggingface hub for my pytorch `dataloader`. Given the size of my model and data, it took around 2 hours to reach 1800 steps and now it will take about an hour to recover the lost 800. It would be better to get a retryable querying strategy.

### Your contribution

It would be better if someone having experience in this area takes this up as this would require some testing.
GitHub issue #6170: feat: Return the name of the currently loaded file
State: open | Created: 2023-08-23T07:08:17Z

Added an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

I added this here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

fixes #5806
GitHub issue #6169: Configurations in yaml not working
State: open | Created: 2023-08-23T00:13:22Z

### Dataset configurations cannot be created in YAML/README

Hello! I'm trying to follow the docs here in order to create structure in my dataset as added from here (#5331): https://github.com/huggingface/datasets/blob/8b8e6ee067eb74e7965ca2a6768f15f9398cb7c8/docs/source/repository_structure.mdx#L110-L118

I have the exact example in my config file for [my data repo](https://huggingface.co/datasets/tsor13/test):
```
configs:
- config_name: main_data
  data_files: "main_data.csv"
- config_name: additional_data
  data_files: "additional_data.csv"
```

Yet, I'm unable to load different configurations:
```
from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test', use_auth_token=True)
```
returns a single split, `['tsor13--test']`

Does anyone have any insights?

@polinaeterna thank you for adding this feature, it is super useful. Do you happen to have any ideas?

### Steps to reproduce the bug

from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test')

### Expected behavior

I would expect there to be two splits, `main_data` and `additional_data`. However, only `['tsor13--test']` test is returned.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.1
GitHub issue #6165: Fix multiprocessing with spawn in iterable datasets
State: closed | Created: 2023-08-22T10:07:23Z

The "Spawn" method is preferred when multiprocessing on macOS or Windows systems, instead of the "Fork" method on linux systems.

This causes some methods of Iterable Datasets to break when using a dataloader with more than 0 workers.

I fixed the issue by replacing lambda and local methods which are not pickle-able.

See the example below:

```python
from datasets import load_dataset
from torch.utils.data import DataLoader


if __name__ == "__main__":
    dataset = load_dataset("lhoestq/demo1", split="train")
    dataset = dataset.to_iterable_dataset(num_shards=3)

    dataset = dataset.remove_columns(["package_name"])
    dataset = dataset.rename_columns({
        "review": "review1"
    })
    dataset = dataset.rename_column("date", "date1")
    for sample in DataLoader(dataset, batch_size=None, num_workers=3):
        print(sample)
```

To notice the fix on a linux system, adding these lines should do the trick:

```python
import multiprocessing
multiprocessing.set_start_method('spawn')
```

I also removed what looks like code duplication between rename_colums and rename_column
GitHub issue #6164: Fix: Missing a MetadataConfigs init when the repo has a `datasets_info.json` but no README
State: closed | Created: 2023-08-21T14:57:54Z

When I try to push to an arrow repo (can provide the link on Slack), it uploads the files but fails to update the metadata, with
```
  File "app.py", line 123, in add_new_eval
    eval_results[level].push_to_hub(my_repo, token=TOKEN, split=SPLIT)
  File "blabla_my_env_path/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5501, in push_to_hub
    if not metadata_configs:
UnboundLocalError: local variable 'metadata_configs' referenced before assignment
```

This fixes it.
GitHub issue #6163: Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32
State: open | Created: 2023-08-19T11:34:40Z

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?
GitHub issue #6159: Add `BoundingBox` feature
State: open | Created: 2023-08-17T20:49:51Z

... to make working with object detection datasets easier. Currently, `Sequence(int_or_float, length=4)` can be used to represent this feature optimally (in the storage backend), so I only see this feature being useful if we make it work with the viewer. Also, bounding boxes usually come in 4 different formats (explained [here](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/)), so we need to decide which one to support (or maybe all of them).

cc @NielsRogge @severo
GitHub issue #6156: Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
State: closed | Created: 2023-08-17T10:58:20Z

### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related
GitHub issue #6152: FolderBase Dataset automatically resolves under current directory when data_dir is not specified
State: open | Created: 2023-08-16T04:38:09Z

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue #6151: Faster sorting for single key items
State: closed | Created: 2023-08-15T14:02:31Z

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps. 

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True) 
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```
GitHub issue #6150: Allow dataset implement .take
State: open | Created: 2023-08-15T00:17:51Z

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True. 

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)
GitHub issue #6149: Dataset.from_parquet cannot load subset of columns
State: closed | Created: 2023-08-14T23:28:22Z

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset


pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
GitHub issue #6148: Ignore parallel warning in map_nested
State: closed | Created: 2023-08-14T10:43:41Z

This warning message was shown every time you pass num_proc to `load_dataset` because of `map_nested`

```
parallel_map is experimental and might be subject to breaking changes in the future
```

This PR removes it for `map_nested`. If someone uses another parallel backend they're already warned when `parallel_backend` is called anyway
GitHub issue #6147: ValueError when running BeamBasedBuilder with GCS path in cache_dir
State: closed | Created: 2023-08-14T03:11:34Z

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets


class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3
GitHub issue #6153: custom load dataset to hub
State: closed | Created: 2023-08-13T04:42:22Z

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub
GitHub issue #6140: Misalignment between file format specified in configs metadata YAML and the inferred builder
State: closed | Created: 2023-08-10T15:07:34Z

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
``` 
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna
GitHub issue #6139: Offline dataset viewer
State: closed | Created: 2023-08-10T11:30:00Z

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.
GitHub issue #6138: Ignore CI lint rule violation in Pickler.memoize
State: closed | Created: 2023-08-10T11:03:15Z

This PR ignores the violation of the lint rule E721 in `Pickler.memoize`.

The lint rule violation was introduced in this PR:
- #3182

@lhoestq is there a reason you did not use `isinstance` instead?

As a hotfix, we just ignore the violation of the lint rule.

Fix #6136.
GitHub issue #6136: CI check_code_quality error: E721 Do not compare types, use `isinstance()`
State: closed | Created: 2023-08-10T10:19:50Z

After latest release of `ruff` (https://pypi.org/project/ruff/0.0.284/), we get the following CI error:
```
src/datasets/utils/py_utils.py:689:12: E721 Do not compare types, use `isinstance()`
```
GitHub issue #6133: Dataset is slower after calling `to_iterable_dataset`
State: open | Created: 2023-08-10T06:36:23Z

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue #6130: default config name doesn't work when config kwargs are specified.
State: closed | Created: 2023-08-09T12:43:15Z

### Describe the bug

https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518-L522

If `config_name` is `None`, `DEFAULT_CONFIG_NAME` should be select. But once users pass `config_kwargs` to their customized `BuilderConfig`, the logic is ignored, and dataset cannot select the default config from multiple configs.

### Steps to reproduce the bug

```python
import datasets

datasets.load_dataset('/dataset/with/multiple/config'')  # Ok
datasets.load_dataset('/dataset/with/multiple/config', some_field_in_config='some')  # Err
```

### Expected behavior

Default config behavior should be consistent.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue #6127: Fix authentication issues
State: closed | Created: 2023-08-07T15:41:25Z

This PR fixes 3 authentication issues:
- Fix authentication when passing `token`.
- Fix authentication in `Audio.decode_example` and `Image.decode_example`.
- Fix authentication to resolve `data_files` in repositories without script.

This PR also fixes our CI so that we properly test when passing `token` and we do not use the token stored in `HfFolder`.

Fix #6126.

## Details


### Fix authentication when passing `token`

See c0a77dc943de68a17f23f141517028c734c78623

The root issue was caused when the `token` was set in an already instantiated `DownloadConfig` and thus not propagated to `self._storage_options`:

  ```python
  download_config.token = token
  ```

As this usage pattern is very common, the fix consists in overriding `DownloadConfig.__setattr__`.

This fixes authentication issues in the following functions:
- `load_dataset` and `load_dataset_builder`
- `Dataset.push_to_hub` and `Dataset.push_to_hub`
- `inspect.get_dataset_config_info`, `inspect.get_dataset_infos` and `inspect.get_dataset_split_names`

### Fix authentication in `Audio.decode_example` and `Image.decode_example`. 

See: 58e62af004b6b8b84dcfd897a4bc71637cfa6c3f

The `token` was not set because the `repo_id` was wrongly tried to be parsed from an HTTP URL (`"http://..."`), instead of an HFFileSystem URL (`"hf://"`)

### Fix authentication to resolve `data_files` in repositories without script

See: e4684fc1032321abf0d494b0c130ea7c82ebda80

This is fixed by passing `download_config` to the function `create_builder_configs_from_metadata_configs`
GitHub issue #6125: Reinforcement Learning and Robotics are not task categories in HF datasets metadata
State: closed | Created: 2023-08-05T23:59:42Z

### Describe the bug

In https://huggingface.co/models there are task categories for RL and robotics but none in https://huggingface.co/datasets

Our lab is currently moving our datasets over to hugging face and would like to be able to add those 2 tags

Moreover we see some older datasets that do have that tag, but we can't seem to add it ourselves.

### Steps to reproduce the bug

1. Create a new dataset on Hugging face
2. Try to type reinforcemement-learning or robotics into the tasks categories, it does not allow you to commit

### Expected behavior

Expected to be able to add RL and robotics as task categories as some previous datasets have these tags

### Environment info

N/A
GitHub issue #6124: Datasets crashing runs due to KeyError
State: closed | Created: 2023-08-05T17:48:56Z

### Describe the bug

Hi all,

I have been running into a pretty persistent issue recently when trying to load datasets. 
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```
I receive a KeyError which crashes the runs.
```
Traceback (most recent call last):
    main()

    train_dataset = load_dataset(
                    ^^^^^^^^^^^^^
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
    raise e1 from None

    ).get_module()
      ^^^^^^^^^^^^
    else get_data_patterns(base_path, download_config=self.download_config)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return _get_data_files_patterns(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data_files = pattern_resolver(pattern)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    fs, _, _ = get_fs_token_paths(pattern, storage_options=storage_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
                               ^^^^^^^^^^^^^^

    allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

    listing = self.ls(path, detail=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    "last_modified": parse_datetime(tree_item["lastCommit"]["date"]),
                                    ~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'lastCommit'
```
Any help would be greatly appreciated.

Thank you,

Enrico

### Steps to reproduce the bug

Load the dataset from the Huggingface hub.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```

### Expected behavior

Loads the dataset.

### Environment info

datasets-2.14.3
CUDA 11.8
Python 3.11
GitHub issue #6122: Upload README via `push_to_hub`
State: closed | Created: 2023-08-04T21:00:27Z

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.
GitHub issue #6120: Lookahead streaming support?
State: open | Created: 2023-08-04T04:01:52Z

### Feature request

From what I understand, streaming dataset currently pulls the data, and process the data as it is requested.
This can introduce significant latency delays when data is loaded into the training process, needing to wait for each segment.

While the delays might be dataset specific (or even mapping instruction/tokenizer specific)

Is it possible to introduce a `streaming_lookahead` parameter, which is used for predictable workloads (even shuffled dataset with fixed seed). As we can predict in advance what the next few datasamples will be. And fetch them while the current set is being trained.

With enough CPU & bandwidth to keep up with the training process, and a sufficiently large lookahead, this will reduce the various latency involved while waiting for the dataset to be ready between batches.

### Motivation

Faster streaming performance, while training over extra large TB sized datasets

### Your contribution

I currently use HF dataset, with pytorch lightning trainer for RWKV project, and would be able to help test this feature if supported.
GitHub issue #6116: [Docs] The "Process" how-to guide lacks description of `select_columns` function
State: closed | Created: 2023-08-03T13:45:10Z

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.
GitHub issue #6114: Cache not being used when loading commonvoice 8.0.0
State: closed | Created: 2023-08-02T23:18:11Z

### Describe the bug

I have commonvoice 8.0.0 downloaded in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.  The folder contains all the arrow files etc, and was used as the cached version last time I touched the ec2 instance I'm working on.  Now, with the same command that downloaded it initially:

```
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")
```

it tries to redownload the dataset to `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/05bdc7940b0a336ceeaeef13470c89522c29a8e4494cbeece64fb472a87acb32`

### Steps to reproduce the bug

Steps to reproduce the behavior:

1.  ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```
2. dataset is updated by maintainers
3. ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```

### Expected behavior

I expect that it uses the already downloaded data in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.

Not sure what's happening in 2. but if, say it's an issue with the dataset referenced by "mozilla-foundation/common_voice_8_0" being modified by the maintainers, how would I force datasets to point to the original version I downloaded?

EDIT:  It was indeed that the maintainers had updated the dataset (v 8.0.0).  However I still cant load the dataset from disk instead of redownloading, with for example:

```
load_dataset(".cache/huggingface/datasets/downloads/extracted/<hash>/cv-corpus-8.0-2022-01-19/en/", "en")

> ...
> File [~/miniconda3/envs/aa_torch2/lib/python3.10/site-packages/datasets/table.py:1938](.../ python3.10/site-packages/datasets/table.py:1938), in cast_array_to_feature(array, feature, allow_number_to_str)
   1937 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1938     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
...
   1794         e = e.__context__
-> 1795     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1797 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```


### Environment info

datasets==2.7.0
python==3.10.8
OS:  AWS Linux
GitHub issue #6113: load_dataset() fails with streamlit caching inside docker
State: closed | Created: 2023-08-02T20:20:26Z

### Describe the bug

When calling `load_dataset` in a streamlit application running within a docker container, get a failure with the error message:

EmptyDatasetError: The directory at hf://datasets/fetch-rewards/inc-rings-2000@bea27cf60842b3641eae418f38864a2ec4cde684 doesn't contain any data files
Traceback:
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
File "/home/user/app/app.py", line 62, in <module>
    dashboard()
File "/home/user/app/app.py", line 47, in dashboard
    feat_dict, path_gml = load_data(hf_repo, model_gml_dict[selected_model], hf_token)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 211, in wrapper
    return cached_func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 240, in __call__
    return self._get_or_create_cached_value(args, kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 266, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 320, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
File "/home/user/app/hf_interface.py", line 16, in load_data
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2109, in load_dataset
    builder_instance = load_dataset_builder(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1795, in load_dataset_builder
    dataset_module = dataset_module_factory(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1486, in dataset_module_factory
    raise e1 from None
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1476, in dataset_module_factory
    ).get_module()
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1032, in get_module
    else get_data_patterns(base_path, download_config=self.download_config)
File "/opt/conda/lib/python3.10/site-packages/datasets/data_files.py", line 458, in get_data_patterns
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None

### Steps to reproduce the bug

```python
@st.cache_resource
def load_data(repo_id: str, hf_token=None):
    """Load data from HuggingFace Hub
    """
    
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
    hf_dataset = hf_dataset.map(lambda x: json.loads(x["ground_truth"]), remove_columns=["ground_truth"])

    return hf_dataset
```

### Expected behavior

Expect to load. 
Note: works fine with datasets==2.13.1

### Environment info

datasets==2.14.2, 
Ubuntu bionic-based Docker container.
GitHub issue #6111: raise FileNotFoundError("Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory." )
State: closed | Created: 2023-08-02T09:17:29Z

### Describe the bug

For researchers in some countries or regions, it is usually the case that the download ability of `load_dataset` is disabled due to the complex network environment. People in these regions often prefer to use git clone or other programming tricks to manually download the files to the disk (for example, [How to elegantly download hf models, zhihu zhuanlan](https://zhuanlan.zhihu.com/p/475260268) proposed a crawlder based solution,  and [Is there any mirror for hf_hub, zhihu answer](https://www.zhihu.com/question/371644077) provided some cloud based solutions, and [How to avoid pitfalls on Hugging face downloading, zhihu zhuanlan] gave some useful suggestions), and then use `load_from_disk` to get the dataset object. 
However, when one finally has the local files on the disk, it is still buggy when trying to load the files into objects. 

### Steps to reproduce the bug

Steps to reproduce the bug:
1. Found CIFAR dataset in hugging face: https://huggingface.co/datasets/cifar100/tree/main
2. Click ":" button to show "Clone repository" option, and then follow the prompts on the box:
 ```bash
  cd my_directory_absolute
  git lfs install
  git clone https://huggingface.co/datasets/cifar100
  ls  my_directory_absolute/cifar100 # confirm that the directory exists and it is OK. 
  ```
3. Write A python file to try to load the dataset
```python
from datasets import load_dataset, load_from_disk
dataset = load_from_disk("my_directory_absolute/cifar100")
```
Notice that according to issue #3700 , it is wrong to use load_dataset("my_directory_absolute/cifar100"), so we must use load_from_disk instead. 

4. Then you will see the error reported:
```log
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[5], line 9
      1 from datasets import load_dataset, load_from_disk
----> 9 dataset = load_from_disk("my_directory_absolute/cifar100")

File [~/miniconda3/envs/ai/lib/python3.10/site-packages/datasets/load.py:2232), in load_from_disk(dataset_path, fs, keep_in_memory, storage_options)
   2230     return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
   2231 else:
-> 2232     raise FileNotFoundError(
   2233         f"Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory."
   2234     )

FileNotFoundError: Directory my_directory_absolute/cifar100 is neither a `Dataset` directory nor a `DatasetDict` directory.
```

### Expected behavior

The dataset should be load successfully. 

### Environment info

```bash
datasets-cli env
```
-> results:
```txt

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.14.2
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
```
GitHub issue #6110: [BUG] Dataset initialized from in-memory data does not create cache.
State: closed | Created: 2023-08-01T11:58:58Z

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #6109: Problems in downloading Amazon reviews from HF
State: closed | Created: 2023-08-01T08:38:29Z

### Describe the bug

I have a script downloading `amazon_reviews_multi`.

When the download starts, I get

```
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.43MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.54s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 842.40it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 928kB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 832.70it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.81MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.40s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 1294.14it/s]
Generating train split:   0%|          | 0/200000 [00:00<?, ? examples/s]
```

the file is clearly too small to contain the requested dataset, in fact it contains en error message:

```
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AGJWSY3ZADT2QVWE</RequestId><HostId>Gx1O2KXnxtQFqvzDLxyVSTq3+TTJuTnuVFnJL3SP89Yp8UzvYLPTVwd1PpniE4EvQzT3tCaqEJw=</HostId></Error>
```

obviously the script fails:

```
>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```



### Steps to reproduce the bug

1. load_dataset("amazon_reviews_multi", name="en", split="train", cache_dir="ADDYOURPATHHERE")

### Expected behavior

I would expect the dataset to be downloaded and processed

### Environment info

* The problem is present with both datasets 2.12.0 and 2.14.2
* python version 3.10.12