jamesLeeeeeee/datasets-github-issues huggingface dataset:
Loading Meteor metric in HF evaluate module crashes due to datasets import issue 
Author: dsashulya | Pull request: no | State: closed

### Describe the bug

When using python3.9 and ```evaluate``` module loading Meteor metric crashes at a non-existent import from ```datasets.config``` in ```datasets v2.14```

### Steps to reproduce the bug

```
from evaluate import load
meteor = load("meteor")
```

produces the following error:
  ```
    from datasets.config import importlib_metadata, version
ImportError: cannot import name 'importlib_metadata' from 'datasets.config' (<path_to_project>/venv/lib/python3.9/site-packages/datasets/config.py)
```

### Expected behavior

```datasets``` of v2.10 has the following workaround in ```config.py```:

```
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```

However, it's absent in v2.14 which might be the cause of the issue.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.6
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Evaluate version: 0.4.0
Map cache with tokenizer
Author: jonathanasdf | Pull request: no | State: open

Similar issue to https://github.com/huggingface/datasets/issues/5985, but across different sessions rather than two calls in the same session.

Unlike that issue, explicitly calling tokenizer(my_args) before the map() doesn't help, because the tokenizer was created with a different hash to begin with...

setup
```
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained('bert-base-uncased').save_pretrained("tok")
```

this prints different value each time
```
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets
print(hash(dumps(AutoTokenizer.from_pretrained("tok"))))
```
'import datasets' throws "invalid syntax error"
Author: elia-ashraf | Pull request: no | State: closed

### Describe the bug

Hi,

I have been trying to import the datasets library but I keep gtting this error.

`Traceback (most recent call last):

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/IPython/core/interactiveshell.py:3508 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[2], line 1
    import datasets

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/__init__.py:22
    from .arrow_dataset import Dataset

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_dataset.py:67
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_writer.py:27
    from .features import Features, Image, Value

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/__init__.py:17
    from .audio import Audio

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/audio.py:11
    from ..download.streaming_download_manager import xopen, xsplitext

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/__init__.py:10
    from .streaming_download_manager import StreamingDownloadManager

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/streaming_download_manager.py:18
    from aiohttp.client_exceptions import ClientError

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/__init__.py:7
    from .connector import *  # noqa

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/connector.py:12
    from .client import ClientRequest

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/client.py:144
    yield from asyncio.async(resp.release(), loop=loop)
                       ^
SyntaxError: invalid syntax`

I have simply used these commands:
`import datasets`
and
`from datasets import load_dataset`

### Environment info

The library has been installed a virtual machine on JupyterHub. Although I have used this library multiple times (on the same VM) before, to train/test an ASR or other ML models, I had never encountered this error.
how to limit the size of memory mapped file?
Author: williamium3000 | Pull request: no | State: open

### Describe the bug

Huggingface datasets use memory-mapped file to map large datasets in memory for fast access.
However, it seems like huggingface will occupy all the memory for memory-mapped files, which makes a troublesome situation since we cluster will distribute a small portion of memory to me (once it's over the limit, memory cannot be allocated), however, when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 
So is there a way to explicitly limit the size of memory mapped file?

### Steps to reproduce the bug

python
>>> from datasets import load_dataset
>>> dataset = load_dataset("c4", "en", streaming=True)

### Expected behavior

In a normal environment, this will not have any problem.
However, when the system allocates a portion of the memory to the program and when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 

### Environment info

linux cluster with SGE（Sun Grid Engine）
PyArrow 13 CI fixes
Author: mariosasko | Pull request: yes | State: closed

Fixes:
* bumps the PyArrow version check in the `cast_array_to_feature` to avoid the offset bug (still not fixed)
* aligns the Pandas formatting tests with the Numpy ones (the current test fails due to https://github.com/apache/arrow/pull/35656, which requires `.to_pandas(coerce_temporal_nanoseconds=True)` to always return `datetime [ns]` objects)

Fix #6173
Fix CI for pyarrow 13.0.0
Author: lhoestq | Pull request: no | State: closed

pyarrow 13.0.0 just came out

```
FAILED tests/test_formatting.py::ArrowExtractorTest::test_pandas_extractor - AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[us, UTC]
[right]: datetime64[ns, UTC]
```

```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```

e.g. in https://github.com/huggingface/datasets/actions/runs/5952253963/job/16143847230

first error may be related to https://github.com/apache/arrow/issues/33321

second one maybe because `feature.length * len(array) == len(array_values)` is not satisfied anymore somehow ?
Make Dataset streaming queries retryable
Author: rojagtap | Pull request: no | State: open

### Feature request

Streaming datasets, as intended, do not load the entire dataset in memory or disk. However, while querying the next data chunk from the remote, sometimes it is possible that the service is down or there might be other issues that may cause the query to fail. In such a scenario, it would be nice to make these queries retryable (perhaps with a backoff strategy).

### Motivation

I was working on a model and the model checkpoints after every 1000 steps. At step 1800 I got a 504 HTTP status code error from Huggingface hub for my pytorch `dataloader`. Given the size of my model and data, it took around 2 hours to reach 1800 steps and now it will take about an hour to recover the lost 800. It would be better to get a retryable querying strategy.

### Your contribution

It would be better if someone having experience in this area takes this up as this would require some testing.
feat: Return the name of the currently loaded file
Author: Amitesh-Patel | Pull request: yes | State: open

Added an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

I added this here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

fixes #5806
Configurations in yaml not working
Author: tsor13 | Pull request: no | State: open

### Dataset configurations cannot be created in YAML/README

Hello! I'm trying to follow the docs here in order to create structure in my dataset as added from here (#5331): https://github.com/huggingface/datasets/blob/8b8e6ee067eb74e7965ca2a6768f15f9398cb7c8/docs/source/repository_structure.mdx#L110-L118

I have the exact example in my config file for [my data repo](https://huggingface.co/datasets/tsor13/test):
```
configs:
- config_name: main_data
  data_files: "main_data.csv"
- config_name: additional_data
  data_files: "additional_data.csv"
```

Yet, I'm unable to load different configurations:
```
from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test', use_auth_token=True)
```
returns a single split, `['tsor13--test']`

Does anyone have any insights?

@polinaeterna thank you for adding this feature, it is super useful. Do you happen to have any ideas?

### Steps to reproduce the bug

from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test')

### Expected behavior

I would expect there to be two splits, `main_data` and `additional_data`. However, only `['tsor13--test']` test is returned.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.1
Fix multiprocessing with spawn in iterable datasets
Author: Hubert-Bonisseur | Pull request: yes | State: closed

The "Spawn" method is preferred when multiprocessing on macOS or Windows systems, instead of the "Fork" method on linux systems.

This causes some methods of Iterable Datasets to break when using a dataloader with more than 0 workers.

I fixed the issue by replacing lambda and local methods which are not pickle-able.

See the example below:

```python
from datasets import load_dataset
from torch.utils.data import DataLoader


if __name__ == "__main__":
    dataset = load_dataset("lhoestq/demo1", split="train")
    dataset = dataset.to_iterable_dataset(num_shards=3)

    dataset = dataset.remove_columns(["package_name"])
    dataset = dataset.rename_columns({
        "review": "review1"
    })
    dataset = dataset.rename_column("date", "date1")
    for sample in DataLoader(dataset, batch_size=None, num_workers=3):
        print(sample)
```

To notice the fix on a linux system, adding these lines should do the trick:

```python
import multiprocessing
multiprocessing.set_start_method('spawn')
```

I also removed what looks like code duplication between rename_colums and rename_column
Fix: Missing a MetadataConfigs init when the repo has a `datasets_info.json` but no README
Author: clefourrier | Pull request: yes | State: closed

When I try to push to an arrow repo (can provide the link on Slack), it uploads the files but fails to update the metadata, with
```
  File "app.py", line 123, in add_new_eval
    eval_results[level].push_to_hub(my_repo, token=TOKEN, split=SPLIT)
  File "blabla_my_env_path/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5501, in push_to_hub
    if not metadata_configs:
UnboundLocalError: local variable 'metadata_configs' referenced before assignment
```

This fixes it.
Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32
Author: shishirCTC | Pull request: no | State: open

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?
Add `BoundingBox` feature
Author: mariosasko | Pull request: no | State: open

... to make working with object detection datasets easier. Currently, `Sequence(int_or_float, length=4)` can be used to represent this feature optimally (in the storage backend), so I only see this feature being useful if we make it work with the viewer. Also, bounding boxes usually come in 4 different formats (explained [here](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/)), so we need to decide which one to support (or maybe all of them).

cc @NielsRogge @severo
Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
Author: npuichigo | Pull request: no | State: closed

### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related
FolderBase Dataset automatically resolves under current directory when data_dir is not specified
Author: npuichigo | Pull request: no | State: open

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
Faster sorting for single key items
Author: jackapbutler | Pull request: no | State: closed

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps. 

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True) 
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```
Allow dataset implement .take
Author: brando90 | Pull request: no | State: open

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True. 

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)
Dataset.from_parquet cannot load subset of columns
Author: dwyatte | Pull request: no | State: closed

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset


pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
ValueError when running BeamBasedBuilder with GCS path in cache_dir
Author: ktrk115 | Pull request: no | State: open

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets


class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3
custom load dataset to hub
Author: andysingal | Pull request: no | State: closed

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub
Misalignment between file format specified in configs metadata YAML and the inferred builder
Author: albertvillanova | Pull request: no | State: closed

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
``` 
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna
Offline dataset viewer
Author: yuvalkirstain | Pull request: no | State: closed

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.
Ignore CI lint rule violation in Pickler.memoize
Author: albertvillanova | Pull request: yes | State: closed

This PR ignores the violation of the lint rule E721 in `Pickler.memoize`.

The lint rule violation was introduced in this PR:
- #3182

@lhoestq is there a reason you did not use `isinstance` instead?

As a hotfix, we just ignore the violation of the lint rule.

Fix #6136.
CI check_code_quality error: E721 Do not compare types, use `isinstance()`
Author: albertvillanova | Pull request: no | State: closed

After latest release of `ruff` (https://pypi.org/project/ruff/0.0.284/), we get the following CI error:
```
src/datasets/utils/py_utils.py:689:12: E721 Do not compare types, use `isinstance()`
```
Dataset is slower after calling `to_iterable_dataset`
Author: npuichigo | Pull request: no | State: open

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
default config name doesn't work when config kwargs are specified.
Author: npuichigo | Pull request: no | State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518-L522

If `config_name` is `None`, `DEFAULT_CONFIG_NAME` should be select. But once users pass `config_kwargs` to their customized `BuilderConfig`, the logic is ignored, and dataset cannot select the default config from multiple configs.

### Steps to reproduce the bug

```python
import datasets

datasets.load_dataset('/dataset/with/multiple/config'')  # Ok
datasets.load_dataset('/dataset/with/multiple/config', some_field_in_config='some')  # Err
```

### Expected behavior

Default config behavior should be consistent.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3