Sulak2020/github-issues-multirepo-datasets-Sulakshana huggingface dataset:
Title: Support saving datasets with custom formatting
State: open

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea. 

@lhoestq WDYT?


Comments: Not a fan of pickling this sort of stuff either.
Note that users can also share the code in their dataset documentation.
Title: `Dataset.to_dict()` ignore `decode=True` with Image feature
State: open

### Describe the bug

`Dataset.to_dict` seems to ignore the decoding instruction passed in features.

### Steps to reproduce the bug

```python
import datasets
import numpy as np
from PIL import Image

img = np.random.randint(0, 256, (5, 5, 3), dtype=np.uint8)
img = Image.fromarray(img)

features = datasets.Features({"image": datasets.Image(decode=True)})
dataset = datasets.Dataset.from_dict({"image": [img]}, features=features)
print({key: dataset[key] for key in dataset.column_names})
# {'image': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=5x5 at 0x7EFBC80E15B0>]}
print(dataset.to_dict())
# {'image': [{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x05\x00\x00\x00\x05\x08\x02\x00\x00\x00\x02\r\xb1\xb2\x00\x00\x00[IDATx\x9c\x01P\x00\xaf\xff\x01\x13\x1b<7\xe7\xe0\xdc^6\xed\x04\xc7M\xd2\x9f\x00X\x1b\xb0?\x1ba\x15\xc5 o\xd0\x80\xbe\x19/\x01\xec\x95\x1f\x9f\xffj\xfa1\xa7\xc4X\xea\xbe\xa4g\x00\xc4\x15\xdeC\xc7 \xbbaqe\xc8\xb9\xa9q\xe7\x00,?M\xc0)\xdaD`}\xb1\xdci\x1e\xafC\xa9]%.@\xa6\xf0\xb3\x00\x00\x00\x00IEND\xaeB`\x82', 'path': None}]}
```




### Expected behavior

I would expect `{key: dataset[key] for key in dataset.column_names}` and `dataset.to_dict()` to be equivalent. If the previous behavior is expected, then it should be stated [in the doc](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.to_dict).

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Pillow 9.5.0
- numpy 1.25.2

Comments: We need to implement the `Image` type as a PyArrow extension type (to allow us to override the Python conversion) for this to work as expected. For now, it's best to use your approach indeed.
Title: Unpin fsspec < 2023.9.0
State: closed
Labels: name: enhancement
description: New feature or request

Once root issue is fixed, remove temporary pin of fsspec < 2023.9.0 introduced by:
- #6210

Related to issue:
- #6209

After investigation, I think the root issue is related to the new glob behavior with double asterisk `**` they have introduced in:
- https://github.com/fsspec/filesystem_spec/pull/1329
Title: No-script datasets with ZIP files do not load
State: closed
Labels: name: bug
description: Something isn't working

While investigating an issue on a Hub dataset, I have discovered the no-script datasets containing ZIP files do not load.

For example, that no-script dataset containing ZIP files, raises NonMatchingSplitsSizesError:
```python
In [2]: ds = load_dataset("sidovic/LearningQ-qg")

NonMatchingSplitsSizesError: [
  {
    'expected': SplitInfo(name='train', num_bytes=0, num_examples=188660, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='validation', num_bytes=0, num_examples=20630, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='validation', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='test', num_bytes=0, num_examples=18227, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }
]
```

As another example, a no-script dataset containing just a (CSV)-ZIP file, raises a DatasetGenerationError:
```
>               num_examples, num_bytes = writer.finalize()

src/datasets/builder.py:1949: 

>           raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
E           datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

src/datasets/arrow_writer.py:598: SchemaInferenceError

The above exception was the direct cause of the following exception:

src/datasets/load.py:2143: in load_dataset
    builder_instance.download_and_prepare(
src/datasets/builder.py:954: in download_and_prepare
    self._download_and_prepare(
src/datasets/builder.py:1049: in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
src/datasets/builder.py:1813: in _prepare_split
    for job_id, done, content in self._prepare_split_single(

>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

src/datasets/builder.py:1958: DatasetGenerationError
```

After investigating, I think this bug was introduced in this PR:
- #5972

Related to:
- https://huggingface.co/datasets/sidovic/LearningQ-qg/discussions/1

CC: @lhoestq
Title: avoid downgrading jax version
State: closed
Labels: name: enhancement
description: New feature or request

### Feature request

Whenever I `pip install datasets[jax]` it downgrades jax to version 0.3.25. I seem to be able to install this library first then upgrade jax back to version 0.4.13.

### Motivation

It would be nice to not overwrite currently installed version of jax if possible.

### Your contribution

I would be willing to beta test. Or maybe write some code if I could get pointed in the right direction, I'm not super familiar with this codebase.

Comments: https://github.com/huggingface/datasets/blob/main/setup.py#L236
Currently has the highest version at 0.3.25; Not sure if there is any reason for this, other than that was the tested version?
Title: Use load_dataset for local json files, but it not works
State: open

### Describe the bug

when I use load_dataset to load my local datasets，it always goes to Hugging Face to download the data instead of loading the local dataset.

### Steps to reproduce the bug

`raw_datasets = load_dataset(
        ‘json’,
        data_files=data_files)`


### Expected behavior

![image](https://github.com/huggingface/datasets/assets/50519434/add3747f-6481-4da7-b374-8f81c5a6472c)


### Environment info

python version 3.8.5
datasets version 2.12
os version unbuntu 18.04

Comments: Hugging Face's datasets library may prioritize remote configurations. Make sure there are no conflicting configurations causing the library to prefer downloading data
May  be try debugging
raw_datasets = load_dataset('json', data_files=data_files)
print(raw_datasets)
It doesn't download them but writes them to the local HF cache. The logging could indeed be better. Does loading the dataset succeed? If it doesn't, can you share the error stack trace?
Title: ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'
State: closed

### Describe the bug

Saving a dataset `.to_json()` fails with a `ValueError` since the latest `pandas` [release](https://pandas.pydata.org/docs/dev/whatsnew/v2.1.0.html) (`2.1.0`)

In their latest release we have:

> Improved error handling when using [DataFrame.to_json()](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) with incompatible index and orient arguments ([GH 52143](https://github.com/pandas-dev/pandas/issues/52143))

i.e. an error is now raised for invalid combinations of `index` and `orient`.

This means that unfortunately the custom logic at this line might sometimes lead to contradictions:

https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/io/json.py#L96

e.g. for the default case `orient=records` leads to `index=True`, which now raises a `ValueError`

### Steps to reproduce the bug

```python
import datasets


if __name__ == '__main__':

    dataset = datasets.Dataset.from_dict({"A": [1, 2, 3], "B": [4, 5, 6]})
    dataset.to_json("dataset.json")
```

```shell
>>>
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'.
```

### Expected behavior

The dataset is successfully saved as `.json`

### Environment info

`python >= 3.9`
`pandas >= 2.1.0`

Comments: Thanks for reporting. We are investigating it.This issue is caused by latest `pandas` release 2.1.0 (released yesterday Aug 30).

See: https://github.com/huggingface/datasets/actions/runs/6035484010/job/16375932085?pr=6198
People using previous releases of `datasets` should pin `pandas` in their local environment:
```
python -m pip install 'pandas<2.1.0'
```
Title: Split order is not preserved
State: closed
Labels: name: bug
description: Something isn't working

I have noticed that in some cases the split order is not preserved.

For example, consider a no-script dataset with configs:
```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path: train.csv
  - split: test
    path: test.csv
```
- Note the defined split order is [train, test]

Once the dataset is loaded, the split order is not preserved:
```python
In [16]: ds
Out[16]: 
DatasetDict({
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 1
    })
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 2
    })
})
```
- Note the obtained split order is [test, train]
Title: Dataset loading script method does not work with .pyc file
State: open

### Describe the bug

The huggingface dataset library specifically looks for ‘.py’ file while loading the dataset using loading script approach and it does not work with ‘.pyc’ file.
While deploying in production, it becomes an issue when we are restricted to use only .pyc files. Is there any work around for this ?

### Steps to reproduce the bug

1. Create a dataset  loading script to read the custom data.
2. compile the code to make sure that .pyc file is created 
3. Delete the loading script and re-run the code. Usually, python should make use of complied .pyc files. However, in this case, the dataset library errors out with the message that it's unable to find the data loader loading script.

### Expected behavior

The code should make use of .pyc file and run without any error.

### Environment info

NA

Comments: Before dynamically loading `.py` scripts with `importlib.import_module`, we also parse their contents to check imports, which is tricky to implement for binary `.pyc` files (requires parsing bytecode), so I don't think this is something we want to support (unless more users request it ofc) as this use case is a bit too specific.

@lhoestq What's your opinion on this?> Before dynamically loading .py scripts with importlib.import_module, we also parse their contents to check imports, which is tricky to implement for binary .pyc files (requires parsing bytecode), so I don't think this is something we want to support (unless more users request it ofc) as this use case is a bit too specific.

Yes indeed. Though you can use a .py that imports a package that contains your .pyc code and that you previously installedHi @lhoestq ,
Could you share some example code related to the approach that you are suggesting?
Title: `Invalid user token` even when correct user token is passed!
State: closed

### Describe the bug

I'm working on a dataset which comprises other datasets on the hub.
URL: https://huggingface.co/datasets/open-asr-leaderboard/datasets-test-only

Note: Some of the sub-datasets in this metadataset require explicit access.

All the other datasets work fine, except, `common_voice`.

### Steps to reproduce the bug

https://github.com/Vaibhavs10/scratchpad/blob/main/cv_datasets_bug_repro.ipynb

### Expected behavior

It should work if the provided access token is valid (as it does for all the other datasets)

### Environment info

datasets version -> 2.14.4

Comments: This is because `download_config.use_auth_token` is deprecated - you should use `download_config.token` insteadWorks! Thanks for the quick fix! <3
Title: [Feature Request] Check the length of batch before writing so that empty batch is allowed
State: closed

### Use Case
I use `dataset.map(process_fn, batched=True)` to process the dataset, with data **augmentations or filtering**. However, when all examples within a batch is filtered out, i.e. **an empty batch is returned**, the following error will be thrown:
  ```
  ValueError: Schema and number of arrays unequal
  ```
This is because the empty batch does not comply with the schema of other batches. I think an empty batch should be allowed to facilitate coding (one does not need to assign an empty list manually for all keys.)

A simple fix is to check the length of `batch` before writing:
```
if len(batch):
  writer.write_batch(batch)
```
instead of 

https://github.com/huggingface/datasets/blob/74d60213dcbd7c99484c62ce1d3dfd90a1df0770/src/datasets/arrow_dataset.py#L3493


Comments: I think this error means you filter all examples within an (input) batch by deleting its columns. In that case, to avoid the error, you can set the column value to an empty list (`input_batch["col"] = []`) instead.
Title: Error in saving the PIL image into *.arrow files using datasets.arrow_writer
State: open

### Describe the bug

I am using the ArrowWriter from datasets.arrow_writer to save a json-style file as arrow files. Within the dictionary, it contains a feature called "image" which is a list of PIL.Image objects.
I am saving the json using the following script:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
```
However, when I attempt to restore the dataset and use the ```Dataset.from_file(path)``` function to load the arrow file, there seems to be an issue with the PIL.Image object in the dataset. The list of PIL.Images appears as follows rather than a normal PIL.Image object:
![1693051705440](https://github.com/huggingface/datasets/assets/14247682/03b204c2-d0fa-4d19-beff-6f4d7b83c848)


### Steps to reproduce the bug

1. Storing the data json into arrow files:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
save_to_arrow( path, json_file )
```
2.  try to load the arrow file into the Dataset object using the ```Dataset.from_file(path)```


### Expected behavior

Except to saving the contained "image" feature as a list PIL.Image objects as the arrow file. And I can restore the dataset from the file.

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.4.4

Comments: You can cast the `input_image` column to the `Image` type to fix the issue:
```python
ds.cast_column("input_image", datasets.Image())
```
Title: Map cache does not detect function changes in another module
State: closed
Labels: name: duplicate
description: This issue or pull request already exists

```python
# dataset.py
import os
import datasets

if not os.path.exists('/tmp/test.json'):
  with open('/tmp/test.json', 'w') as file:
    file.write('[{"text": "hello"}]')

def transform(example):
  text = example['text']
  # text += ' world'
  return {'text': text}

data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
```

```python
# test.py
import dataset
print(next(iter(dataset.data)))
```

Initialize cache
```
python3 test.py
# {'text': 'hello'}
```

Edit dataset.py and uncomment the commented line, run again
```
python3 test.py
# {'text': 'hello'}
# expected: {'text': 'hello world'}
```

Clear cache and run again
```
rm -rf ~/.cache/huggingface/datasets/*
python3 test.py
# {'text': 'hello world'}
```


If instead the two files are combined, then changes to the function are detected correctly. But it's expected when working on any realistic codebase that things will be modularized into separate files.

Comments: This issue is a duplicate of https://github.com/huggingface/datasets/issues/3297.  This is a limitation of `dill`, a package we use for caching (non-`__main__` module objects are serialized by reference). You can find more info about it here: https://github.com/uqfoundation/dill/issues/424.

In your case, moving 
```
data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
``` 
to `test.py`  and setting `transform.__module__ = None` at the end of `dataset.py` should fix the issue.I understand this may be a limitation of an upstream tool, but for a user for datasets this is very annoying, as when you have dozens of different datasets with different preprocessing functions you can't really move them all into the same file. It may be worth seeing if there is a way to specialize the dependency (eg. subclass it) and enforce behaviors that makes sense for your product.

I was able to work around this for now by setting `__module__ = None`. If such workarounds are required for now it may be better to document it somewhere than a single obscure issue from a long time ago.

As this is a duplicate issue I'm closing it.

I have another issue with the cache https://github.com/huggingface/datasets/issues/6179 can you take a look?
Title: Loading Meteor metric in HF evaluate module crashes due to datasets import issue 
State: closed

### Describe the bug

When using python3.9 and ```evaluate``` module loading Meteor metric crashes at a non-existent import from ```datasets.config``` in ```datasets v2.14```

### Steps to reproduce the bug

```
from evaluate import load
meteor = load("meteor")
```

produces the following error:
  ```
    from datasets.config import importlib_metadata, version
ImportError: cannot import name 'importlib_metadata' from 'datasets.config' (<path_to_project>/venv/lib/python3.9/site-packages/datasets/config.py)
```

### Expected behavior

```datasets``` of v2.10 has the following workaround in ```config.py```:

```
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```

However, it's absent in v2.14 which might be the cause of the issue.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.6
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Evaluate version: 0.4.0

Comments: Our minimal Python version requirement is 3.8, so we dropped `importlib_metadata`. 

Feel free to open a PR in the `evaluate` repo to replace the problematic import with
```python
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```Any idea when you guys will release the next version which deals with this problem?
I'm still having the same issue with py 3.10 when I install the lib with pip.
I'm assuming that it has not yet been updated since the merge was 3 days ago.Yes, this requires a new `evaluate` release (cc @lvwerra for this). 

In the meantime, you can get the fixed version by installing `evaluate` from `main`: `pip install git+https://github.com/huggingface/evaluate.git`I'll aim for a release this week!
Title: 'import datasets' throws "invalid syntax error"
State: closed

### Describe the bug

Hi,

I have been trying to import the datasets library but I keep gtting this error.

`Traceback (most recent call last):

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/IPython/core/interactiveshell.py:3508 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[2], line 1
    import datasets

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/__init__.py:22
    from .arrow_dataset import Dataset

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_dataset.py:67
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_writer.py:27
    from .features import Features, Image, Value

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/__init__.py:17
    from .audio import Audio

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/audio.py:11
    from ..download.streaming_download_manager import xopen, xsplitext

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/__init__.py:10
    from .streaming_download_manager import StreamingDownloadManager

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/streaming_download_manager.py:18
    from aiohttp.client_exceptions import ClientError

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/__init__.py:7
    from .connector import *  # noqa

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/connector.py:12
    from .client import ClientRequest

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/client.py:144
    yield from asyncio.async(resp.release(), loop=loop)
                       ^
SyntaxError: invalid syntax`

I have simply used these commands:
`import datasets`
and
`from datasets import load_dataset`

### Environment info

The library has been installed a virtual machine on JupyterHub. Although I have used this library multiple times (on the same VM) before, to train/test an ASR or other ML models, I had never encountered this error. 

Comments: This seems to be related to your environment and not the `datasets` code (e.g., this could happen when exposing the  Python 3.9 site packages to a lower Python version (interpreter))
Title: Fix CI for pyarrow 13.0.0
State: closed

pyarrow 13.0.0 just came out

```
FAILED tests/test_formatting.py::ArrowExtractorTest::test_pandas_extractor - AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[us, UTC]
[right]: datetime64[ns, UTC]
```

```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```

e.g. in https://github.com/huggingface/datasets/actions/runs/5952253963/job/16143847230

first error may be related to https://github.com/apache/arrow/issues/33321

second one maybe because `feature.length * len(array) == len(array_values)` is not satisfied anymore somehow ?
Title: feat: Return the name of the currently loaded file
State: open

Added an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

I added this here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

fixes #5806

Comments: Your change adds a new element in the key used to avoid duplicates when generating the examples of a dataset. I don't think it fixes the issue you're trying to solve.