jamesLeeeeeee/datasets-github-issues huggingface dataset:
`Dataset` changed dict data and may cause memory leak
Author: scruel | Pull request: no | State: open

### Describe the bug

Will unexpected get keys with `None` value in the parsed json dict.

### Steps to reproduce the bug

```jsonl test.jsonl
{"id": 0, "indexs": {"-1": [0, 10]}}
{"id": 1, "indexs": {"-1": [0, 10]}}
```

```python
dataset = Dataset.from_json('.test.jsonl')
print(dataset[0])
```

Result:
```
{'id': 0, 'indexs': {'-1': [...], '-2': None, '-3': None, '-4': None, '-5': None, '-6': None, '-7': None, '-8': None, '-9': None, ...}}
```

Those keys with `None` value will unexpected appear in the dict.


### Expected behavior

Result should be
```
{'id': 0, 'indexs': {'-1': [0, 10]}}
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
__add__ for Dataset, IterableDataset
Author: oh-gnues-iohc | Pull request: yes | State: open

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```
load_dataset() does not support tsv
Author: dipsivenkatesh | Pull request: no | State: closed

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand
.load_dataset() method defaults to zstandard
Author: ElleLeonne | Pull request: no | State: open

### Describe the bug

Regardless of what method I use, datasets defaults to zstandard for unpacking my datasets.

This is poor behavior, because not only is zstandard not a dependency in the huggingface package (and therefore, your dataset loading will be interrupted while it asks you to install the package), but it happens on datasets that are uploaded in json format too, meaning the dataset loader will attempt to convert the data to a zstandard compatible format, and THEN try to unpackage it.

My 4tb drive runs out of room when using zstandard on slimpajama. It loads fine on 1.5tb when using json, however I lack the understanding of the "magic numbers" system used to select the unpackaging algorithm, so I can't push a change myself.

Commenting out this line, in "/datasets/utils/extract.py" fixes the issue, and causes SlimPajama to properly extract using rational amounts of storage, however it completely disables zstandard, which is probably undesirable behavior. Someone with an understanding of the "magic numbers" system should probably take a pass over this issue.

```
class Extractor:
    #  Put zip file to the last, b/c it is possible wrongly detected as zip (I guess it means: as tar or gzip)
    extractors: Dict[str, Type[BaseExtractor]] = {
        "tar": TarExtractor,
        "gzip": GzipExtractor,
        "zip": ZipExtractor,
        "xz": XzExtractor,
        #"zstd": ZstdExtractor,   # This line needs to go, in order for datasets to work w/o non-dependent packages
        "rar": RarExtractor,
        "bz2": Bzip2Extractor,
        "7z": SevenZipExtractor,  # <Added version="2.4.0"/>
        "lz4": Lz4Extractor,  # <Added version="2.4.0"/>
    }
```

### Steps to reproduce the bug

'''
from datasaets import load_dataset
load_dataset(path="/cerebras/SlimPajama-627B")
'''

This alone should trigger the error on any system that does not have zstandard pip installed.

### Expected behavior

This repository (which is encoded in json format, not zstandard) should check whether zstandard is installed before defaulting to it. Additionally, using zstandard should not use more than 3x the required space that other extraction mechanisms use.

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.0
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Tensor type (e.g. from `return_tensors`) ignored in map
Author: srossi93 | Pull request: no | State: open

### Describe the bug

I don't know if it is a bug or an expected behavior, but the tensor type seems to be ignored after applying map. For example, mapping over to tokenize text with a transformers' tokenizer always returns lists and it ignore the `return_tensors` argument.

If this is an expected behaviour (e.g., for caching/Arrow compatibility/etc.) it should be clearly documented. For example, current documentation (see [here](https://huggingface.co/docs/datasets/v2.17.1/en/nlp_process#map)) clearly state to "set  `return_tensors="np"` when you tokenize your text" to have Numpy arrays.

### Steps to reproduce the bug

```py
# %%%

import datasets
import numpy as np
import tensorflow as tf
import torch
from transformers import AutoTokenizer


# %%
ds = datasets.load_dataset("cnn_dailymail", "1.0.0", split="train[:1%]")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

#%%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** no map, return_tensors={return_tensors} **********")
  _ds = tokenizer(ds["article"], return_tensors=return_tensors, truncation=True, padding=True)
  print('Type <input_ids>:', type(_ds["input_ids"]))

# %%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** map, return_tensors={return_tensors} **********")
  _ds = ds.map(
    lambda examples: tokenizer(examples["article"], return_tensors=return_tensors, truncation=True, padding=True),
    batched=True,
    remove_columns=["article"],
  )

  print('Type <input_ids>:', type(_ds[0]["input_ids"]))
```


### Expected behavior

The output from the script above. I would expect the second half to be the same.

```
********** no map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** no map, return_tensors=np **********
Type <input_ids>: <class 'numpy.ndarray'>
********** no map, return_tensors=pt **********
Type <input_ids>: <class 'torch.Tensor'>
********** no map, return_tensors=tf **********
Type <input_ids>: <class 'tensorflow.python.framework.ops.EagerTensor'>
********** no map, return_tensors=jax **********
Type <input_ids>: <class 'jaxlib.xla_extension.ArrayImpl'>


********** map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=np **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=pt **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=tf **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=jax **********
Type <input_ids>: <class 'list'>
```

### Environment info

- `datasets` version: 2.17.1
- Platform: Redacted (linux)
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
fsspec: support fsspec>=2023.12.0 glob changes
Author: pmrowla | Pull request: yes | State: open

- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)
Question: Is there any way for uploading a large image dataset?
Author: zhjohnchan | Pull request: no | State: open

I am uploading an image dataset like this:
```
dataset = load_dataset(
    "json",
    data_files={"train": "data/custom_dataset/train.json", "validation": "data/custom_dataset/val.json"},
)
dataset = dataset.cast_column("images", Sequence(Image()))
dataset.push_to_hub("StanfordAIMI/custom_dataset", max_shard_size="1GB")
```
where it takes a long time in the `Map` process. Do you think I can use multi-processing to map all the image data to the memory first? For the `Map()` function, I can set `num_proc`. But for `push_to_hub` and `cast_column`, I can not find it.

Thanks in advance!

Best,
Node.js 16 GitHub Actions are deprecated
Author: albertvillanova | Pull request: no | State: open

`Node.js` 16 GitHub Actions are deprecated. See: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

We should update them to Node 20.

See warnings in our CI, e.g.: https://github.com/huggingface/datasets/actions/runs/7957295009?pr=6678
```
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
```
Can't Read List of JSON Files Properly
Author: lordsoffallen | Pull request: no | State: open

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Allow image model (color conversion) to be specified as part of datasets Image() decode
Author: rwightman | Pull request: no | State: open

### Feature request

Typical torchvision / torch Datasets in image applications apply color conversion in the Dataset portion of the code as part of image decode, separately from the image transform stack. This is true for PIL.Image where convert is usually called in dataset, for native torchvision https://pytorch.org/vision/main/generated/torchvision.io.decode_jpeg.html, and similarly in tensorflow.data pipelines decode_jpeg or https://www.tensorflow.org/api_docs/python/tf/io/decode_and_crop_jpeg have a channels arg that allows controlling the image mode in the decode step.

datasets currently requires this pattern (from [examples](https://huggingface.co/docs/datasets/main/en/image_process)):
```
from torchvision.transforms import Compose, ColorJitter, ToTensor

jitter = Compose(
    [
         ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.7),
         ToTensor(),
    ]
)

def transforms(examples):
    examples["pixel_values"] = [jitter(image.convert("RGB")) for image in examples["image"]]
    return examples
```




### Motivation

It would be nice to be able to handle `image.convert("RGB")` (or other modes) in the decode step, before applying torchvision transforms, this would reduce differences in code when handling pipelines that can handle torchvision, webdatset, or hf datasets with fewer code differences and without needing to handle image mode argument passing in two different stages of the pipelines...

### Your contribution

Can do a PR with guidance on how mode should be passed / set on the dataset.
Depprcated Overview.ipynb Link to new Quickstart Notebook invalid
Author: Codeblockz | Pull request: no | State: closed

### Describe the bug

For the dreprecated notebook found  [here](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb). The link to the new notebook is broken. 

### Steps to reproduce the bug

Click the [Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/datasets_doc/quickstart.ipynb) link in the notebook.

### Expected behavior

I believe is it suposed to link [here](https://github.com/huggingface/notebooks/blob/main/datasets_doc/en/quickstart.ipynb). That is mentioned in the readme.

### Environment info

Colab
IterableDataset `set_epoch` is ignored when DataLoader `persistent_workers=True`
Author: rwightman | Pull request: no | State: open

### Describe the bug

When persistent workers are enabled, the epoch that's set via the IterableDataset instance held by the training process is ignored by the workers as they are disconnected across processes.

PyTorch samplers for non-iterable datasets have a mechanism to sync this, datasets.IterableDataset does not.

In my own use of IterableDatasets I usually track the epoch count which crosses process boundaries in a multiprocessing.Value 

### Steps to reproduce the bug

Use a streaming dataset (Iterable) w/ the recommended pattern below and `persistent_workers=True` in the torch DataLoader. 

```
for epoch in range(epochs):
    shuffled_dataset.set_epoch(epoch)
    for example in shuffled_dataset:
        ...
```

### Expected behavior

When the canonical bit of code above is used with `num_workers > 0` and `persistent_workers=True`, the epoch set via `set_epoch()` is propagated to the IterableDataset instances in the worker processes


### Environment info

N/A
CSV builder raises deprecation warning on verbose parameter
Author: albertvillanova | Pull request: no | State: closed

CSV builder raises a deprecation warning on `verbose` parameter:
```
FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.
```

See:
- https://github.com/pandas-dev/pandas/pull/56556
- https://github.com/pandas-dev/pandas/pull/57450
ValueError
Author: prashanth19bolukonda | Pull request: no | State: closed

### Describe the bug

ValueError                                Traceback (most recent call last)
[<ipython-input-11-9b99bc80ec23>](https://localhost:8080/#) in <cell line: 11>()
      9 import numpy as np
     10 import matplotlib.pyplot as plt
---> 11 from datasets import DatasetDict, Dataset
     12 from transformers import AutoTokenizer, AutoModelForSequenceClassification
     13 from transformers import Trainer, TrainingArguments

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     16 __version__ = "2.17.0"
     17 
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     65 
     66 from . import config
---> 67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     69 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     27 
     28 import pyarrow as pa
---> 29 import pyarrow.parquet as pq
     30 from tqdm.contrib.concurrent import thread_map
     31 

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/__init__.py](https://localhost:8080/#) in <module>
     18 # flake8: noqa
     19 
---> 20 from .core import *

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/core.py](https://localhost:8080/#) in <module>
     34 import pyarrow as pa
     35 import pyarrow.lib as lib
---> 36 import pyarrow._parquet as _parquet
     37 
     38 from pyarrow._parquet import (ParquetReader, Statistics,  # noqa

/usr/local/lib/python3.10/dist-packages/pyarrow/_parquet.pyx in init pyarrow._parquet()

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Expected behavior

Resolve the binary incompatibility

### Environment info

Google Colab Note book
Default config for squad is incorrect
Author: kiddyboots216 | Pull request: no | State: open

### Describe the bug

If you download Squad, it will download the plain_text version, but the config still specifies "default", so if you set the offline mode the cache will try to look it up according to the config_id which is "default" and this will say;
ValueError: Couldn't find cache for squad for config 'default'
Available configs in the cache: ['plain_text']

### Steps to reproduce the bug

1. export HF_DATASETS_OFFLINE=0
2. load_dataset("squad")
3. export HF_DATASETS_OFFLINE=1
4. load_dataset("squad")

### Expected behavior

We should change the config_name I guess? 

### Environment info

linux, latest version of datasets
Allow SplitDict setitem to replace existing SplitInfo
Author: lhoestq | Pull request: yes | State: open

Fix this code provided by @clefourrier 

```python
import datasets
import os

token = os.getenv("TOKEN")

results = datasets.load_dataset("gaia-benchmark/results_public", "2023", token=token, download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)
results["test"] = datasets.Dataset.from_list([row for row in results["test"] if row["model"] != "StateFlow"])
results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")
```

```
ValueError                                Traceback (most recent call last)
Cell In[43], line 1
----> 1 results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/arrow_dataset.py:5498, in Dataset.push_to_hub(self, repo_id, config_name, split, private, token, branch, max_shard_size, num_shards, embed_external_files)
   5496         repo_info.dataset_size = (repo_info.dataset_size or 0) + dataset_nbytes
   5497         repo_info.size_in_bytes = repo_info.download_size + repo_info.dataset_size
-> 5498         repo_info.splits[split] = SplitInfo(
   5499             split, num_bytes=dataset_nbytes, num_examples=len(self), dataset_name=dataset_name
   5500         )
   5501         info_to_dump = repo_info
   5502 # create the metadata configs if it was uploaded with push_to_hub before metadata configs existed

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/splits.py:541, in SplitDict.__setitem__(self, key, value)
    539     raise ValueError(f"Cannot add elem. (key mismatch: '{key}' != '{value.name}')")
    540 if key in self:
--> 541     raise ValueError(f"Split {key} already present")
    542 super().__setitem__(key, value)

ValueError: Split test already present
```
`write_examples_on_file` and `write_batch` are broken in `ArrowWriter`
Author: bryant1410 | Pull request: no | State: closed

### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Import error on Google Colab
Author: kithogue | Pull request: no | State: closed

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes
Author: mohalisad | Pull request: yes | State: open

This PR addresses an issue encountered when utilizing uint16 or uint32 datatypes with datasets, followed by attempting to convert these datasets into PyTorch-compatible formats. Currently, doing so results in a TypeError due to incompatible datatype conversion, as illustrated by the following example:
```python
from datasets import Dataset, Sequence, Value, Features

def gen():
    for i in range(100):
        yield {'seq': list(range(i, i + 20))}

ds = Dataset.from_generator(gen, features=Features({'seq': Sequence(feature=Value(dtype='uint16'), length=-1)}))

ds.set_format('torch')

print(ds[0])
```
This code snippet triggers the following error due to the inability to convert numpy.uint16 arrays to a PyTorch-supported format:
```
TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
```

This PR introduces an automatic mechanism to convert np.uint16 and np.uint32 datatypes to np.int64 for seamless compatibility with PyTorch formats, simplifying workflows and improving developer experience by eliminating the need for manual conversion handling.
Change default compression argument for JsonDatasetWriter
Author: Rexhaif | Pull request: yes | State: open

Change default compression type from `None` to "infer", to align with pandas' defaults.

Documentation asks the user to supply `to_json_kwargs` with arguments suitable for pandas' `to_json` method. At the same time, while pandas' by default uses ["infer"](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html) for compression, datasets enforce `None` as default. This, likely, confuses user, as they expect the same behaviour, i.e they expect that if they name their output file as "dataset.jsonl.zst" then the compression would be inferred as "zstd" and file will be compressed before writing.

Moreover, while it is probably outside of the scope of this pull request, `compression` argument needs to be capable of taking `dict` as input (along with `str`), as it does in pandas, in order to allow user to specify compression parameters. Current implementation will likely fail with `NotImplementedError`, as it expects either `None` or `str` specifying compression algo.