noamaanMulla-03/datasets-issues huggingface dataset:
ValueError

### Describe the bug

ValueError                                Traceback (most recent call last)
[<ipython-input-11-9b99bc80ec23>](https://localhost:8080/#) in <cell line: 11>()
      9 import numpy as np
     10 import matplotlib.pyplot as plt
---> 11 from datasets import DatasetDict, Dataset
     12 from transformers import AutoTokenizer, AutoModelForSequenceClassification
     13 from transformers import Trainer, TrainingArguments

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     16 __version__ = "2.17.0"
     17 
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     65 
     66 from . import config
---> 67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     69 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     27 
     28 import pyarrow as pa
---> 29 import pyarrow.parquet as pq
     30 from tqdm.contrib.concurrent import thread_map
     31 

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/__init__.py](https://localhost:8080/#) in <module>
     18 # flake8: noqa
     19 
---> 20 from .core import *

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/core.py](https://localhost:8080/#) in <module>
     34 import pyarrow as pa
     35 import pyarrow.lib as lib
---> 36 import pyarrow._parquet as _parquet
     37 
     38 from pyarrow._parquet import (ParquetReader, Statistics,  # noqa

/usr/local/lib/python3.10/dist-packages/pyarrow/_parquet.pyx in init pyarrow._parquet()

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Expected behavior

Resolve the binary incompatibility

### Environment info

Google Colab Note book


Hi @prashanth19bolukonda,

You have to restart the notebook runtime session after the installation of `datasets`.

Duplicate of:
- #5923

Thank you soo much

On Fri, Feb 16, 2024 at 8:14 PM Albert Villanova del Moral <
***@***.***> wrote:

> Closed #6670 <https://github.com/huggingface/datasets/issues/6670> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6670#event-11829788289>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A2Y44YDQOBUFUWMR4C5O3QTYT5WDJAVCNFSM6AAAAABDL24S5SVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRHAZDSNZYHAZDQOI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Cannot import name 'load_dataset' from .... module ‘datasets’

### Describe the bug


use python -m pip install datasets to install

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

it doesn't work

### Environment info

datasets version==2.15.0
python == 3.10.12
linux  version I don't know??


I don't know  My conpany conputer cannot work. but in my computer, it work?

Do you have a folder in your working directory called datasets?
Default config for squad is incorrect

### Describe the bug

If you download Squad, it will download the plain_text version, but the config still specifies "default", so if you set the offline mode the cache will try to look it up according to the config_id which is "default" and this will say;
ValueError: Couldn't find cache for squad for config 'default'
Available configs in the cache: ['plain_text']

### Steps to reproduce the bug

1. export HF_DATASETS_OFFLINE=0
2. load_dataset("squad")
3. export HF_DATASETS_OFFLINE=1
4. load_dataset("squad")

### Expected behavior

We should change the config_name I guess? 

### Environment info

linux, latest version of datasets


you can try: pip install datasets==2.16.1
Adding option for multipart achive download

Right now we can only download multiple separate archives or a single file archive, but not multipart archives, such as those produced by `tar --multi-volume`. This PR allows for downloading and extraction of archives split into multiple parts.

With the new `multi_part` field of the `DownloadConfig` set, the downloader will first retrieve all the files and attempt to concatenate them before starting extraction. This will obviously fail if files retrieved are actually multiple separate archives, so the option is set to `False` by default.

Tests and docs incoming. 


On closer examination, this appears to be unnecessary.
Import error on Google Colab

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


Hi! This can happen if an incompatible `pyarrow` version (`pyarrow<12.0.0`) has been imported before the `datasets` installation and the Colab session hasn't been restarted afterward. To avoid the error, go to "Runtime -> Restart session" after `!pip install -U datasets` and before `import datasets`, or insert the `import os; os.kill(os.getpid(), 9)` cell between `!pip install -U datasets` and `import datasets` to do the same programmatically.

One possible cause might be the one pointed out by @mariosasko above, and you get the following warning on Colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```

On the other hand, if the old version of `pyarrow` is not previously imported (before the installation of `datasets`), the reported issue here is not reproducible: `datasets` can be installed, imported and used on Colab.

Duplicate of:
- #5923

Google Colab now pre-installs PyArrow 14.0.2, making this issue unlikely to happen. So, I'm unpinning it.
Depprcated Overview.ipynb Link to new Quickstart Notebook invalid

### Describe the bug

For the dreprecated notebook found  [here](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb). The link to the new notebook is broken. 

### Steps to reproduce the bug

Click the [Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/datasets_doc/quickstart.ipynb) link in the notebook.

### Expected behavior

I believe is it suposed to link [here](https://github.com/huggingface/notebooks/blob/main/datasets_doc/en/quickstart.ipynb). That is mentioned in the readme.

### Environment info

Colab


Good catch! Feel free to open a PR to fix the link.
load_dataset() does not support tsv

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand

Labels: name: enhancement

#self-assign

Hi @dipsivenkatesh,

Please note that this functionality is already implemented. Our CSV builder uses `pandas.read_csv` under the hood, and you can pass the parameter `delimiter="\t"` to read TSV files.

See the list of CSV config parameters in our docs: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.packaged_modules.csv.CsvConfig
Enhancement: Enable loading TSV files in load_dataset()

Fix #6691


Hi @harsh1504660,

Thanks for your work, but this functionality already exists. See my comment in the corresponding issue: https://github.com/huggingface/datasets/issues/6691#issuecomment-1963449923

Next time you would like to contribute, I would suggest you take on an issue that is previously validated by one of the maintainers. Thanks anyway.
Node.js 16 GitHub Actions are deprecated

`Node.js` 16 GitHub Actions are deprecated. See: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

We should update them to Node 20.

See warnings in our CI, e.g.: https://github.com/huggingface/datasets/actions/runs/7957295009?pr=6678
> Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.


Labels: name: maintenance
Support fsspec 2023.12

Support fsspec 2023.12 by handling previous and new glob behavior.

Labels: name: enhancement

The pinned fsspec version range dependency conflict has been affecting several of our users in https://github.com/iterative/dvc. I've opened an initial PR that I think should resolve the glob behavior changes with using datasets + the latest fsspec release.

Please let us know if there's any other fsspec related behavior in datasets that needs to be updated to get 2024.2 supported, we'd like to get this conflict resolved as quickly as possible and we're willing to contribute any additional work that's required here.

cc @dberenbaum
Add `__add__()`, `__iadd__()` and similar to `Dataset` class

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.


Labels: name: enhancement

name: generic discussion

I was going through the codebase, and I believe the implementation of __add__() and __iadd__() will be similar to concatenate_datasets() after the elimination of code for arguments other than the list of datasets (info, split, axis). 
(Assuming elimination of axis means concatenating over axis 1.)

Most data frame libraries (Polars, Pandas, etc.) override `__add__` to perform (mathematical) summation, so having different behavior could lead to confusion.
__add__ for Dataset, IterableDataset

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```


Hi! You can find a reason why we are against this feature in https://github.com/huggingface/datasets/issues/3449. 

> It's too cumbersome to write this command every time we perform a dataset merging operation

Explicit is better than implicit, so this isn't a good enough reason. 

Thanks for the effort nonetheless :)!
Can't Read List of JSON Files Properly

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0



Found the issue, if there are other files in the directory, it gets caught into this `*` so essentially it should be `*.json`. Could we possibly to check for list of files to make sure the pattern matches json files and raise error if not?

I don't think we should filter for `*.json` as this might silently remove desired files for many users. And this could be a major breaking change for many organizations.

You could do the globbing yourself which would keep the code clean.

```python
from glob import glob

Dataset.from_json(glob('folder/*.json'))
```

I think it should still be fine to log a warning message in case the folder contains different files? I also don't get why would this be breaking as in the end using `from_FILE_TYPE` should be able to read a specific file type only. Maybe some other use case I am not aware of but since globbing or this case not mentioned anywhere in the doc, I spent quite a bit of time trying to figure out where the issue was. Just making sure it's clear for users.
Is there any way to solve hanging of IterableDataset using split by node + filtering during inference

### Describe the bug

I am using an iterable dataset in a multi-node setup, trying to do training/inference while filtering the data on the fly. I usually do not use `split_dataset_by_node` but it is very slow using the IterableDatasetShard in `accelerate` and `transformers`. When I filter after applying `split_dataset_by_node`, it results in shards that are not equal sizes due to unequal samples filtered from each one. 

The distributed process hangs when trying to accomplish this. Is there any way to resolve this or is it impossible to implement?



### Steps to reproduce the bug

Here is a toy example of what I am trying to do that reproduces the behavior

```
# torchrun --nproc-per-node 2 file.py


import os

import pandas as pd
import torch
from accelerate import Accelerator
from datasets import Features, Value, load_dataset
from datasets.distributed import split_dataset_by_node
from torch.utils.data import DataLoader

accelerator = Accelerator(device_placement=True, dispatch_batches=False)
if accelerator.is_main_process:
    if not os.path.exists("scratch_data"):
        os.mkdir("scratch_data")

    n_shards = 4
    for i in range(n_shards):
        df = pd.DataFrame({"id": list(range(10 * i, 10 * (i + 1)))})
        df.to_parquet(f"scratch_data/shard_{i}.parquet")


world_size = accelerator.num_processes
local_rank = accelerator.process_index


def collate_fn(examples):
    input_ids = []
    for example in examples:
        input_ids.append(example["id"])
    return torch.LongTensor(input_ids)


dataset = load_dataset(
    "parquet", data_dir="scratch_data", split="train", streaming=True
)
dataset = (
    split_dataset_by_node(dataset, rank=local_rank, world_size=world_size)
    .filter(lambda x: x["id"] < 35)
    .shuffle(seed=42, buffer_size=100)
)

batch_size = 2
train_dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    num_workers=2
)  

for x in train_dataloader:
    x = x.to(accelerator.device)
    print({"rank": local_rank, "id": x})
   
    y = accelerator.gather_for_metrics(x)
    if accelerator.is_main_process:
        print("gathered", y)


```

### Expected behavior

Is there any way to continue training/inference on the GPUs that have remaining data left without waiting for the others? Is it impossible to filter when 

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.10.209-198.812.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.6.0
Support webdataset format

Webdataset is an efficient format for iterable datasets. It would be nice to support it in `datasets`, as discussed in https://github.com/rom1504/img2dataset/issues/234.

In particular it would be awesome to be able to load one using `load_dataset` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.

It terms of implementation, we can have something similar to the Parquet loader.
I also think it's fine to have webdataset as an optional dependency.


I like the idea of having `webdataset` as an optional dependency to ensure our loader generates web datasets the same way as the main project.

Webdataset is the one of the most popular dataset formats for large scale computer vision tasks. Upvote for this issue. 

Any updates on this?

We haven't had the bandwidth to implement it so far, but if someone wants to give it a shot please don't hesitate ^^

Done in #6391
Release not pushed to conda channel

### Describe the bug

The github actions step to publish the release 2.17.0 to conda channel has failed due to expired token. Can some one please update the anaconda token rerun the failed action? @albertvillanova ?
 
![image](https://github.com/huggingface/datasets/assets/7138162/1b56ad3d-7643-4778-9cce-4bf531717700)


### Steps to reproduce the bug

Please see this actions [link](https://github.com/huggingface/datasets/actions/runs/7842473662)

### Expected behavior

The action runs successfully and the latest release is pushed to HuggingFace conda channel

### Environment info

Not applicable.


Thanks for reporting, @atulsaurav.

We are investigating the issue. 

I can't fix this issue because I do not appear as a team member of the huggingface datasets project: https://anaconda.org/huggingface/datasets

@lhoestq could you please add `datasets` team members to the corresponding Anaconda project?

Once this done, I could recreate and update the Anaconda token, as mentioned above it seems the current one has expired.

I think @LysandreJik has access ?

FYI it failed for 2.18.0 too: https://github.com/huggingface/datasets/actions/runs/8117132330/job/22188677936

We updated the token and I re-ran the conda releases :)
Support `fsspec` in `Dataset.to_<format>` methods

Supporting this should be fairly easy.

Requested on the forum [here](https://discuss.huggingface.co/t/how-can-i-convert-a-loaded-dataset-in-to-a-parquet-file-and-save-it-to-the-s3/48353).

Labels: name: enhancement

Hi @mariosasko unless someone's already working on it, I guess I can tackle it!

Hi! Sure, feel free to tackle this.

#self-assign

I'm assuming this should just cover `to_csv`, `to_parquet`, and `to_json`, right? As `to_list` and `to_dict` just return Python objects, `to_pandas` returns a `pandas.DataFrame` and `to_sql` just inserts into a SQL DB, is that right?

Fixed by #6096.
Dataset with loading script does not work in renamed repos

### Describe the bug

My data repository was first called `BramVanroy/hplt-mono-v1-2` but I then renamed to use underscores instead of dashes. However, it seems that `datasets` retrieves the old repo name when it checks whether the repo contains data loading scripts in this line.

https://github.com/huggingface/datasets/blob/6fb6c834f008996c994b0a86c3808d0a33d44525/src/datasets/load.py#L1845

When I print `filename` it returns `hplt-mono-v1-2.py` but the files in the repo are of course `['.gitattributes', 'README.md', 'hplt_mono_v1_2.py']`. So the `filename` is the original reponame instead of the renamed one.

I am not sure if this is a caching issue or not or how I can resolve it.

### Steps to reproduce the bug

```
from datasets import load_dataset


ds = load_dataset(
    "BramVanroy/hplt-mono-v1-2",
    "ky",
    trust_remote_code=True
)
```

### Expected behavior

That the most recent repo name is used when `filename` is generated.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Feature request

Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Motivation

I see huggingface datasets uses Apache Arrow as its backend, it seems to be great, but I'm curious about how it is good compared with other dataset format, like webdataset, what's the pros/cons of them.

### Your contribution

More information

Labels: name: enhancement
Support JSON file with an array of strings

Support loading a dataset from a JSON file with an array of strings.

See: https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1

Labels: name: enhancement

https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1 has been fixed, but how can we check if there are other datasets with the same error, in datasets-server's database? I don't know how to get the list of erroneous cache entries, since we only copied `Error code:   JobManagerCrashedError`, but not the traceback in `details`... Do you remember the error message, or the underlying exception, we had?
Push samples to dataset on hub without having the dataset locally

### Feature request

Say I have the following code:

```
from datasets import Dataset
import pandas as pd
new_data = {
    "column_1": ["value1", "value2"],
    "column_2": ["value3", "value4"],
}
df_new = pd.DataFrame(new_data)
dataset_new = Dataset.from_pandas(df_new)
# add these samples to a remote dataset
```
It would be great to have a way to push dataset_new to a remote dataset that respects the same schema. This way one would not have to do the following:
```
from datasets import load_dataset
dataset = load_dataset('username/dataset_name', use_auth_token='your_hf_token_here')
updated_dataset = dataset['train'].concatenate(dataset_new)
updated_dataset.push_to_hub('username/dataset_name', use_auth_token='your_hf_token_here')
```





### Motivation

No need to download the dataset.

### Your contribution

Maybe this feature already exists, didnt see it though. I do not have the expertise to do this.

Labels: name: enhancement

Hi ! For now I would recommend creating a new Parquet file using `dataset_new.to_parquet()` and upload it to HF using `huggingface_hub` every time you get a new batch of data. You can name the Parquet files `0000.parquet`, `0001.parquet`, etc.

Though maybe make sure to not upload one file per sample since that would be inefficient. You can buffer your data and upload when you have enough new samples for example

This is excellent, thanks!