noamaanMulla-03/datasets-issues huggingface dataset:
Issue with train_test_split maintaining the same underlying PyArrow Table

### Describe the bug

I've been using the train_test_split method in the datasets module to split my HuggingFace Dataset into separate training, validation, and testing subsets. However, I've noticed an issue where the split datasets appear to maintain the same underlying PyArrow Table.

### Steps to reproduce the bug

1. Load any dataset  ```dataset = load_dataset("lhoestq/demo1")``` 
2. Try the next code:
```python
from datasets import Dataset, DatasetDict

train_size = 0.6

split_train = dataset["train"].train_test_split(
    train_size=train_size,
)

separate_dataset_dict = DatasetDict({
    "train": split_train["train"],
    "test": split_train["test"],
})
```
3.  The next code ```print(separate_dataset_dict)``` when printing the dataset it gives the indication that they have 3 and 2 rows respectively.
4. But the next code: 
 ```python
print(len(separate_dataset_dict["train"].data['id']))
print(len(separate_dataset_dict["test"].data['id'])) 
```

 Indicates that both tables still have 5 rows.

### Expected behavior

However, I've noticed that train_test_split["train"].data, test_val_split["train"].data, and test_val_split["test"].data are identical, suggesting that they all point to the same underlying PyArrow Table. This means that the split datasets are not independent, as I expected.

I believe this is a bug in the train_test_split implementation, as I would expect this function to return datasets with separate underlying PyArrow Tables. Could you please help me understand if this is expected behavior, or if there's a workaround to create truly independent split datasets?

I would appreciate any assistance with this issue. Thank you.

### Environment info

I tried in Colab:

- `datasets` version: 2.13.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

and my PC:

- `datasets` version: 2.13.0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
The index column created with .to_sql() is dependent on the batch_size when writing

### Describe the bug

It seems like the "index" column is designed to be unique? The values are only unique per batch. The SQL index is not a unique index.

This can be a problem, for instance, when building a faiss index on a dataset and then trying to match up ids with a sql export.

### Steps to reproduce the bug

```
from datasets import Dataset
import sqlite3
db = sqlite3.connect(":memory:")
nice_numbers = Dataset.from_dict({"nice_number": range(101,106)})
nice_numbers.to_sql("nice1", db, batch_size=1)
nice_numbers.to_sql("nice2", db, batch_size=2)
print(db.execute("select * from nice1").fetchall()) # [(0, 101), (0, 102), (0, 103), (0, 104), (0, 105)]
print(db.execute("select * from nice2").fetchall()) # [(0, 101), (1, 102), (0, 103), (1, 104), (0, 105)]
```

### Expected behavior

I expected the "index" column to be unique

### Environment info

```
 % datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2

zsh: segmentation fault  datasets-cli env
```


Thanks for reporting, @lsb. 

We are investigating it.

On the other hand, please note that in the next `datasets` release, the index will not be created by default (see #5583). If you would like to have it, you will need to explicitly pass `index=True`. 

I think this is low enough priority for me to close this as Won't Fix. If I need any primary keys I can generate them beforehand. Feel free to reopen.
Breaking API change in datasets.list_datasets caused by change in HfApi.list_datasets

### Describe the bug

Hi all,

after an update of the `datasets` library, we observer crashes in our code. We relied on `datasets.list_datasets` returning a `list`. Now, after the API of the HfApi.list_datasets was changed and it returns a `list` instead of an `Iterable`, the `datasets.list_datasets` now sometimes returns a `list` and somesimes an `Iterable`. 

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

Thanks,
Martin 

### Steps to reproduce the bug

Here, the code crashed after we updated the `datasets` library:

```python
# list_datasets no longer returns a list, which leads to an error when one tries to slice it
for datasets.list_datasets(with_details=True)[:limit]:
    ...
```

### Expected behavior

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

### Environment info

Ubuntu 22.04
datasets 2.12.0
`datasets.map` not reusing cached copy by default

### Describe the bug

When I load the dataset from local directory, it's cached copy is picked up after first time. However, for `map` operation, the operation is applied again and cached copy is not picked up. Is there any way to pick cached copy instead of processing it again? The only solution I could think of was to use `save_to_disk` after my last transform and then use that in my DataLoader pipeline. Are there any other solutions for the same?

One more thing, my dataset is occupying 6GB storage memory after I use `map`, is there any way I can reduce that memory usage?


### Steps to reproduce the bug

```
# make sure that dataset decodes audio with correct sampling rate
dataset_sampling_rate = next(iter(self.raw_datasets.values())).features["audio"].sampling_rate
if dataset_sampling_rate != self.feature_extractor.sampling_rate:
    self.raw_datasets = self.raw_datasets.cast_column(
        "audio", datasets.features.Audio(sampling_rate=self.feature_extractor.sampling_rate)
    )

vectorized_datasets = self.raw_datasets.map(
    self.prepare_dataset,
    remove_columns=next(iter(self.raw_datasets.values())).column_names,
    num_proc=self.num_workers,
    desc="preprocess datasets",
)
# filter data that is longer than max_input_length
self.vectorized_datasets = vectorized_datasets.filter(
    self.is_audio_in_length_range,
    num_proc=self.num_workers,
    input_columns=["input_length"],
        )

def prepare_dataset(self, batch):
    # load audio
    sample = batch["audio"]
    inputs = self.feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])

    batch["labels"] = self.tokenizer(batch["target_text"]).input_ids
    return batch

```

### Expected behavior

`map` to use cached copy and if possible an alternative technique to reduce memory usage after using `map`

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2



This can happen when a map transform cannot be hashed deterministically (e.g., an object referenced by the transform changes its state after the first call - an issue with fast tokenizers). The solution is to provide `cache_file_name` in the `map` call to check this file for the cached result instead of relying on the default caching mechanism.
Remove columns in interable dataset

### Feature request

Right now, remove_columns() produces a NotImplementedError for iterable style datasets

### Motivation

It would be great to have the same functionality irrespective of whether one is using an iterable or a map-style dataset

### Your contribution

hope and courage.


Transferring this issue as it's related to the 🤗 Datasets library 

Hi @surya-narayanan! Could you provide some code snippet?

This method has been recently added to the `IterableDataset`, so you need to update the `datasets`' installation (`pip install -U datasets`) to use it.
CSV datasets should only read the CSV data files in the repo

When a no-script dataset has many CSV files and a JPG file, the library infers to use the Csv builder, but tries to read as CSV all files in the repo, also the JPG file.

I think the Csv builder should filter out non-CSV files when reading.

An analogue solution should be implemented for other packaged builders.

Related to:
- https://huggingface.co/datasets/abidlabs/img2text/discussions/1
- https://github.com/gradio-app/gradio/pull/3973#issuecomment-1545409061

CC: @abidlabs @severo 

Labels: name: bug
404 on Datasets Documentation Page

### Describe the bug

Getting a 404 from the Hugging Face Datasets docs page:
https://huggingface.co/docs/datasets/index


### Steps to reproduce the bug

1. Go to URL https://huggingface.co/docs/datasets/index
2. Notice 404 not found

### Expected behavior

URL should either show docs or redirect to new location

### Environment info

hugginface.co


This wasn’t working for me a bit earlier, but it looks to be back up now

We had a minor issue updating the docs after the latest release. It should work now :).
Building C4 takes forever

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Labels: name: bug

Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...: 
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.
Viewing dataset card returns “502 Bad Gateway”

The url is: https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams

I am able to successfully view the “Files and versions” tab: [Confirm-Labs/pile_ngrams_trigrams at main](https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams/tree/main)

Any help would be appreciated! Thanks! I hope this is the right place to report an issue like this.



Can you try again? Maybe there was a minor outage.

Yes, it seems to be working now. In case it's helpful, the outage lasted several days. It was failing as late as yesterday morning. 

we fixed something on the server side, glad it's fixed now
ValueError: Table schema does not match schema used to create file

### Describe the bug

Saving a dataset as parquet fails with a `ValueError: Table schema does not match schema used to create file`  if the dataset was obtained out of a `.select_columns()` call with columns selected out of order.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_parquet("demo.parquet")
```

```shell
>>>
ValueError: Table schema does not match schema used to create file: 
table:
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53 vs. 
file:
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53
```

--- 

I think this is because after the `.select_columns()` call with out of order columns, the output dataset features' schema ends up being out of sync with the schema of the arrow table backing it. 

```python
ds.features.arrow_schema
>>>
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53

ds.data.schema
>>>
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53
```


So when we call `.to_parquet()`, the call behind the scenes to `datasets.io.parquet.ParquetDatasetWriter(...).write()` which initialises the backend `pyarrow.parquet.ParquetWriter` with `schema = self.dataset.features.arrow_schema` triggers `pyarrow` on write when [it checks](https://github.com/apache/arrow/blob/11b140a734a516e436adaddaeb35d23f30dcce44/python/pyarrow/parquet/core.py#L1086-L1090) that the `ParquetWriter` schema matches the schema of the table being written 🙌 

https://github.com/huggingface/datasets/blob/6ed837325cb539a5deb99129e5ad181d0269e050/src/datasets/io/parquet.py#L139-L141


### Expected behavior

The dataset gets successfully saved as parquet. 

*In the same way as it does if saving it as csv:

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_csv("demo.csv")
```

### Environment info

`python==3.11`
`datasets==2.13.1`



We'll do a new release of `datasets` soon to make the fix available :)

In the meantime you can use `datasets` from source (main)

Thank you very much @lhoestq ! 🚀
Config name / split name lost after map with multiproc

### Describe the bug

Performing a `.map` method on a dataset loses it's config name / split name only if run with multiproc

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor
import numpy as np

# load dummy dataset
libri = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

# make train / test splits
libri = libri["validation"].train_test_split(seed=42, shuffle=True, test_size=0.1)

# example feature extractor
model_id = "ntu-spml/distilhubert"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)

sampling_rate = feature_extractor.sampling_rate

libri = libri.cast_column("audio", Audio(sampling_rate=sampling_rate))

max_duration = 30.0

def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=int(feature_extractor.sampling_rate * max_duration),
        truncation=True,
        return_attention_mask=True,
    )
    return inputs

# single proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=1
)

print(10 * "=" ,"Single processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)

# multi proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=2
)

print(10 * "=" ,"Multi processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)
```

**Print Output:**
```
========== Single processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  clean  Split name after:  validation
========== Multi processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  None  Split name after:  None
```

=> we can see that the config/split names are lost in the multiprocessing setting




### Expected behavior

Should retain both config / split names in the multiproc setting

### Environment info

- `datasets` version: 2.13.1.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2


This must be due to DatasetInfo.from_merge which drops them and is used in `concatenate_datasets`.

And you're experiencing this issue because multiprocessing does concatenate the resulting datasets from each process.

Maybe they should be kept if all the subdatasets share the same values for config_name and split

That sounds like a clean workaround!
Why max_shard_size is not supported in load_dataset and passed to download_and_prepare

### Describe the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

What I can to is break the `load_dataset` and use `load_datset_builder` + `download_and_prepare` instead.

### Steps to reproduce the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

### Expected behavior

Users can define the max shard size.

### Environment info

datasets==2.13.1


Can you explain your use case for `max_shard_size`? 

On some systems, there is a limit to the size of a memory-mapped file, so we could consider exposing this parameter in `load_dataset`.

In my use case, users may choose a proper size to balance the cost and benefit of using large shard size. (On azure blob or hdfs which may automatically download the shard from background)

But `load_dataset` doesn't support caching (and reading) Arrow datasets from remote storage. 

`load_datset_builder` + `download_and_prepare` is not equal to `load_dataset`. The latter has one more step, `builder.as_dataset`, that memory-maps Arrow files, which only works for local files.

Thanks. So if I want to use `IterableDataset` and control the size of single arrow file, how should I organize the data loader? Maybe `load_dataset_build` + `download_and_prepare` + `builder.as_dataset` + `dataset.to_iterable_dataset`?

Yes, this should work.

I think we can expose `max_shard_size` in `load_dataset`, so feel free to open a PR.
Stream data concatenation issue

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', 
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path': 
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string', 
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either 
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])    
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0


Hi! 

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```

Thanks it is solved

Hi! 
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!
Getting a 409 error while loading xglue dataset

### Describe the bug

Unable to load xglue dataset

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.load_dataset("xglue", "ntg")
```

> ConnectionError: Couldn't reach https://xglue.blob.core.windows.net/xglue/xglue_full_dataset.tar.gz (error 409)

### Expected behavior

Expected the dataset to load

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


Thanks for reporting, @Praful932.

Let's continue the conversation on the Hub: https://huggingface.co/datasets/xglue/discussions/5
(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

### Describe the bug

M2 MBP can't run
```python
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Steps to reproduce the bug

1. Use M2 MBP
2. Python 3.10.10 from pyenv
3. Run 
```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Expected behavior

Be able to run normally

### Environment info

```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```
OSX: 13.2
CPU: M2



Can you paste the error stack trace?

That is weird. I can't reproduce it again after reboot.
```python
In [2]: import platform

In [3]: platform.platform()
Out[3]: 'macOS-13.2-arm64-arm-64bit'

In [4]: from datasets import load_dataset
   ...:
   ...: jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
Found cached dataset parquet (/Users/sarit/.cache/huggingface/datasets/nomic-ai___parquet/nomic-ai--gpt4all-j-prompt-generations-a3b62015e2e52043/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.25it/s]
```