jamesLeeeeeee/datasets-github-issues huggingface dataset:
Length of table does not accurately reflect the split
Author: amogkam | Pull request: no | State: closed

### Describe the bug

I load a Huggingface Dataset and do `train_test_split`. I'm expecting the underlying table for the dataset to also be split, but it's not.

### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/8068268/83e5768f-8b4c-422a-945c-832a7585afff)


### Expected behavior

The expected behavior is when `len(hf_dataset["train"].data)` should match the length of the train split, and not be the entire unsplit dataset.

### Environment info

datasets 2.10.1
python 3.10.11
add support for storage_options for load_dataset API
Author: janineguo | Pull request: yes | State: closed

to solve the issue in #5880 

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.
File not found for audio dataset
Author: RobertBaruch | Pull request: no | State: open

### Describe the bug

After loading an audio dataset, and looking at a sample entry, the `path` element, which is supposed to be the path to the audio file, doesn't actually exist.


### Steps to reproduce the bug

Run bug.py:

```py
import os.path

from datasets import load_dataset

def run() -> None:
    cv13 = load_dataset(
        "mozilla-foundation/common_voice_13_0",
        "hi",
        split="train",
    )

    print(cv13[0])
    audio_file = cv13[0]["path"]
    if not os.path.exists(audio_file):
        raise ValueError(f'File {audio_file} does not exist.')

if __name__ == "__main__":
    run()
```

The result (on my machine):

```json
{'client_id': '0f018a99663f33afbb7d38aee281fb1afcfd07f9e7acd00383f604e1e17c38d6ed8adf1bd2ccbf927a52c5adefb8ac4b158ce27a7c2ed9581e71202eb302dfb3', 'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'audio': {'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'array': array([ 6.46234854e-26, -1.35709319e-25, -8.07793567e-26, ...,
        1.06425944e-07,  4.46417090e-08,  2.61451660e-09]), 'sampling_rate': 48000}, 'sentence': 'हमने उसका जन्मदिन मनाया।', 'up_votes': 2, 'down_votes': 0, 'age': '', 'gender': '', 'accent': '', 'locale': 'hi', 'segment': '' ', 'variant': ''}
```

```txt
Traceback (most recent call last):
  File "F:\eo-reco\bug.py", line 18, in <module>
    run()
  File "F:\eo-reco\bug.py", line 15, in run
    raise ValueError(f'File {audio_file} does not exist.')
ValueError: File C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\common_voice_hi_26008353.mp3 does not exist.
```


### Expected behavior

The `path` element points to the correct file, which happens to be:

```
C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\hi_train_0\common_voice_hi_26008353.mp3
```

That is, there's an extra directory `hi_train_0` that is not in the `path` element.


### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
-
I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.
Author: cjt222 | Pull request: no | State: closed

### Describe the bug

File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
Downloading and preparing dataset json/default to /home/kas/diffusers/examples/dreambooth/cache_data/datasets/json/default-acf423d8c6ef99d0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s] Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 84.35it/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s] for _, table in generator:
File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 258, in pyarrow._json.read_json
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 27.72it/s]
Generating train split: 0 examples [00:00, ? examples/s] File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 125, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2390448764

### Steps to reproduce the bug

1、data_files = ["1.json", "2.json", "3.json"]
2、dataset = load_dataset('json', data_files=data_files)

### Expected behavior

Read the dataset normally.



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.3.5
Missing elements in `map` a batched dataset
Author: sachinruk | Pull request: no | State: closed

### Describe the bug

As outlined [here](https://discuss.huggingface.co/t/length-error-using-map-with-datasets/40969/3?u=sachin), the following collate function drops 5 out of possible 6 elements in the batch (it is 6 because out of the eight, two are bad links in laion). A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

The weirdest part is when inspecting the sizes of the tensors as shown below, both `tokenized_captions["input_ids"]` and `image_features` show the correct shapes. Simply the output only has one element (with the batch dimension squeezed out).

```python
class CollateFn:        
    def get_image(self, url):
        try:
            response = requests.get(url)
            return Image.open(io.BytesIO(response.content)).convert("RGB")
        except PIL.UnidentifiedImageError:
            logger.info(f"Reading error: Could not transform f{url}")
            return None
        except requests.exceptions.ConnectionError:
            logger.info(f"Connection error: Could not transform f{url}")
            return None
        
        
    def __call__(self, batch):
        images = [self.get_image(url) for url in batch["url"]]
        captions = [caption for caption, image in zip(batch["caption"], images) if image is not None]
        images = [image for image in images if image is not None]
        
        tokenized_captions = tokenizer(
            captions,
            padding="max_length",
            truncation=True,
            max_length=tokenizer.model_max_length,
            return_tensors="pt",
        )
        
        image_features = torch.stack([torch.Tensor(feature_extractor(image)["pixel_values"][0]) for image in images])
#         import pdb; pdb.set_trace()
        return {"input_ids": tokenized_captions["input_ids"], "images": image_features}

collate_fn = CollateFn()
laion_ds = datasets.load_dataset("laion/laion400m", split="train", streaming=True)
laion_ds_batched = laion_ds.map(collate_fn, batched=True, batch_size=8, remove_columns=next(iter(laion_ds)).keys())
```

### Steps to reproduce the bug

A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

### Expected behavior

Would expect `next(iter(laion_ds_batched))` to produce two tensors of shape `(batch_size, 77)` and `batch_size, image_shape`.

### Environment info

datasets==2.12.0
python==3.10
Cannot use both set_format and set_transform
Author: ybouane | Pull request: no | State: closed

### Describe the bug

I need to process some data using the set_transform method but I also need the data to be formatted for pytorch before processing it.

I don't see anywhere in the documentation something that says that both methods cannot be used at the same time.

### Steps to reproduce the bug

```
from datasets import load_dataset
ds = load_dataset("mnist", split="train")
ds.set_format(type="torch")
def transform(entry):
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

### Expected behavior

It should print the pytorch tensor image as a double, but it errors because "entry" in the transform function doesn't receive a pytorch tensor to begin with, it receives a PIL Image -> entry.double() errors because entry isn't a pytorch tensor.

### Environment info
Latest versions.


### Note:
It would be at least handy to have access to a function that can do the dataset.set_format in the set_transform function.

Something like:
```
from datasets import load_dataset, do_format
ds = load_dataset("mnist", split="train")
def transform(entry):
    entry = do_format(entry, type="torch")
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```
Use more efficient and idiomatic way to construct list.
Author: ttsugriy | Pull request: yes | State: closed

Using `*` is ~2X faster according to [benchmark](https://colab.research.google.com/gist/ttsugriy/c964a2604edf70c41911b10335729b6a/for-vs-mult.ipynb) with just 4 patterns. This doesn't matter much since this tiny difference is not going to be noticeable, but why not?
Unbearably slow sorting on big mapped datasets
Author: maximxlss | Pull request: no | State: open

### Describe the bug

For me, with ~40k lines, sorting took 3.5 seconds on a flattened dataset (including the flatten operation) and 22.7 seconds on a mapped dataset (right after sharding), which is about x5 slowdown. Moreover, it seems like it slows down exponentially with bigger datasets (wasn't able to sort 700k lines at all, with flattening takes about a minute).

### Steps to reproduce the bug

```Python
from datasets import load_dataset
import time

dataset = load_dataset("xnli", "en", split="train")

dataset = dataset.shard(10, 0)

print(len(dataset))

t = time.time()

# dataset = dataset.flatten_indices()  # uncomment this line and it's fast

dataset = dataset.sort("label", reverse=True, load_from_cache_file=False)

print(f"finished in {time.time() - t:.4f} seconds")

```

### Expected behavior

Expect sorting to take the same or less time than flattening and then sorting.

### Environment info

- `datasets` version: 2.12.1.dev0 (same with 2.12.0 too)
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
Could you unpin responses version?
Author: kenimou | Pull request: no | State: closed

### Describe the bug

Could you unpin [this](https://github.com/huggingface/datasets/blob/main/setup.py#L139) or move it to test requirements? This is a testing library and we also use it for our tests as well. We do not want to use a very outdated version.

### Steps to reproduce the bug

could not install this library due to dependency conflict.

### Expected behavior

can install datasets

### Environment info

linux 64
Offer an alternative to Iterable Dataset that allows lazy loading and processing while skipping batches efficiently
Author: Hubert-Bonisseur | Pull request: no | State: open

### Feature request

I would like a way to resume training from a checkpoint without waiting for a very long time when using an iterable dataset.

### Motivation

I am training models on the speech-recognition task. I have very large datasets that I can't comfortably store on a disk and also quite computationally intensive audio processing to do. As a result I want to load data from my remote when it is needed and perform all processing on the fly.

I am currently using the iterable dataset feature of _datasets_. It does everything I need with one exception. My issue is that when resuming training at a step n, we have to download all the data and perform the processing of steps < n, just to get the iterable at the right step. In my case it takes almost as long as training for the same steps, which make resuming training from a checkpoint useless in practice.

I understand that the nature of iterators make it probably nearly impossible to quickly resume training.

I thought about a possible solution nonetheless : 

I could in fact index my large dataset and make it a mapped dataset. Then I could use set_transform to perform the processing on the fly. Finally, if I'm not mistaken, the _accelerate_ package allows to [skip steps efficiently](https://github.com/huggingface/accelerate/blob/a73898027a211c3f6dc4460351b0ec246aa824aa/src/accelerate/data_loader.py#L827) for a mapped dataset.

Is it possible to lazily load samples of a mapped dataset ? I'm used to [dataset scripts](https://huggingface.co/docs/datasets/dataset_script), maybe something can be done there.
If not, I could do it using a plain _Pytorch_ dataset. Then I would need to convert it to a _datasets_' dataset to get all the features of _datasets_. Is it something possible ?

### Your contribution

I could provide a PR to allow lazy loading of mapped dataset or the conversion of a mapped _Pytorch_ dataset into a _Datasets_ dataset if you think it is an useful new feature.
Validate name parameter in make_file_instructions
Author: albertvillanova | Pull request: yes | State: closed

Validate `name` parameter in `make_file_instructions`.

This way users get more informative error messages, instead of:
```stacktrace
.../huggingface/datasets/src/datasets/arrow_reader.py in make_file_instructions(name, split_infos, instruction, filetype_suffix, prefix_path)
    110     name2len = {info.name: info.num_examples for info in split_infos}
    111     name2shard_lengths = {info.name: info.shard_lengths for info in split_infos}
--> 112     name2filenames = {
    113         info.name: filenames_for_dataset_split(
    114             path=prefix_path,

.../huggingface/datasets/src/datasets/arrow_reader.py in <dictcomp>(.0)
    111     name2shard_lengths = {info.name: info.shard_lengths for info in split_infos}
    112     name2filenames = {
--> 113         info.name: filenames_for_dataset_split(
    114             path=prefix_path,
    115             dataset_name=name,

.../huggingface/datasets/src/datasets/naming.py in filenames_for_dataset_split(path, dataset_name, split, filetype_suffix, shard_lengths)
     68 
     69 def filenames_for_dataset_split(path, dataset_name, split, filetype_suffix=None, shard_lengths=None):
---> 70     prefix = filename_prefix_for_split(dataset_name, split)
     71     prefix = os.path.join(path, prefix)
     72 

.../huggingface/datasets/src/datasets/naming.py in filename_prefix_for_split(name, split)
     52 
     53 def filename_prefix_for_split(name, split):
---> 54     if os.path.basename(name) != name:
     55         raise ValueError(f"Should be a dataset name, not a path: {name}")
     56     if not re.match(_split_re, split):

.../lib/python3.9/posixpath.py in basename(p)
    140 def basename(p):
    141     """Returns the final component of a pathname"""
--> 142     p = os.fspath(p)
    143     sep = _get_sep(p)
    144     i = p.rfind(sep) + 1

TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Related to #5895.
Relax `ci.yml` trigger for `pull_request` based on modified paths
Author: alvarobartt | Pull request: yes | State: open

## What's in this PR?

As of a previous PR at #5902, I've seen that the CI was automatically trigger on any file, in that case when modifying a Jupyter Notebook (.ipynb), which IMO could be skipped, as the modification on the Jupyter Notebook has no effect/impact on the `ci.yml` outcome. So this PR controls the paths that trigger the `ci.yml` to avoid wasting resources when not needed.

## What's pending in this PR?

I would like to confirm whether this should affect both `push` and `pull_request`, since just modifications in those files won't change the `ci.yml` outcome, so maybe it's worth skipping it too in the `push` trigger.
Fix `Overview.ipynb` & detach Jupyter Notebooks from `datasets` repository
Author: alvarobartt | Pull request: yes | State: closed

## What's in this PR?

This PR solves #5887 since there was a mismatch between the tokenizer and the model used, since the tokenizer was `bert-base-cased` while the model was `distilbert-base-case` both for the PyTorch and TensorFlow alternatives. Since DistilBERT doesn't use/need the `token_type_ids`, the `**batch` was failing, as the batch contained `input_ids`, `attention_mask`, `token_type_ids`, `start_positions` and `end_positions`, and `token_type_ids` was not required.

Besides that, at the end `seqeval` was being used to evaluate the model predictions, and just `evaluate` was being installed, so I've also included the `seqeval` installation.

Finally, I've re-run everything in Google Colab, and every cell was successfully executed!

## What was done on top of the original PR?

Based on the comments from @mariosasko and @stevhliu, I've updated the contents of this PR to also review the `quickstart.mdx` and update what was needed, besides that, we may eventually move the `Overview.ipynb` dataset to `huggingface/notebooks` following @stevhliu suggestions.
Make prepare_split more robust if errors in metadata dataset_info splits
Author: albertvillanova | Pull request: yes | State: closed

This PR uses `split_generator.split_info` as default value for `split_info` if any exception is raised while trying to get `split_generator.name` from `self.info.splits` (this may happen if there is any error in the metadata dataset_info splits).

Please note that `split_info` is only used by the logger.

Fix #5895 if passed `verification_mode="no_checks"`:
```python
ds = load_dataset(
    "ArmelR/stack-exchange-instruction", 
    data_dir="data/finetune", 
    split="train", 
    verification_mode="no_checks", 
    revision="c609f1caade5cfbf3b9fe9cfa17d7cb000b457bd",
)
```
Loading The flores data set for specific language
Author: 106AbdulBasit | Pull request: no | State: closed

### Describe the bug

I am trying to load the Flores data set

the code which is given is
```

from datasets import load_dataset

dataset = load_dataset("facebook/flores")

```
This gives the error of config name 

""ValueError: Config name is missing"

Now if I add some config it gives me the some error

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

How I can load the data of the specific language ?

Couldn't find any tutorial

any one can help me out?

### Steps to reproduce the bug

step one load the data set 

`from datasets import load_dataset

dataset = load_dataset("facebook/flores")`

it gives the error of config

once config is given 

it gives the error of

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

### Expected behavior

Data set should be loaded but  I am receiving error

### Environment info

Datasets , python ,
HuggingFace does not cache downloaded files aggressively/early enough
Author: geajack | Pull request: no | State: open

### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10
The dir name and split strings are confused when loading ArmelR/stack-exchange-instruction dataset
Author: DongHande | Pull request: no | State: closed

### Describe the bug

When I load the ArmelR/stack-exchange-instruction dataset, I encounter a bug that may be raised by confusing the dir name string and the split string about the dataset. 

When I use the script "datasets.load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)", it fails. But it succeeds when I add the "streaming = True" parameter. 

The website of the dataset is https://huggingface.co/datasets/ArmelR/stack-exchange-instruction/ .

The traceback logs are as below: 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 1706, in _prepare_split
    split_info = self.info.splits[split_generator.name]
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/splits.py", line 530, in __getitem__
    instructions = make_file_instructions(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 112, in make_file_instructions
    name2filenames = {
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 113, in <dictcomp>
    info.name: filenames_for_dataset_split(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 70, in filenames_for_dataset_split
    prefix = filename_prefix_for_split(dataset_name, split)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 54, in filename_prefix_for_split
    if os.path.basename(name) != name:
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/posixpath.py", line 142, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType





### Steps to reproduce the bug

1. import datasets library function: ```from datasets import load_dataset```
2. load dataset: ```ds=load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)```

### Expected behavior

The dataset can be loaded successfully without the streaming setting. 

### Environment info

Linux, 
python=3.9
datasets=2.12.0