mengta666/huggingface_github huggingface dataset:
Issue: Only two cores are getting used in sagemaker with pytorch 3.10 kernel

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel

Comments:
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625

From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.

This is too specific and unrelated to `datasets`, so this shouldn't be fixed here.

@mariosasko @mmr-crexi  I had the exact same problem on my kubernetes cluster. the datasets subprocess only user 1 and 17 core
Issue: Viewing dataset card returns “502 Bad Gateway”

The url is: https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams

I am able to successfully view the “Files and versions” tab: [Confirm-Labs/pile_ngrams_trigrams at main](https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams/tree/main)

Any help would be appreciated! Thanks! I hope this is the right place to report an issue like this.


Comments:
Can you try again? Maybe there was a minor outage.

Yes, it seems to be working now. In case it's helpful, the outage lasted several days. It was failing as late as yesterday morning. 

we fixed something on the server side, glad it's fixed now
Issue: Docs: make "repository structure" easier to find

The page https://huggingface.co/docs/datasets/repository_structure explains how to create a simple repository structure without a dataset script.
It's the simplest way to create a dataset and should be easier to find, particularly on the docs' first pages.

Comments:
Loading a local dataset also works the same way when `data_files` are not specified, so I agree we should make this info  easier to discover 

cc @stevhliu 

Is this issue open? If so, I will self assign. 

@benjaminbrown038 Yes, it is. Maybe @stevhliu can give some pointers on improving this doc page's discoverability.

I think we can add a version of the [Main use-case](https://huggingface.co/docs/datasets/repository_structure#main-usecase) section to the [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset) tutorial. 

Currently, it doesn't tell you *how* to structure the repository; it only tells you how to create it. So adding the "main use-case" will help bridge the gap and make it easier to find. We should also add a link to the [Structure your repository](https://huggingface.co/docs/datasets/repository_structure) guide for users who want to learn about the other options.

#self-assign
Issue: Config name / split name lost after map with multiproc

### Describe the bug

Performing a `.map` method on a dataset loses it's config name / split name only if run with multiproc

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor
import numpy as np

# load dummy dataset
libri = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

# make train / test splits
libri = libri["validation"].train_test_split(seed=42, shuffle=True, test_size=0.1)

# example feature extractor
model_id = "ntu-spml/distilhubert"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)

sampling_rate = feature_extractor.sampling_rate

libri = libri.cast_column("audio", Audio(sampling_rate=sampling_rate))

max_duration = 30.0

def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=int(feature_extractor.sampling_rate * max_duration),
        truncation=True,
        return_attention_mask=True,
    )
    return inputs

# single proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=1
)

print(10 * "=" ,"Single processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)

# multi proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=2
)

print(10 * "=" ,"Multi processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)
```

**Print Output:**
```
========== Single processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  clean  Split name after:  validation
========== Multi processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  None  Split name after:  None
```

=> we can see that the config/split names are lost in the multiprocessing setting




### Expected behavior

Should retain both config / split names in the multiproc setting

### Environment info

- `datasets` version: 2.13.1.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2

Comments:
This must be due to DatasetInfo.from_merge which drops them and is used in `concatenate_datasets`.

And you're experiencing this issue because multiprocessing does concatenate the resulting datasets from each process.

Maybe they should be kept if all the subdatasets share the same values for config_name and split

That sounds like a clean workaround!
Issue: read metric glue.py from local file 

### Describe the bug

Currently, The server is off-line.  I am using the glue metric from the local file downloaded from the hub. 
I download / cached datasets using `load_dataset('glue','sst2', cache_dir='/xxx')` to cache them and then in the off-line mode, I use `load_dataset('xxx/glue.py','sst2', cache_dir='/xxx')`. I can successfully reuse cached datasets.

My problem is about the load_metric. 
When I run `load_dataset('xxx/glue_metric.py','sst2',cache_dir='/xxx')` , it returns 

`  File "xx/lib64/python3.9/site-packages/datasets/utils/deprecation_utils.py", line 46, in wrapper
    return deprecated_function(*args, **kwargs)
  File "xx//lib64/python3.9/site-packages/datasets/load.py", line 1392, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable`

Thanks in advance for help! 
### Steps to reproduce the bug

N/A

### Expected behavior

N/A

### Environment info

`datasets == 2.12.0`

Comments:
Sorry, I solve this by call `evaluate.load('glue_metric.py','sst-2')`
Issue: What is the Right way to use discofuse dataset??

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?

Comments:
Thanks for opening https://huggingface.co/datasets/discofuse/discussions/3, let's continue the discussion over there if you don't mind

I have posted there also sir, please check
@lhoestq
Issue: Return the audio filename when decoding fails due to corrupt files

### Feature request

Return the audio filename when the audio decoding fails. Although currently there are some checks for mp3 and opus formats with the library version there are still cases when the audio decoding could fail, eg. Corrupt file. 

### Motivation

When you try to load an object file dataset and the decoding fails you can't know which file is corrupt
```

raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7f5ab7e38290>: Format not recognised.
```

### Your contribution

Make a PR to Add exceptions for LIbsndfileError to return the audio filename or path when soundfile decoding fails.

Comments:
Hi ! The audio data don't always exist as files on disk - the blobs are often stored in the Arrow files. For now I'd suggest disabling decoding with `.cast_column("audio", Audio(decode=False))` and apply your own decoding that handles corrupted files (maybe to filter them out ?)

cc @sanchit-gandhi since it's related to our discussion about allowing users to make decoding return `None` and show a warning when there are corrupted files

Thanks @lhoestq, I wasn't aware of the decode flag. It makes more sense as you say to show a warning when there are corrupted files together with some metadata of the file that allows to filter them from the dataset.

My workaround was to catch the LibsndfileError and generate a dummy audio with an unsual sample rate to filter it later. However returning `None` seems better. 

`try:
    array, sampling_rate = sf.read(file)
except sf.LibsndfileError:
    print("bad file")
    array = np.array([0.0])
    sampling_rate = 99.000`
Issue: Failing to upload dataset to the hub

### Describe the bug

Trying to upload a dataset of hundreds of thousands of audio samples (the total volume is not very large, 60 gb) to the hub with push_to_hub, it doesn't work.
From time to time one piece of the data (parquet) gets pushed and then I get RemoteDisconnected even though my internet is stable.
Please help.
I'm trying to upload the dataset for almost a week.
Thanks

### Steps to reproduce the bug

not relevant 

### Expected behavior

Be able to upload thedataset

### Environment info

python: 3.9

Comments:
Hi ! Feel free to re-run your code later, it will resume automatically where you left

Tried many times in the last 2 weeks, problem remains.

Alternatively you can save your dataset in parquet files locally and upload them to the hub manually

```python
from tqdm import tqdm
num_shards = 60
for index in tqdm(range(num_shards)):
    ds.shard(num_shards=num_shards, index=index, contiguous=True).to_parquet(f"{index:05d}.parquet")
````
Issue: `datasets.map` not reusing cached copy by default

### Describe the bug

When I load the dataset from local directory, it's cached copy is picked up after first time. However, for `map` operation, the operation is applied again and cached copy is not picked up. Is there any way to pick cached copy instead of processing it again? The only solution I could think of was to use `save_to_disk` after my last transform and then use that in my DataLoader pipeline. Are there any other solutions for the same?

One more thing, my dataset is occupying 6GB storage memory after I use `map`, is there any way I can reduce that memory usage?


### Steps to reproduce the bug

```
# make sure that dataset decodes audio with correct sampling rate
dataset_sampling_rate = next(iter(self.raw_datasets.values())).features["audio"].sampling_rate
if dataset_sampling_rate != self.feature_extractor.sampling_rate:
    self.raw_datasets = self.raw_datasets.cast_column(
        "audio", datasets.features.Audio(sampling_rate=self.feature_extractor.sampling_rate)
    )

vectorized_datasets = self.raw_datasets.map(
    self.prepare_dataset,
    remove_columns=next(iter(self.raw_datasets.values())).column_names,
    num_proc=self.num_workers,
    desc="preprocess datasets",
)
# filter data that is longer than max_input_length
self.vectorized_datasets = vectorized_datasets.filter(
    self.is_audio_in_length_range,
    num_proc=self.num_workers,
    input_columns=["input_length"],
        )

def prepare_dataset(self, batch):
    # load audio
    sample = batch["audio"]
    inputs = self.feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])

    batch["labels"] = self.tokenizer(batch["target_text"]).input_ids
    return batch

```

### Expected behavior

`map` to use cached copy and if possible an alternative technique to reduce memory usage after using `map`

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2


Comments:
This can happen when a map transform cannot be hashed deterministically (e.g., an object referenced by the transform changes its state after the first call - an issue with fast tokenizers). The solution is to provide `cache_file_name` in the `map` call to check this file for the cached result instead of relying on the default caching mechanism.
Issue: Importing PyTorch reduces multiprocessing performance for map

### Describe the bug

I noticed that the performance of my dataset preprocessing with `map(...,num_proc=32)` decreases when PyTorch is imported.

### Steps to reproduce the bug

I created two example scripts to reproduce this behavior:

```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
Takes around 4 seconds on my machine.

While the same code, but with an `import torch`:
```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
import torch
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
takes around 22 seconds.



### Expected behavior

I would expect that the import of torch to not have such a significant effect on the performance of map using multiprocessing.


### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
- torch: 2.0.1

Comments:
Hi! The times match when I run this code locally or on Colab.

Also, we use `multiprocess`, not `multiprocessing`, for parallelization, and torch's `__init__.py` (executed on `import torch` ) slightly modifies the latter.

Hey Mariosasko,

Thanks for looking into it. We further did some investigations after your comment and figured out it's only affecting some hardware/software configurations with the `pytorch` installation of `conda-forge`. Based on this we found the following issue in PyTorch: https://github.com/pytorch/pytorch/issues/102269 with a quick fix for now.

Since it seems to be a deeper issue with forking processes, the difference between`multiprocess` and `multiprocessing` didn't make a difference.

Closing this, since the issue comes from `pytorch` not `dataset`.
Issue: Length of table does not accurately reflect the split

### Describe the bug

I load a Huggingface Dataset and do `train_test_split`. I'm expecting the underlying table for the dataset to also be split, but it's not.

### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/8068268/83e5768f-8b4c-422a-945c-832a7585afff)


### Expected behavior

The expected behavior is when `len(hf_dataset["train"].data)` should match the length of the train split, and not be the entire unsplit dataset.

### Environment info

datasets 2.10.1
python 3.10.11

Comments:
As already replied by @lhoestq (private channel):
> `.train_test_split` (as well as `.shard`, `.select`) doesn't create a new arrow table to save time and disk space. Instead, it uses an indices mapping on top of the table that locate which examples are part of train or test.

This is an optimization that we don't plan to "fix", so I'm closing this issue.
Issue: File not found for audio dataset

### Describe the bug

After loading an audio dataset, and looking at a sample entry, the `path` element, which is supposed to be the path to the audio file, doesn't actually exist.


### Steps to reproduce the bug

Run bug.py:

```py
import os.path

from datasets import load_dataset

def run() -> None:
    cv13 = load_dataset(
        "mozilla-foundation/common_voice_13_0",
        "hi",
        split="train",
    )

    print(cv13[0])
    audio_file = cv13[0]["path"]
    if not os.path.exists(audio_file):
        raise ValueError(f'File {audio_file} does not exist.')

if __name__ == "__main__":
    run()
```

The result (on my machine):

```json
{'client_id': '0f018a99663f33afbb7d38aee281fb1afcfd07f9e7acd00383f604e1e17c38d6ed8adf1bd2ccbf927a52c5adefb8ac4b158ce27a7c2ed9581e71202eb302dfb3', 'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'audio': {'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'array': array([ 6.46234854e-26, -1.35709319e-25, -8.07793567e-26, ...,
        1.06425944e-07,  4.46417090e-08,  2.61451660e-09]), 'sampling_rate': 48000}, 'sentence': 'हमने उसका जन्मदिन मनाया।', 'up_votes': 2, 'down_votes': 0, 'age': '', 'gender': '', 'accent': '', 'locale': 'hi', 'segment': '' ', 'variant': ''}
```

```txt
Traceback (most recent call last):
  File "F:\eo-reco\bug.py", line 18, in <module>
    run()
  File "F:\eo-reco\bug.py", line 15, in run
    raise ValueError(f'File {audio_file} does not exist.')
ValueError: File C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\common_voice_hi_26008353.mp3 does not exist.
```


### Expected behavior

The `path` element points to the correct file, which happens to be:

```
C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\hi_train_0\common_voice_hi_26008353.mp3
```

That is, there's an extra directory `hi_train_0` that is not in the `path` element.


### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
- 

Comments:
load_dataset () did not work for loading local files either
Issue: I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.

### Describe the bug

File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
Downloading and preparing dataset json/default to /home/kas/diffusers/examples/dreambooth/cache_data/datasets/json/default-acf423d8c6ef99d0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s] Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 84.35it/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s] for _, table in generator:
File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 258, in pyarrow._json.read_json
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 27.72it/s]
Generating train split: 0 examples [00:00, ? examples/s] File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 125, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2390448764

### Steps to reproduce the bug

1、data_files = ["1.json", "2.json", "3.json"]
2、dataset = load_dataset('json', data_files=data_files)

### Expected behavior

Read the dataset normally.



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.3.5

Comments:
Thanks for reporting, @cjt222.

What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead. 

> Thanks for reporting, @cjt222.
> 
> What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.

Thanks! I have encountered similar problems. I modify the json format from list to line and works!
Issue: Missing elements in `map` a batched dataset

### Describe the bug

As outlined [here](https://discuss.huggingface.co/t/length-error-using-map-with-datasets/40969/3?u=sachin), the following collate function drops 5 out of possible 6 elements in the batch (it is 6 because out of the eight, two are bad links in laion). A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

The weirdest part is when inspecting the sizes of the tensors as shown below, both `tokenized_captions["input_ids"]` and `image_features` show the correct shapes. Simply the output only has one element (with the batch dimension squeezed out).

```python
class CollateFn:        
    def get_image(self, url):
        try:
            response = requests.get(url)
            return Image.open(io.BytesIO(response.content)).convert("RGB")
        except PIL.UnidentifiedImageError:
            logger.info(f"Reading error: Could not transform f{url}")
            return None
        except requests.exceptions.ConnectionError:
            logger.info(f"Connection error: Could not transform f{url}")
            return None
        
        
    def __call__(self, batch):
        images = [self.get_image(url) for url in batch["url"]]
        captions = [caption for caption, image in zip(batch["caption"], images) if image is not None]
        images = [image for image in images if image is not None]
        
        tokenized_captions = tokenizer(
            captions,
            padding="max_length",
            truncation=True,
            max_length=tokenizer.model_max_length,
            return_tensors="pt",
        )
        
        image_features = torch.stack([torch.Tensor(feature_extractor(image)["pixel_values"][0]) for image in images])
#         import pdb; pdb.set_trace()
        return {"input_ids": tokenized_captions["input_ids"], "images": image_features}

collate_fn = CollateFn()
laion_ds = datasets.load_dataset("laion/laion400m", split="train", streaming=True)
laion_ds_batched = laion_ds.map(collate_fn, batched=True, batch_size=8, remove_columns=next(iter(laion_ds)).keys())
```

### Steps to reproduce the bug

A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

### Expected behavior

Would expect `next(iter(laion_ds_batched))` to produce two tensors of shape `(batch_size, 77)` and `batch_size, image_shape`.

### Environment info

datasets==2.12.0
python==3.10

Comments:
Hi ! in your code batching is **only used within** `map`, to process examples in batch. The dataset itself however is not batched and returns elements one by one.

To iterate on batches, you can do
```python
for batch in dataset.iter(batch_size=8):
    ...
```
Issue: Cannot use both set_format and set_transform

### Describe the bug

I need to process some data using the set_transform method but I also need the data to be formatted for pytorch before processing it.

I don't see anywhere in the documentation something that says that both methods cannot be used at the same time.

### Steps to reproduce the bug

```
from datasets import load_dataset
ds = load_dataset("mnist", split="train")
ds.set_format(type="torch")
def transform(entry):
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

### Expected behavior

It should print the pytorch tensor image as a double, but it errors because "entry" in the transform function doesn't receive a pytorch tensor to begin with, it receives a PIL Image -> entry.double() errors because entry isn't a pytorch tensor.

### Environment info
Latest versions.


### Note:
It would be at least handy to have access to a function that can do the dataset.set_format in the set_transform function.

Something like:
```
from datasets import load_dataset, do_format
ds = load_dataset("mnist", split="train")
def transform(entry):
    entry = do_format(entry, type="torch")
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

Comments:
Currently, it's not possible to chain `set_format`/`set_transform` calls (plus, this is a breaking change if we decide to implement it), so I see two possible solutions:
* using `set_format`/`set_transform` for the 1st transform and then passing the transformed example/batch to the 2nd transform
* implementing and registering a custom formatter (the relevant code is [here](https://github.com/huggingface/datasets/tree/main/src/datasets/formatting))

Btw, your example requires a single `set_format` call:
```python
ds.set_format("torch", columns=["image"], output_all_columns=True, dtype=torch.double)
```

Hey Mario,
Thanks, for getting back to me. the toDouble was just an example my real life case requires many more transforms.

What do you mean by:
> using set_format/set_transform for the 1st transform and then passing the transformed example/batch to the 2nd transform

How would that go, I thought you can't chain them?

As for the custom formatter, is it possible to reference an existing formatter, in my case `torch_formatter` inside of my custom formatter?

maybe I can inherit from it and just call `super.recursive_tensorize()`?

> How would that go, I thought you can't chain them?

Yes, they cannot be chained. This is what I meant:
```python
ds.set_transform(first_transform)
# calling the 2nd transform on each accessed batch
second_transform(ds[2:3])
```

> As for the custom formatter, is it possible to reference an existing formatter, in my case torch_formatter inside of my custom formatter?
>
>maybe I can inherit from it and just call super.recursive_tensorize()?

Yes, subclassing makes the most sense.

Great, thank you for the details.

https://github.com/huggingface/datasets/issues/6012
Issue: Unbearably slow sorting on big mapped datasets

### Describe the bug

For me, with ~40k lines, sorting took 3.5 seconds on a flattened dataset (including the flatten operation) and 22.7 seconds on a mapped dataset (right after sharding), which is about x5 slowdown. Moreover, it seems like it slows down exponentially with bigger datasets (wasn't able to sort 700k lines at all, with flattening takes about a minute).

### Steps to reproduce the bug

```Python
from datasets import load_dataset
import time

dataset = load_dataset("xnli", "en", split="train")

dataset = dataset.shard(10, 0)

print(len(dataset))

t = time.time()

# dataset = dataset.flatten_indices()  # uncomment this line and it's fast

dataset = dataset.sort("label", reverse=True, load_from_cache_file=False)

print(f"finished in {time.time() - t:.4f} seconds")

```

### Expected behavior

Expect sorting to take the same or less time than flattening and then sorting.

### Environment info

- `datasets` version: 2.12.1.dev0 (same with 2.12.0 too)
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Comments:
Hi ! `shard` currently returns a slow dataset by default, with examples evenly distributed in the dataset.

You can get a fast dataset using `contiguous=True` (which should be the default imo):

```python
dataset = dataset.shard(10, 0, contiguous=True)
```

This way you don't need to flatten_indices() and sort should be fast as well

@lhoestq 

> contiguous=True (which should be the default imo)

For `IterableDataset`, it's not possible to implement contiguous sharding without knowing the number of examples in advance, so setting the default value to `contiguous=True` would result in an inconsistency between `Dataset` and `IterableDataset` (when we add `IterableDataset.shard`)

Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

If the dataset is made of one shard it's indeed not possible to shard it contiguously though

> Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

But sharding an iterable dataset by sharding its `gen_kwargs` would still yield approximate shards(not equal to `Dataset.shard`), no? 

Yes indeed !

I understand the issue doesn't exist with non-mapped datasets, but if flattening is so much more efficient than sorting the indices, that's an issue in itself.

There are plenty of issues people posted for which the root cause turns out to be the same. It seems like mapped datasets are terribly inefficient. I think I saw some issue like that somewhere (about the mapped datasets in general), but can't find it now.

Maybe indices should be flattened before any additional processing, then.
Issue: Loading The flores data set for specific language

### Describe the bug

I am trying to load the Flores data set

the code which is given is
```

from datasets import load_dataset

dataset = load_dataset("facebook/flores")

```
This gives the error of config name 

""ValueError: Config name is missing"

Now if I add some config it gives me the some error

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

How I can load the data of the specific language ?

Couldn't find any tutorial

any one can help me out?

### Steps to reproduce the bug

step one load the data set 

`from datasets import load_dataset

dataset = load_dataset("facebook/flores")`

it gives the error of config

once config is given 

it gives the error of

"HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'facebook/flores, 'ace_Arab''.
"

### Expected behavior

Data set should be loaded but  I am receiving error

### Environment info

Datasets , python , 

Comments:
got that the syntax is like this

dataset = load_dataset("facebook/flores", "ace_Arab")
Issue: HuggingFace does not cache downloaded files aggressively/early enough

### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10

Comments:
I also faced this. Any update?

We've dropped the `apache-beam` dependency in https://huggingface.co/datasets/wikipedia/discussions/19, so you should no longer get this error.
Issue: User access requests with manual review do not notify the dataset owner

### Describe the bug

When a user access requests are enabled, and new requests are set to Manual Review, the dataset owner should be notified of the pending requests. However, instead, currently nothing happens, and so the dataset request can go unanswered for quite some time until the owner happens to check that particular dataset's Settings pane.

### Steps to reproduce the bug

1. Enable a dataset's user access requests
2. Set to Manual Review
3. Ask another HF user to request access to the dataset
4. Dataset owner is not notified

### Expected behavior

The dataset owner should receive some kind of notification, perhaps in their HF site inbox, or by email, when a dataset access request is made and manual review is enabled.

### Environment info

n/a

Comments:
cc @SBrandeis

I think this has been addressed.

Please open a new issue if you are still not getting notified.
Issue: Use work-stealing algorithm when parallel computing

### Feature request

when i used Dataset.map api to process data concurrently, i found that
 it gets slower and slower as it gets closer to completion. Then i read the source code of arrow_dataset.py and found that it shard the dataset and use multiprocessing pool to execute each shard.It may cause the slowest task to drag out the entire program's execution time,especially when processing huge dataset.

### Motivation

using work-stealing algorithm instead of sharding and parallel computing to optimize performance. 

### Your contribution

just an idea.

Comments:
Alternatively we could set the number of shards to be a factor than the number of processes (current they're equal) - this way it will be less likely to end up with a shard that is significantly slower than all the other ones.
Issue: `Dataset.to_tf_dataset` fails when strings cannot be encoded as `np.bytes_`

### Describe the bug

When loading any dataset that contains a column with strings that are not ASCII-compatible, looping over those records raises the following exception e.g. for `é` character `UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)`.

### Steps to reproduce the bug

Running the following script will eventually fail, when reaching to the batch that contains non-ASCII compatible strings.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
>>> UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)
```

### Expected behavior

The following script to run properly, making sure that the strings are either `numpy.unicode_` or `numpy.string` instead of `numpy.bytes_` since some characters are not ASCII compatible and that would lead to an issue when applying the `map`.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
```

### Environment info

- `datasets` version: 2.12.1.dev0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments:
May eventually be solved in #5883 

#self-assign
Issue: load_dataset from s3 file system through streaming can't not iterate data 

### Describe the bug

I have a JSON file in my s3 file system(minio), I can use load_dataset to get the file link, but I can't iterate it
<img width="816" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/cc0778d3-36f3-45b5-ac68-4e7c664c2ed0">
<img width="1144" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/76872af3-8b3c-42ff-9f55-528c920a7af1">

we can change 4 lines to fix this bug, you can check whether it is ok for us.
<img width="941" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5a22155a-ece7-496c-8506-047e5c235cd3">

### Steps to reproduce the bug

1. storage a file in you s3 file system
2. use load_dataset to read it through streaming
3. iterate it

### Expected behavior

can iterate it successfully

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1


Comments:
This sounds related to #5281.

Can you try passing `storage_options=s3_client.storage_options` instead passing it to `use_auth_token=` ?

I tried `storage_options` before, but it doesn't work, I checked our source code and I found that we even didn't pass this parameter to the following process. if I use `storage_options` instead of `use_auth_token`, then I also need to change another place of the code. the last line of `streaming_download_manager.py`. our code only passes the `use_auth_token` to the following handler, but does nothing to the `storage_options`
<img width="1050" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5be90933-3331-4ecf-9e11-34f9852d8f92">


Cloud storage support is still experimental indeed and you can expect some bugs.

I think we need to pass the storage options anywhere use_auth_token is passed in indeed. Let me know if you'd be interested in contributing a fix !

Oh, that's great, I really like to fix it. because datasets is really useful and most of our projects need to use it, but we can store our data on the internet due to security reasons. fix it not only make our own work more efficient but also can benefit others who use it.
Issue: Request for text deduplication feature

### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort. 

Comments:
The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).

This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main

I have been using polars to remove duplicates but it would be nice to do it directly in pyarrow.

For example,

1. Read dataset with pyarrow
2. Use scan_pyarrow_dataset() with Polars to create a LazyFrame
3. Use sort and unique to remove duplicates based on a subset of columns
4. Convert to table and save data with ds.write_dataset()

There are times where that workflow makes perfect sense because I do additional transformations with Polars. Most of the time I am simply just reading dataset A and writing dataset B without duplicates though, and I wish I could use a pyarrow scanner or table directly. 

Hi
see this new release  from hf [datatrove](https://github.com/huggingface/datatrove)
DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality
Issue: Why split slicing doesn't behave like list slicing ?

### Describe the bug

If I want to get the first 10 samples of my dataset, I can do :

```
ds = datasets.load_dataset('mnist', split='train[:10]')
```

But if I exceed the number of samples in the dataset, an exception is raised : 

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

> ValueError: Requested slice [:999999999] incompatible with 60000 examples.

### Steps to reproduce the bug

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

### Expected behavior

I would expect it to behave like python lists (no exception raised, the whole list is kept) : 

```
d = list(range(1000))[:999999]
print(len(d))  # > 1000
```

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments:
A duplicate of https://github.com/huggingface/datasets/issues/1774
Issue: Using as_dataset on a "parquet" builder  

### Describe the bug

I used a custom builder to ``download_and_prepare`` a dataset. The first (very minor) issue is that the doc seems to suggest ``download_and_prepare`` will return the dataset, while it does not ([builder.py](https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L718-L738)).
```
        >>> from datasets import load_dataset_builder
        >>> builder = load_dataset_builder("rotten_tomatoes")
        >>> ds = builder.download_and_prepare("./output_dir", file_format="parquet")
```

The main issue I am facing is loading the dataset from those parquet files. I used the `as_dataset` method suggested by the doc, however it returns:
`
FileNotFoundError: [Errno 2] Failed to open local file 'output_dir/__main__-train-00000-of-00245.arrow'. Detail:
[errno 2] No such file or directory.
` 

### Steps to reproduce the bug

1. Create a custom builder of some sort: `builder = CustomBuilder()`.
2. Run `download_and_prepare` with the parquet format: `builder.download_and_prepare("./output_dir", file_format="parquet")`.
3. Run `dataset = builder.as_dataset()`.  

### Expected behavior

I guess I'd expect `as_dataset` to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with `load_dataset` to no avail, probably due to misunderstandings on my part). 

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.14.1
- PyArrow version: 8.0.0
- Pandas version: 1.5.3
```

Comments:
Hi! You can refer to [this doc](https://huggingface.co/docs/datasets/filesystems#load-and-save-your-datasets-using-your-cloud-storage-filesystem) to see the intended usage (basically, it skips the Arrow -> Parquet conversion step in `ds = load_dataset(...); ds.to_parquet("path/to/parquet")`) and allows writing Parquet to remote storage unlike `to_parquet`).

> I guess I'd expect as_dataset to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with load_dataset to no avail, probably due to misunderstandings on my part).

`as_dataset` does not work with `file_format="parquet"` files as Parquet files cannot be memory-mapped, so I think we should just raise an error in that case.