liuqingwen/github-issues huggingface dataset:
extend the map function so it can wrap around long text that does not fit in the context window
Labels: name: enhancement
description: New feature or request

### Feature request

I understand `dataset` provides a [`map`](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L2849) function. This function in turn takes in a callable that is used to tokenize the text on which a model is trained. Frequently this text will not fit within a models's context window. In this case it would be useful to wrap around the text into multiple rows with each row fitting the model's context window. I tried to do it using this code as example which in turn I have borrowed from [here](https://stackoverflow.com/a/76343993/147530):

```
data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
```

but running the code gives me this error:

```
File "/llm/fine-tune.py", line 117, in <module>
    data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 580, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 545, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3087, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3480, in _map_single
    writer.write_batch(batch)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_writer.py", line 556, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 3798, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 2962, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 394 but got length 447
```

The lambda function I have provided is correctly chopping up long text so it wraps around (and because of this 394 samples become 447 after wrap around) but the dataset `map` function does not like it.

### Motivation

please see above

### Your contribution

I'm afraid I don't have much knowledge to help

Comments:
I just noticed the [docs](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L2881C11-L2881C200) say:

>If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.

so maybe this is a bug then.
`map` with any joblib backend
Labels: name: enhancement
description: New feature or request

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).

Comments:
Set a rule on the config and split names

> should we actually allow characters like spaces? maybe it's better to add validation for whitespace symbols and directly in datasets and raise

https://github.com/huggingface/datasets-server/issues/853


Comments:
in this case we need to decide what to do with the existing datasets with white space characters (there shouldn't be a lot of them I think)

I imagine that we should stop supporting them, and help the user fix them?
Make IterableDataset.from_spark more efficient

Moved the code from using collect() to using toLocalIterator, which allows for prefetching partitions that will be selected next, thus allowing for better performance when iterating. 

Comments:
AutoSharding IterableDataset's when num_workers > 1
Labels: name: enhancement
description: New feature or request

### Feature request


Minimal Example

```
import torch
from datasets import IterableDataset

d = IterableDataset.from_file(<file_name>)
dl = torch.utils.data.dataloader.DataLoader(d,num_workers=3)

for sample in dl:
    print(sample)

```

Warning:
Too many dataloader workers: 2 (max is dataset.n_shards=1). Stopping 1 dataloader workers.
To parallelize data loading, we give each process some shards (or data sources) to process. Therefore it's unnecessary to have a number of workers greater than dataset.n_shards=1. To enable more parallelism, please split the dataset in more files than 1.

Expected Behavior:
Dataset is sharded each cpu uses subset (contiguously - so you can do checkpoint loading/saving) 

### Motivation

I have a lot of unused cpu's and would like to be able to shard iterable datasets with pytorch's dataloader when num_workers > 1. This is for a very large single file. I am aware that we can use the `split_dataset_by_node` to ensure that each node (for distributed) gets different shards, but we should extend it so that this also continues for multiple workers. 

### Your contribution

If someone points me to what needs to change, I can create a PR.

Comments:
For this to be possible, we would have to switch from the "Streaming" Arrow format to the "Random Access" (IPC/Feather) format, which allows reading arbitrary record batches (explained [here](https://arrow.apache.org/docs/python/ipc.html)). We could then use these batches to construct shards.

@lhoestq @albertvillanova Do you think this use case is worth the switch? Also, we currently shard files, not inner row groups/chunks. Should we also support sharding row groups (e.g. if the number of input files is 1)?

PS: I don't expect significant speed-up for local, uncompressed Arrow files.

Alternatively we could support multiprocessing map for iterable datasets and let the user do the CPU intensive task there ?

This way it would work on arrow data but also on any iterable dataset
Only two cores are getting used in sagemaker with pytorch 3.10 kernel

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel

Comments:
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625

From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.
Avoid stuck map operation when subprocesses crashes

I've been using Dataset.map() with `num_proc=os.cpu_count()` to leverage multicore processing for my datasets, but from time to time I get stuck processes waiting forever. Apparently, when one of the subprocesses is abruptly killed (OOM killer, segfault, SIGKILL, etc), the main process keeps waiting for the async task sent to that child process to finish.

It seems to be easy to reproduce the issue with the following script:

```
import os
from datasets import Dataset, Features, Value


def do_stuck(item):
    os.kill(os.getpid(), 9)

data = {
    "col1": list(range(5)),
    "col2": list(range(5)),
}

ds = Dataset.from_dict(
    data,
    features=Features({
        "col1": Value("int64"),
        "col2": Value("int64"),
    }),
)

print(ds.map(do_stuck, num_proc=4))
```

This is an old behavior in Python, which apparently was fixed a few years ago in `concurrent.futures.ProcessPoolExecutor` ([ref](https://bugs.python.org/issue9205)), but not in `multiprocessing.pool.Pool` / `multiprocess.pool.Pool`, which is used by `Dataset.map` ([ref](https://bugs.python.org/issue22393)).

This PR is an idea to try to detect when a child process gets killed, and raises a `RuntimeError` warning the dataset.map() caller.

EDIT: Related proposal for future improvement: https://github.com/huggingface/datasets/discussions/5977

Comments:
Hi ! Do you think this can be fixed at the Pool level ? Ideally it should be the Pool responsibility to handle this, not the `map` code. We could even subclass Pool if needed (at least the one from `multiprocess`)

@lhoestq it makes sense to me. Just pushed a refactoring creating a `class ProcessPool(multiprocess.pool.Pool)` to keep track of the PID changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5976). All of your documentation changes will be reflected on that endpoint.

I managed to raise an error without subclassing Pool with two additions to `iflatmap_unordered`:

1. at the beggining
```python
original_pool = list(pool._pool)
```

2. in the loop
```python
if any(async_result._pool != original_pool for async_result in async_results) and queue.empty():
    raise RuntimeError(
        "One of the subprocesses has abruptly died during map operation."
        "To debug the error, disable multiprocessing."
    )
```

It's still a fix that only works for `iflatmap_unordered` (so not for map, imap etc) but is maybe simpler that subclassing. It also works for both multiprocessing.Pool and multiprocess.Pool
Docs: make "repository structure" easier to find
Labels: name: documentation
description: Improvements or additions to documentation

The page https://huggingface.co/docs/datasets/repository_structure explains how to create a simple repository structure without a dataset script.
It's the simplest way to create a dataset and should be easier to find, particularly on the docs' first pages.

Comments:
Loading a local dataset also works the same way when `data_files` are not specified, so I agree we should make this info  easier to discover 

cc @stevhliu 

Is this issue open? If so, I will self assign. 

@benjaminbrown038 Yes, it is. Maybe @stevhliu can give some pointers on improving this doc page's discoverability.

I think we can add a version of the [Main use-case](https://huggingface.co/docs/datasets/repository_structure#main-usecase) section to the [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset) tutorial. 

Currently, it doesn't tell you *how* to structure the repository; it only tells you how to create it. So adding the "main use-case" will help bridge the gap and make it easier to find. We should also add a link to the [Structure your repository](https://huggingface.co/docs/datasets/repository_structure) guide for users who want to learn about the other options.
Common Voice datasets still need `use_auth_token=True`

### Describe the bug

We don't need to pass `use_auth_token=True` anymore to download gated datasets or models, so the following should work if correctly logged in.

```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

However it throws an error - probably because something weird is hardcoded into the dataset loading script.

### Steps to reproduce the bug

1.) 
```
huggingface-cli login
```

2.) Make sure that you have accepted the license here:
https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1

3.) Run:
```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

4.)  You'll get:

```
File ~/hf/lib/python3.10/site-packages/datasets/builder.py:963, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    961 split_dict = SplitDict(dataset_name=self.name)
    962 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 963 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    965 # Checksums verification
    966 if verification_mode == VerificationMode.ALL_CHECKS and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_1/f4d7854c466f5bd4908988dbd39044ec4fc634d89e0515ab0c51715c0127ffe3/common_voice_6_1.py:150, in CommonVoice._split_generators(self, dl_manager)
    148 hf_auth_token = dl_manager.download_config.use_auth_token
    149 if hf_auth_token is None:
--> 150     raise ConnectionError(
    151         "Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset"
    152     )
    154 bundle_url_template = STATS["bundleURLTemplate"]
    155 bundle_version = bundle_url_template.split("/")[0]

ConnectionError: Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset
```

### Expected behavior

One should not have to pass `use_auth_token=True`. Also see discussion here: https://github.com/huggingface/blog/pull/1243#discussion_r1235131150

### Environment info

```
- `datasets` version: 2.13.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
```

Comments:
cc @pcuenca as well. 

Not super urgent btw

The issue commes from the dataset itself and is not related to the `datasets` lib

see https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1/blob/2c475b3b88e0f2e5828f830a4b91618a25ff20b7/common_voice_6_1.py#L148-L152

Let's remove these lines in the dataset no? cc @anton-l @Vaibhavs10
IterableDataset: split by node and map may preprocess samples that will be skipped anyway

              There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051_
            

Comments:
Does "number of shards" refer to the total number of data?

my config:
nproc_per_node=2
ds=ds['train'] = load_dataset(streaming=True).take(50000)

I'm test again: in prepare_data(),  data have the same for each GPU


The number of shards is `ds.n_shards`. It corresponds generally to the number of files the dataset is made of, to be able to distribute to several nodes.

**You don't end up with the same data per GPU**. But all the samples are going through your preprocessing function you pass to map. They are just skipped afterwards to only keep 1 sample out of n(GPUs)

For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end. 
Is my understanding correct?

Where can I print the actual training data for each GPU?

> For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

> Where can I print the actual training data for each GPU?

You should call print in the data_collator

I print out n_shards, and under multiple GPUs, this value is always 1.
Is this value correct?

Yes it's correct, and it explains why you always have the same data passed to your map function (the data can't be split).

But after being passed to `map`, each GPU keeps one example out of n(GPUs) so that you don't end up with duplicate data across GPUs

> > For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
> > Is my understanding correct?
> 
> Yes exactly :)
> 
> > Where can I print the actual training data for each GPU?
> 
> You should call print in the data_collator

OK, when printing the train data in the data collator, each GPU sees different data.

Thanks for your reply
Support for data with instance-wise dictionary as features
Labels: name: enhancement
description: New feature or request

### Feature request

I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.

It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).

### Motivation

I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
    }
},
...
{
    "index": 9999,
    "feature": {
        "x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
        ...
    }
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
        "x >= 6": None,  # keys from other instances
        ...
    }
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.

A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.

### Your contribution

N/A

Comments:
Hi ! We use the Arrow columnar format under the hood, which doesn't support such dictionaries: each field must have a fixed type and exist in each sample.

Instead you can restructure your data like
```
{
    "index": 0,
    "keys": ["2 * x + y >= 3"],
    "values": [["2 * x + y >= 3", "4 * x + 2 * y >= 6"]],
    }
},
...
{
    "index": 9999,
    "keys": ["x >= 6"],
    "values": [["x >= 6", "x >= 0", "x >= -1"]],
},
...
```
Return the audio filename when decoding fails due to corrupt files
Labels: name: enhancement
description: New feature or request

### Feature request

Return the audio filename when the audio decoding fails. Although currently there are some checks for mp3 and opus formats with the library version there are still cases when the audio decoding could fail, eg. Corrupt file. 

### Motivation

When you try to load an object file dataset and the decoding fails you can't know which file is corrupt
```

raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7f5ab7e38290>: Format not recognised.
```

### Your contribution

Make a PR to Add exceptions for LIbsndfileError to return the audio filename or path when soundfile decoding fails.

Comments:
Hi ! The audio data don't always exist as files on disk - the blobs are often stored in the Arrow files. For now I'd suggest disabling decoding with `.cast_column("audio", Audio(decode=False))` and apply your own decoding that handles corrupted files (maybe to filter them out ?)

cc @sanchit-gandhi since it's related to our discussion about allowing users to make decoding return `None` and show a warning when there are corrupted files

Thanks @lhoestq, I wasn't aware of the decode flag. It makes more sense as you say to show a warning when there are corrupted files together with some metadata of the file that allows to filter them from the dataset.

My workaround was to catch the LibsndfileError and generate a dummy audio with an unsual sample rate to filter it later. However returning `None` seems better. 

`try:
    array, sampling_rate = sf.read(file)
except sf.LibsndfileError:
    print("bad file")
    array = np.array([0.0])
    sampling_rate = 99.000`
Pass datasets-cli additional args as kwargs to DatasetBuilder in `run_beam.py`

Hi,

Following this <https://discuss.huggingface.co/t/how-to-preprocess-a-wikipedia-dataset-using-dataflowrunner/41991/3>, here is a simple PR to pass any additional args to datasets-cli as kwargs in the DatasetBuilder in `run_beam.py`.

I also took the liberty to add missing setup steps to the `beam.mdx` docs in order to help everyone.

@lhoestq 

Comments: