mrvsrc/github-issues huggingface dataset:
Dedicated RNG object for fingerprinting

Closes https://github.com/huggingface/datasets/issues/6604, closes https://github.com/huggingface/datasets/issues/2775

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ELI5 no longer available, but referenced in example code

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.
datasets map `cache_file_name` does not work

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2
Loading CSV exported dataset has unexpected format

### Describe the bug

I wanted to be able to save a HF dataset for translations and load it again in another script, but I'm a bit confused with the documentation and the result I've got so I'm opening this issue to ask if this behavior is as expected.

### Steps to reproduce the bug

The documentation I've mainly consulted is https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/loading_methods#datasets.load_dataset and https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset (where I've found `.to_csv()`)

```python
# Load a dataset of translations
test_dataset = load_dataset("opus100", name="en-fr", split="test")

# Save with .to_csv()
test_csv_path = "try_testset_save.csv"
test_dataset.to_csv(test_csv_path)

# Load dataset from the CSV
loaded_dataset = load_dataset("csv", data_files=test_csv_path)
print(test_dataset_fromfile[0]["translation"])
print(test_dataset_fromfile[0]["translation"]["en"])
```
```
Creating CSV from Arrow format: 100%
2/2 [00:00<00:00, 47.99ba/s]
Downloading data files: 100%
1/1 [00:00<00:00, 65.33it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 42.10it/s]
Generating train split:
2000/0 [00:00<00:00, 47486.09 examples/s]

{'en': "She wasn't going to vaccinate her kid against polio,  no way.", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[29], line 11
      9 loaded_dataset = load_dataset("csv", data_files=test_csv_path)
     10 print(test_dataset_fromfile[0]["translation"])
---> 11 print(test_dataset_fromfile[0]["translation"]["en"])

TypeError: string indices must be integers, not 'str'
```


### Expected behavior

Each translation was saved as a stringified dict like `"{'en': ""She wasn't going to vaccinate her kid against polio,  no way."", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}"` where I would have expected 2 columns (1st with English segments, and 2nd with French segments), and I was expecting `load_dataset` to infer the type of feature automatically as I haven't seen anything about it in the documentation.

Do you have an example of how to effectively save and load datasets of translations ?

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.5
- `huggingface_hub` version: 0.16.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
Easy way to segment into 30s snippets given an m4a file and a vtt file 

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.
Dataset.push_to_hub of a canonical dataset creates an additional dataset under the user namespace

While using `Dataset.push_to_hub` of a canonical dataset, an additional dataset was created under my user namespace.

## Steps to reproduce the bug
The command:
```python
commit_info = ds.push_to_hub(
    "caner",
    config_name="default",
    commit_message="Convert dataset to Parquet",
    commit_description="Convert dataset to Parquet.",
    create_pr=True,
    token=token,
)
```
creates the additional dataset `albertvillanova/caner`.

It is caused by these code lines: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/dataset_dict.py#L1688-L1694

Also note the information in the docstring: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/dataset_dict.py#L1582-L1585

> Also accepts `<dataset_name>`, which will default to the namespace of the logged-in user.

This behavior was "reverted" by the PR: 
- #6519

We have therefore contradictory requirements. We should decide:
- whether to support passing dataset_namespace without user/org that defaults to the logged-in user (and not support canonical datasets)
- or vice-versa, to support canonical datasets and not support passing only dataset_name

As canonical datasets are "deprecated" (and will eventually disappear), I would choose the first option. However, if so, the Space to convert datasets to Parquet will not work for canonical datasets: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

IIUC, this could also be "fixed" by `create_repo("dataset_name")` not defaulting to `create_repo("user/dataset_name")` (when the user's token is available), which would be consistent with the rest of the `HfApi` ops used in the `push_to_hub` implementation. This is a (small) breaking change for `huggingface_hub`, but justified to make the API more consistent.
Logs are delayed when doing .map when `docker logs`

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...
The datasets models housed in Dropbox can't support a lot of users downloading them

### Describe the bug

I'm using the datasets 


```
from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```


And it seems that sometimes when I imagine a lot of users are accessing the same resources, the Dropbox host fails:

`raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})") ConnectionError: Couldn't reach https://www.dropbox.com/s/e2us0hcs3ilr20e/MInDS-14.zip?dl=1 (error 429)`

My question is if we can somehow host these files elsewhere or can you change the limit of simultaneous users accessing those resources or any other solution?

Also, has anyone had this issue before?

Thanks

### Steps to reproduce the bug

1: Create a python script like so:
```

from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

2: Execute this by a certain number of users at the same time

### Expected behavior

I woudl expect that this shouldnt happen unless its a huge amount of users, which it is not the case

### Environment info

This was done in an Ubuntu 22 environment.
After `2.16.0` version, there are `PermissionError` when users use shared cache_dir

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)
Allow concatenation of datasets with mixed structs

Fixes #6466 

The idea is to do a recursive check for structs. PyArrow handles it well enough.


For a demo you can do:

```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

Now both the features and the rows are fixed.

I note that Sequence suffers from the same problem, so I can fix that in a future PR once this one is merged.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
keep more info in DatasetInfo.from_merge #6585

* try not to merge DatasetInfos if they're equal

* fixes losing DatasetInfo during parallel Dataset.map

@JochenSiegWork fyi, that seems to also affect the `trainer.push_to_hub()` method, which I guess also needs to parse that DatasetInfo from the `kwargs` used by `push_to_hub`.
There is short discussion about it [here](https://github.com/huggingface/blog/issues/1623).
Would be great if you can check if your PR would also fix that!

> @JochenSiegWork fyi, that seems to also affect the `trainer.push_to_hub()` method, which I guess also needs to parse that DatasetInfo from the `kwargs` used by `push_to_hub`. There is short discussion about it [here](https://github.com/huggingface/blog/issues/1623). Would be great if you can check if your PR would also fix that!

Hi @thiagobarbosa, it might be related but I didn't worked with `push_to_hub` yet. I don't see a minimal example reproducing the specific error in your link. However, if you have a running version producing the error locally you can test it by pulling this PR and run your specific example locally.
fix os.listdir return name is empty string

fix #6588

xlistdir return name is empty string

for example：
`
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"


download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)

`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6581). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Obj ["name"] ends with "/"
Faster webdataset streaming

requests.get(..., streaming=True) is faster than using HTTP range requests when streaming large TAR files

it can be enabled using chunk_size=0 in fsspec

cc @rwightman 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I added faster streaming support using streaming Requests instances in `huggingface_hub`.

This PR can be used with https://github.com/huggingface/huggingface_hub/pull/1967 to get fast WebDataset streaming
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

### Describe the bug

The issue shall be open: https://github.com/huggingface/datasets/issues/6435

When i try to pass `with_rank` to `Dataset.filter()`, i get this:

`Dataset.filter() got an unexpected keyword argument 'with_rank'`

### Steps to reproduce the bug

Run notebook:

https://colab.research.google.com/drive/1WUNKph8BdP0on5ve3gQnh_PE0cFLQqTn?usp=sharing

### Expected behavior

Should work?

### Environment info

NVIDIA RTX 4090
Fix imagefolder with one image

A dataset repository with one image and one metadata file was considered a JSON dataset instead of an ImageFolder dataset. This is because we pick the dataset type with the most compatible data file extensions present in the repository and it results in a tie in this case.

e.g. for https://huggingface.co/datasets/multimodalart/repro_1_image

I fixed this by deprioritizing metadata files in the count.

fix https://github.com/huggingface/datasets/issues/6545

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6556). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Do not use Parquet exports if revision is passed

Fix #6554.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6555). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As shared on slack, `HubDatasetModuleFactoryWithParquetExport` raises a `DatasetsServerError` already if the user tries to load another revision that the one from the parquet export. And therefore it fall backs on using `HubDatasetModuleFactoryWithScript`
Cannot import name 'load_dataset' from .... module ‘datasets’

### Describe the bug


use python -m pip install datasets to install

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

it doesn't work

### Environment info

datasets version==2.15.0
python == 3.10.12
linux  version I don't know??

I don't know  My conpany conputer cannot work. but in my computer, it work?
Impossible to only download a test split

I've spent a significant amount of time trying to locate the split object inside my _split_generators() custom function.
Then after diving [in the code](https://github.com/huggingface/datasets/blob/5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54/src/datasets/load.py#L2558) I realized that `download_and_prepare` is executed before! split is passed to the dataset builder in `as_dataset`.

If I'm not missing something, this seems like bad design, for the following use case:

> Imagine there is a huge dataset that has an evaluation test set and you want to just download and run just to compare your method.

Is there a current workaround that can help me achieve the same result?

Thank you,

The only way right now is to load with streaming=True
Deprecate Beam API and download from HF GCS bucket

Deprecate the Beam API and download from the HF GCS bucked. 

TODO:
- [ ] Deprecate the Beam-based [`wikipedia`](https://huggingface.co/datasets/wikipedia) in favor of [`wikimedia/wikipedia`](https://huggingface.co/datasets/wikimedia/wikipedia) ([Hub PR](https://huggingface.co/datasets/wikipedia/discussions/19))
- [ ] Make [`natural_questions`](https://huggingface.co/datasets/natural_questions) a no-code dataset
- [ ] Make [`wiki40b`](https://huggingface.co/datasets/wiki40b) a no-code dataset
- [ ] Make [`wiki_dpr`](https://huggingface.co/datasets/wiki_dpr) an Arrow-based dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Can't align optional features of struct

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.

Friendly bump, I would be happy to work on this issue once I get the go-ahead from the dev team.
Stuck in "Resolving data files..."

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Most likely, the data file inference logic is the problem here.

You can run the following code to verify this:
```python
import time
from datasets.data_files import get_data_patterns
start_time = time.time()
get_data_patterns("/path/to/img_dir")
end_time = time.time()
print(f"Elapsed time: {end_time - start_time:.2f}s")
```
 
We plan to optimize this for the next version (or version after that). In the meantime, specifying the split patterns manually should give better performance:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/img_dir/train/**", ...}, split="train")
```

Hi, @mariosasko, you are right; data file inference logic is extremely slow.

I have done a similar test, that is I modify the source code of datasets/load.py to measure the cost of two suspicious operations:
```python
def get_module(self) -> DatasetModule:
        base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
        start = time.time()
        patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
        print(f"patterns: {time.time() - start}")
        start = time.time()
        data_files = DataFilesDict.from_patterns(
            patterns,
            download_config=self.download_config,
            base_path=base_path,
        )
        print(f"data_files: {time.time() - start}")
```
It gaves:
patterns: 3062.2050700187683
data_files: 413.9576675891876

Thus, these two operations contribute to almost all of load time. What's going on in them?

Furthermore, what's my current workaround about this problem? Should I save it by `save_to_disk()` and load dataset through `load_from_disk`?
Can't load ds = load_dataset("imdb")

### Describe the bug

I did `from datasets import load_dataset, load_metric` and then `ds = load_dataset("imdb")` and it gave me the error:
ExpectedMoreDownloadedFiles: {'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'}

I tried doing `ds = load_dataset("imdb",download_mode="force_redownload")` as well as reinstalling dataset. I still face this problem.

### Steps to reproduce the bug

1. from datasets import load_dataset, load_metric
2. ds = load_dataset("imdb")

### Expected behavior

It should load and give me this when I run `ds`

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-164-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.16.2
- PyArrow version: 13.0.0
- Pandas version: 2.0.2

I'm unable to reproduce this error. The server hosting the files may have been down temporarily, so try again.
FolderBase Dataset automatically resolves under current directory when data_dir is not specified

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

@lhoestq 

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)

I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

@mariosasko can you please assign this issue to me,I want to work on this

#self-assign

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

@zutarich It is unless @debrupf2946 is working on it.
Offline dataset viewer

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.

Hi, thanks for the suggestion. It's not possible at the moment. The viewer is part of the Hub codebase and only works on public datasets. Also, it relies on [Datasets Server](https://github.com/huggingface/datasets-server/), which prepares the data and provides an API to access the rows, size, etc.

If you're interested in hosting your data as a private dataset on the Hub, you might want to look at https://github.com/huggingface/datasets-server/issues/39.

Hi, we are building an offline dataset viewer: https://github.com/Renumics/spotlight
It supports many HF datasets, but currently you have to use it via Pandas:
df=ds.to_pandas()
spotlight.show(df)

Would love to hear from you if that works for your use case. If not, feel free to open an issue on the repo: https://github.com/Renumics/spotlight/issues

@ssuwelack thank you! I will definitely try it out.

Related issues:
- https://github.com/huggingface/datasets-server/issues/213
- https://github.com/huggingface/datasets-server/issues/441
- https://github.com/huggingface/datasets/issues/6014

Closing for now, as developing and maintaining an offline viewer is not planned.
Inconsistent "The features can't be aligned" error when combining map, multiprocessing, and variable length outputs

### Describe the bug

I'm using a dataset with map and multiprocessing to run a function that returned a variable length list of outputs. This output list may be empty. Normally this is handled fine, but there is an edge case that crops up when using multiprocessing. In some cases, an empty list result ends up in a dataset shard consisting of a single item. This results in a `The features can't be aligned` error that is difficult to debug because it depends on the number of processes/shards used.

I've reproduced a minimal example below. My current workaround is to fill empty results with a dummy value that I filter after, but this was a weird error that took a while to track down.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_list([{'idx':i} for i in range(60)])

def test_func(row, idx):
    if idx==58:
        return {'output': []}
    else:
        return {'output' : [{'test':1}, {'test':2}]}

# this works fine
test1 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=4)

# this fails
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32)
>ValueError: The features can't be aligned because the key output of features {'idx': Value(dtype='int64', id=None), 'output': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None)} has unexpected type - Sequence(feature=Value(dtype='null', id=None), length=-1, id=None) (expected either [{'test': Value(dtype='int64', id=None)}] or Value("null").
```

The error occurs during the check

```python
_check_if_features_can_be_aligned([dset.features for dset in dsets])
```

When the multiprocessing splitting lines up just right with the empty return value, one of the `dset` in `dsets` will have a single item with an empty list value, causing the error.

### Expected behavior

Expected behavior is the result would be the same regardless of the `num_proc` value used.

### Environment info

Datasets version 2.11.0
Python 3.9.16

This scenario currently requires explicitly passing the target features (to avoid the error): 
```python
import datasets

...

features = dataset.features
features["output"] = = [{"test": datasets.Value("int64")}]
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32, features=features)
```

I just encountered the same error in the same situation (multiprocessing with variable length outputs).

The funny (or dangerous?) thing is, that this error only showed up when testing with a small test dataset (16 examples, ValueError with `num_proc` >1) but the same code works fine for the full dataset (~70k examples).

@mariosasko Any idea on how to do that with a nested feature with lists of variable lengths containing dicts?

EDIT: Was able to narrow it down: >200 Examples: no error, <150 Examples: Error. 
Now idea what to make of this but pretty obvious that this is a bug....
Request for text deduplication feature

### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort. 

The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).

This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main

I have been using polars to remove duplicates but it would be nice to do it directly in pyarrow.

For example,

1. Read dataset with pyarrow
2. Use scan_pyarrow_dataset() with Polars to create a LazyFrame
3. Use sort and unique to remove duplicates based on a subset of columns
4. Convert to table and save data with ds.write_dataset()

There are times where that workflow makes perfect sense because I do additional transformations with Polars. Most of the time I am simply just reading dataset A and writing dataset B without duplicates though, and I wish I could use a pyarrow scanner or table directly.