Minimartzz/github-issues huggingface dataset:
Add support to create a Dataset from spark dataframe
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.

Comments:
if i read spark Dataframe , got an error on  multi-node Spark cluster.
Did the Api (Dataset.from_spark) support Spark cluster, read dataframe and save_to_disk?

Error: 
_pickle.PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforma
tion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
23/06/16 21:17:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)



How to perform predictions on Dataset object in Spark with multi-node cluster parallelism?

Addressed in #5701

Hi ! for your information we are working on some more documentation on how to use Spark with HF Datasets repositories (without the need for the `datasets` library) 
https://github.com/huggingface/datasets/issues/5678

Cc @lu-wang-dl @maddiedawson let me know what you think !

sorry, wrong link: https://github.com/huggingface/hub-docs/pull/1392
Dataset.map() crashes when any column contains more than 1000 empty dictionaries
State: closed

### Describe the bug

`Dataset.map()` crashes any time any column contains more than `writer_batch_size` (default 1000) empty dictionaries, regardless of whether the column is being operated on. The error does not occur if the dictionaries are non-empty.

### Steps to reproduce the bug

Example:

```
import datasets

def add_one(example):
    example["col2"] += 1
    return example

n = 1001  # crashes
# n = 999 # works
ds = datasets.Dataset.from_dict({"col1": [{}] * n, "col2": [1] * n})

ds = ds.map(add_one, writer_batch_size=1000)
```

### Expected behavior

Above code should not crash

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.10
- Python version: 3.8.15
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Filter datasets by language code
State: closed

Hi! I use the language search field on https://huggingface.co/datasets
However, some of the datasets tagged by ISO language code are not accessible by this search form.
For example, [myv_ru_2022](https://huggingface.co/datasets/slone/myv_ru_2022) is has `myv` language tag but it is not included in Languages search form.
I've also noticed the same problem with `mhr` (see https://huggingface.co/datasets/AigizK/mari-russian-parallel-corpora)

Comments:
The dataset still can be found, if instead of using the search form you just enter the language code in the url, like https://huggingface.co/datasets?language=language:myv. 

But of course having a more complete list of languages in the search form (or just a fallback to the language codes, if they are missing from the code=>language mapping) would be much more convenient!

Hi! I've opened a PR to make these languages searchable on the Hub.

Thanks @mariosasko!
Do you think it is possible to turn this into a more scalable pipeline? Such as:
1. Looping through all the datasets on the hub and collecting the set of all their language codes;
2. Selecting the codes not covered yet in `Language.ts`
3. Looking up their codes at https://iso639-3.sil.org/code_tables/639/data
4. Adding all the newly found language codes to `Language.ts`

@avidale This has been discussed in https://github.com/huggingface/datasets/issues/4881, so also feel free to share your opinion there.
How to use `load_dataset('glue', 'cola')`
State: closed

### Describe the bug

I'm new to use HuggingFace datasets but I cannot use `load_dataset('glue', 'cola')`.

- I was stacked by the following problem:

```python
from datasets import load_dataset
cola_dataset = load_dataset('glue', 'cola')

---------------------------------------------------------------------------
InvalidVersion                            Traceback (most recent call last)
File <timed exec>:1

(Omit because of long error message)

File /usr/local/lib/python3.8/site-packages/packaging/version.py:197, in Version.__init__(self, version)
    195 match = self._regex.search(version)
    196 if not match:
--> 197     raise InvalidVersion(f"Invalid version: '{version}'")
    199 # Store the parsed out pieces of the version
    200 self._version = _Version(
    201     epoch=int(match.group("epoch")) if match.group("epoch") else 0,
    202     release=tuple(int(i) for i in match.group("release").split(".")),
   (...)
    208     local=_parse_local_version(match.group("local")),
    209 )

InvalidVersion: Invalid version: '0.10.1,<0.11'
```

- You can check this full error message in my repository: [MLOps-Basics/week_0_project_setup/experimental_notebooks/data_exploration.ipynb](https://github.com/makinzm/MLOps-Basics/blob/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup/experimental_notebooks/data_exploration.ipynb)


### Steps to reproduce the bug

- This is my repository to reproduce: [MLOps-Basics/week_0_project_setup](https://github.com/makinzm/MLOps-Basics/tree/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup)

1. cd `/DockerImage` and command `docker build . -t week0`
2. cd `/` and command `docker-compose up`
3. Run `experimental_notebooks/data_exploration.ipynb`

----

Just to be sure, I wrote down Dockerfile and requirements.txt

- Dockerfile
```Dockerfile
FROM python:3.8

WORKDIR /root/working

RUN apt-get update && \
    apt-get install -y python3-dev python3-pip python3-venv && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .

RUN pip3 install --no-cache-dir jupyter notebook && pip install --no-cache-dir -r requirements.txt

CMD ["bash"]
```

- requirements.txt
```txt
pytorch-lightning==1.2.10
datasets==1.6.2
transformers==4.5.1
scikit-learn==0.24.2
```


### Expected behavior

There is no bug to implement `load_dataset('glue', 'cola')`

### Environment info

I already wrote it.

Comments:
Sounds like an issue with incompatible `transformers` dependencies versions.

Can you try to update `transformers` ?

EDIT: I checked the `transformers` dependencies and it seems like you need `tokenizers>=0.10.1,<0.11` with `transformers==4.5.1`

EDIT2: this old version of `datasets` seems to import `transformers` but it's no longer the case, so you could also simply update `datasets` and `transformers` won't be imported

Thank you for advising me to update these libraries versions.

I can implement codes using `datasets==2.10.1` and `transformers==4.27.3`
Support for downloading only provided split
State: open

We can pass split to `_split_generators()`.
But I'm not sure if it's possible to solve cache issues, mostly with `dataset_info.json`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5668). All of your documentation changes will be reflected on that endpoint.

My previous comment didn't create the retro-link in the PR. I write it here again.

You can check the context and the discussions we had about this feature enhancement in this PR:
- #2249
Feature request: IterableDataset.push_to_hub
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

It'd be great to have a lazy push to hub, similar to the lazy loading we have with `IterableDataset`.

Suppose you'd like to filter [LAION](https://huggingface.co/datasets/laion/laion400m) based on certain conditions, but as LAION doesn't fit into your disk, you'd like to leverage streaming:
```
from datasets import load_dataset

dataset = load_dataset("laion/laion400m", streaming=True, split="train")
```
Then you could filter the dataset based on certain conditions:
```
filtered_dataset = dataset.filter(lambda example: example['HEIGHT'] > 400)
```

In order to persist this dataset and push it back to the hub, one currently needs to first load the entire filtered dataset on disk and then push:

```
from datasets import Dataset

Dataset.from_generator(filtered_dataset.__iter__).push_to_hub(...)
```
It would be great if we can instead lazy push to the data to the hub (basically stream the data to the hub), not being limited by our disk size:
```
filtered_dataset.push_to_hub("my-filtered-dataset")
```

### Motivation

This feature would be very useful for people that want to filter huge datasets without having to load the entire dataset or a filtered version thereof on their local disk.

### Your contribution

Happy to test out a PR :)

Comments:
+1

+1

+1, should be possible now? :) https://huggingface.co/blog/xethub-joins-hf

Haha we're working hard to integrate Xet in the HF back-end, it will enable cool use cases :)

Anyway about `IterableDataset.push_to_hub`, I'd be happy to to provide guidance and answer questions if anyone wants to start a first simple implementation of this

+1

+1

+1

+1

Currently running into this when filtering [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) for Dutch entries.

Extra points for somehow making it resumable on error. 11 TB is a lot of data to stream on a home connection without encountering any sort of errors along the way.

If it can help, IterableDataset does implement `.state_dict()` and `.load_state_dict()` that you can use to resume a stream already

+1

+1

+1

Just added a first implementation for `IterableDataset.push_to_hub()` :)

I'll do a new release soon, in the meantime feel free to install `datasets` from source to try it out !
CI is broken: ModuleNotFoundError: jax requires jaxlib to be installed
State: closed
Labels: description: Something isn't working
name: bug

CI test_py310 is broken: see https://github.com/huggingface/datasets/actions/runs/4498945505/jobs/7916194236?pr=5662
```
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_in_memory - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_on_disk - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_audio - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_device - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_image - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_jnp_array_kwargs - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/features/test_features.py::CastToPythonObjectsTest::test_cast_to_python_objects_jax - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
===== 8 failed, 2147 passed, 10 skipped, 37 warnings in 228.69s (0:03:48) ======
```

Comments:
integration with imbalanced-learn
State: closed
Labels: description: New feature or request
name: enhancement

description: This will not be worked on
name: wontfix

### Feature request

Wouldn't it be great if the various class balancing operations from imbalanced-learn were available as part of datasets?

### Motivation

I'm trying to use imbalanced-learn to balance a dataset, but it's not clear how to get the two to interoperate - what would be great would be some examples.  I've looked online, asked gpt-4, but so far not making much progress.

### Your contribution

If I can get this working myself I can submit a PR with example code to go in the docs

Comments:
You can convert any dataset to pandas to be used with imbalanced-learn using `.to_pandas()`

Otherwise if you want to keep a `Dataset` object and still use e.g. [make_imbalance](https://imbalanced-learn.org/stable/references/generated/imblearn.datasets.make_imbalance.html#imblearn.datasets.make_imbalance), you just need to pass the list of rows ids and labels:

```python
row_indices = list(range(len(dataset)))
resampled_row_indices, _ = make_imbalance(
    row_indices,
    dataset["label"],
    sampling_strategy={0: 25, 1: 50, 2: 50},
    random_state=RANDOM_STATE,
)

resampled_dataset = dataset.select(resampled_row_indices)
```
Doc: save_to_disk, `num_proc` will affect `num_shards`, but it's not documented
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

description: Good for newcomers
name: good first issue

### Describe the bug

[`num_proc`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_proc) will affect `num_shards`, but it's not documented


### Steps to reproduce the bug

Nothing to reproduce

### Expected behavior

[document of `num_shards`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_shards) explicitly says that it depends on `max_shard_size`, it should also mention `num_proc`.


### Environment info

datasets main document

Comments:
I agree this should be documented
expanduser in save_to_disk
State: closed
Labels: description: Good for newcomers
name: good first issue

### Describe the bug

save_to_disk() does not expand `~`

1. `dataset = load_datasets("any dataset")`
2. `dataset.save_to_disk("~/data")`
3. a folder named "~" created in current folder
4. FileNotFoundError is raised, because the expanded path does not exist (`/home/<user>/data`)

related issue https://github.com/huggingface/transformers/issues/10628

### Steps to reproduce the bug

As described above.

### Expected behavior

expanduser correctly

### Environment info

- datasets 2.10.1
- python 3.10

Comments:
`save_to_disk` should indeed expand `~`. Marking it as a "good first issue".

#self-assign

File path to code: 

https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_dataset.py#L1364

@RmZeta2718 I created a pull request for this issue. 

Hello, 
It says `save_to_disk` is deprecated in 2.8.0, so the alternative to this will be `storage_options`? 

https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.save_to_disk

@ashikshafi08 I think you misunderstood the warning. The method `save_to_disk` is not deprecated only the optional parameter `fs`.
Also @benjaminbrown038 as I cannot find your PR I would like to work on this if you don't mind.

@mariosasko It's been several months and the PR is not reviewed. Could you please take a look? I assume this is not complicated and could be merged fairly soon.
The index column created with .to_sql() is dependent on the batch_size when writing
State: closed

### Describe the bug

It seems like the "index" column is designed to be unique? The values are only unique per batch. The SQL index is not a unique index.

This can be a problem, for instance, when building a faiss index on a dataset and then trying to match up ids with a sql export.

### Steps to reproduce the bug

```
from datasets import Dataset
import sqlite3
db = sqlite3.connect(":memory:")
nice_numbers = Dataset.from_dict({"nice_number": range(101,106)})
nice_numbers.to_sql("nice1", db, batch_size=1)
nice_numbers.to_sql("nice2", db, batch_size=2)
print(db.execute("select * from nice1").fetchall()) # [(0, 101), (0, 102), (0, 103), (0, 104), (0, 105)]
print(db.execute("select * from nice2").fetchall()) # [(0, 101), (1, 102), (0, 103), (1, 104), (0, 105)]
```

### Expected behavior

I expected the "index" column to be unique

### Environment info

```
 % datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2

zsh: segmentation fault  datasets-cli env
```

Comments:
Thanks for reporting, @lsb. 

We are investigating it.

On the other hand, please note that in the next `datasets` release, the index will not be created by default (see #5583). If you would like to have it, you will need to explicitly pass `index=True`. 

I think this is low enough priority for me to close this as Won't Fix. If I need any primary keys I can generate them beforehand. Feel free to reopen.
flatten_indices doesn't work with pandas format
State: open
Labels: description: Something isn't working
name: bug

### Describe the bug

Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output

### Steps to reproduce the bug

tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1

Comments:
Thanks for reporting! This can be fixed by setting the format to `arrow` in `flatten_indices` and restoring the original format after the flattening. I'm working on a PR that reduces the number of the `flatten_indices` calls in our codebase and makes `flatten_indices` a no-op when a dataset does not have an indices mapping, so I'll incorporate the fix in that PR.
Features cannot be named "self"
State: closed

### Describe the bug

Hi,
I noticed that we cannot create a HuggingFace dataset from Pandas DataFrame with a column named `self`.
The error seems to be coming from arguments validation in the `Features.from_dict` function.



### Steps to reproduce the bug

```python
import datasets
dummy_pandas = pd.DataFrame([0,1,2,3], columns = ["self"])
datasets.arrow_dataset.Dataset.from_pandas(dummy_pandas)
```

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.8.0
- Python version: 3.9.5
- PyArrow version: 6.0.1
- Pandas version: 1.4.1

Comments:
Parquet file wrongly recognized as zip prevents loading a dataset
State: closed

### Describe the bug

When trying to `load_dataset_builder` for `HuggingFaceGECLM/StackExchange_Mar2023`, extraction fails, because parquet file [devops-00000-of-00001-22fe902fd8702892.parquet](https://huggingface.co/datasets/HuggingFaceGECLM/StackExchange_Mar2023/resolve/1f8c9a2ab6f7d0f9ae904b8b922e4384592ae1a5/data/devops-00000-of-00001-22fe902fd8702892.parquet) is wrongly identified by python as being a zip not a parquet.
(Full thread on [Slack](https://huggingface.slack.com/archives/C02V51Q3800/p1678890880803599))

### Steps to reproduce the bug

```python
from datasets import load_dataset_builder

ds = load_dataset_builder("HuggingFaceGECLM/StackExchange_Mar2023")
```

### Expected behavior

Loading the file normally.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1058-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Comments:
xPath to implement all operations for Path
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.

Comments:
 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).

Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?

`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.



Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.

It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)
IterableDataset with_format does not support 'device' keyword for jax
State: open

### Describe the bug

As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`

Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword? 
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029



### Steps to reproduce the bug

1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)

### Expected behavior

I expect to call `with_format('jax', device=device)`  as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error

### Environment info

Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Comments:
Hi! Yes, only `torch` is currently supported. Unlike `Dataset`, `IterableDataset` is not PyArrow-backed, so we cannot simply call `to_numpy` on the underlying subtables to format them numerically. Instead, we must manually convert examples to (numeric) arrays while preserving consistency with `Dataset`, which is not trivial, so this is still a to-do.

Any plans to support it in the future? Or would streaming dataset be left without support for jax and tensorflow?

@mariosasko 
`IterableDataset` and `with_format('torch', device=device)` also does not work (using `datasets==3.2.0`)