SurAyush/github_issues huggingface dataset:
Issue: Add IterableDataset.shard()
State: closed
Author: lhoestq

Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7252). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Basic XML support (mostly copy pasted from text)
State: closed
Author: lhoestq

enable the viewer for datasets like https://huggingface.co/datasets/FrancophonIA/e-calm (there will be more and more apparently)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7250). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: How to debugging
State: open
Author: ShDdu

### Describe the bug

I wanted to use my own script to handle the processing, and followed the tutorial documentation by rewriting the MyDatasetConfig and MyDatasetBuilder (which contains the _info,_split_generators and _generate_examples methods) classes. Testing with simple data was able to output the results of the processing, but when I wished to do more complex processing, I found that I was unable to debug (even the simple samples were inaccessible). There are no errors reported, and I am able to print the _info,_split_generators and _generate_examples messages, but I am unable to access the breakpoints.

### Steps to reproduce the bug

# my_dataset.py
import json
import datasets


class MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, **kwargs):
        super(MyDatasetConfig, self).__init__(**kwargs)


class MyDataset(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        MyDatasetConfig(
            name="default",
            version=VERSION,
            description="myDATASET"
        ),
    ]

    def _info(self):
        print("info")  # breakpoints
        return datasets.DatasetInfo(
            description="myDATASET",
            features=datasets.Features(
                {
                    "id": datasets.Value("int32"),
                    "text": datasets.Value("string"),
                    "label": datasets.ClassLabel(names=["negative", "positive"]),
                }
            ),
            supervised_keys=("text", "label"),
        )

    def _split_generators(self, dl_manager):
 
        print("generate")  # breakpoints
        data_file = "data.json"  

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": data_file}
            ),
        ]

    def _generate_examples(self, filepath):
        print("example")  # breakpoints
        with open(filepath, encoding="utf-8") as f:
            data = json.load(f)
            for idx, sample in enumerate(data):
                yield idx, {
                    "id": sample["id"],
                    "text": sample["text"],
                    "label": sample["label"],
                }

#main.py
import os
os.environ["TRANSFORMERS_NO_MULTIPROCESSING"] = "1" 

from datasets import load_dataset

dataset = load_dataset("my_dataset.py", split="train", cache_dir=None)

print(dataset[:5])

### Expected behavior

Pause at breakpoints while running debugging

### Environment info

pycharm


Comments:
Issue: Adding column with dict struction when mapping lead to wrong order
State: open
Author: chchch0109

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A

Comments:
Issue: use huggingface_hub offline mode
State: closed
Author: lhoestq

and better handling of LocalEntryNotfoundError cc @Wauplin 

follow up to #7234 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7244). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: ArrayXD with None as leading dim incompatible with DatasetCardData
State: open
Author: alex-hh

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq 

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub

Comments:
It looks like `CardData` in `huggingface_hub` removes None values where it shouldn't. Indeed it calls `_remove_none` on the return of `to_dict()`:

```python
    def to_dict(self) -> Dict[str, Any]:
        """Converts CardData to a dict.

        Returns:
            `dict`: CardData represented as a dictionary ready to be dumped to a YAML
            block for inclusion in a README.md file.
        """

        data_dict = copy.deepcopy(self.__dict__)
        self._to_dict(data_dict)
        return _remove_none(data_dict)
```

Would it be ok to remove `list()` from being scanned in `_remove_none` ? it could also be a specific behavior to DatasetCardData if necessary @Wauplin 

I have actually no idea why none values are removed in model and dataset card data... :see_no_evil:
Looks like `_remove_none` has been introduced at the same time as the entire repocard module (see https://github.com/huggingface/huggingface_hub/pull/940). I would be tempted to remove `_remove_none` entirely actually and only remove "top-level" None values (i.e. if something like `pipeline_tag=None` due to a default value in kwargs => we remove it). Hard to tell what could be the side effects but I'm not against trying.


However, I'm not really in favor in making an exception only for lists. It would mean that tuples, sets and dicts are filtered but not lists, which is pretty inconsistent.

let's do it for top level attributes yes

I opened https://github.com/huggingface/huggingface_hub/pull/2626 to address it :)

thanks !
Issue: `push_to_hub` overwrite argument
State: closed
Author: ceferisbarov

### Feature request

Add an `overwrite` argument to the `push_to_hub` method.

### Motivation

I want to overwrite a repo without deleting it on Hugging Face. Is this possible? I couldn't find anything in the documentation or tutorials.

### Your contribution

I can create a PR.

Comments:
Hi ! Do you mean deleting all the files ? or erasing the repository git history before push_to_hub ?

Hi! I meant the latter.

I don't think there is a `huggingface_hub` utility to erase the git history, cc @Wauplin maybe ?

What is the goal exactly of deleting all the git history without deleting the repo? 

You can use [`super_squash_commit`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.super_squash_history) to squash all the commits into a single one, hence deleting the git history. This is not exactly what you asked for since it squashes the commits for a specific revision (example: "all commits on main"). This means that if other branches exists, they are kept the same. Also if some PRs are already opened on the repo, they will become unmergeable since the commits will have diverted.

So the solution is:

```python
from huggingface_hub import HfApi
repo_id = "username/dataset_name"
ds.push_to_hub(repo_id)
HfApi().super_squash_commit(repo_id)
```

This way you erase previous git history to end up with only 1 commit containing your dataset.
Still, I'd be curious why it's important in your case. Is it to save storage space ? or to disallow loading old versions of the data ?

Thanks, everyone! I am building a new dataset and playing around with column names, splits, etc. Sometimes I push to the hub to share it with other teammates, I don't want those variations to be part of the repo. Deleting the repo from the website takes a little time, but it also loses repo settings that I have set, since I always set it to public with manually approved requests.

BTW, I had to write `HfApi().super_squash_history(repo_id, repo_type="dataset")`, but otherwise it works.

@ceferisbarov just to let you know, recreating a gated repo + granting access to your teammates is something that you can automate with something like this (not fully tested but should work):

```py
from huggingface_hub import HfApi

api = HfApi()
api.delete_repo(repo_id, repo_type="dataset", missing_ok=True)
api.create_repo(repo_id, repo_type="dataset", private=False)
api.update_repo_settings(repo_id, repo_type="dataset", gated="manual")
for user in ["user1", "user2"]  # list of teammates
    api.grant_access(repo_id, user, repo_type="dataset")
```

I think it'd be a better solution than squashing commits (which is more of a hack), typically if you are using the dataset viewer.

This is great, @Wauplin. If we can achieve this with HfApi, then we probably don't need to add another parameter to push_to_hub. I am closing the issue.
Issue: Feature Request: Add functionality to pass split types like train, test in DatasetDict.map
State: closed
Author: jp1924

Hello datasets!

We often encounter situations where we need to preprocess data differently depending on split types such as train, valid, and test. 

However, while DatasetDict.map has features to pass rank or index, there's no functionality to pass split types. 
Therefore, I propose adding a 'with_splits' parameter to DatasetDict, which would allow passing the split type through fn_kwargs.

Comments:
Issue: incompatibily issue when using load_dataset with datasets==3.0.1
State: open
Author: jupiterMJM

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10


The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1

Comments:
Hi! I'm also getting the same issue - have you been able to find a solution to this? 

From what I remember, I stayed at the "downgraded" version of dataset (2.21.0)
Issue: No need for dataset_info
State: closed
Author: lhoestq

save a useless call to /api/datasets/repo_id

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7234). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merging this one for now, let me know if you'd like to see additional changes for error handling (I'll take care of them before doing a release)
Issue: Fix typo in image dataset docs
State: closed
Author: albertvillanova

Fix typo in image dataset docs.

Typo reported by @datavistics.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7231). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Video support
State: closed
Author: lhoestq

(wip and experimental)

adding the `Video` type based on `VideoReader` from `decord`

```python
>>>from datasets import load_dataset
>>> ds = load_dataset("path/to/videos", split="train").with_format("torch")
>>> print(ds[0]["video"])
<decord.video_reader.VideoReader object at 0x337a47910>
>>> print(ds[0]["video"][0])
tensor([[[73, 73, 73],
         [73, 73, 73],
         [73, 73, 73],
         ...,
         [23, 23, 23],
         [23, 23, 23],
         [23, 23, 23]]], dtype=torch.uint8)
```

the storage is the same as for audio and images: `{"path": pa.string(), "bytes": pa.binary()}` and I did a small to keep the hf:// URL in the "path" field if possible, this way the viewer can link to fiels on the hub if possible

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7230). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue: Composite (multi-column) features
State: open
Author: alex-hh

### Feature request

Structured data types (graphs etc.) might often be most efficiently stored as multiple columns, which then need to be combined during feature decoding

Although it is currently possible to nest features as structs, my impression is that in particular when dealing with e.g. a feature composed of multiple numpy array / ArrayXD's, it would be more efficient to store each ArrayXD as a separate column (though I'm not sure by how much)

Perhaps specification / implementation could be supported by something like:

```
features=Features(**{("feature0", "feature1")=Features(feature0=Array2D((None,10), dtype="float32"), feature1=Array2D((None,10), dtype="float32"))
```

### Motivation

Defining efficient composite feature types based on numpy arrays for representing data such as graphs with multiple node and edge attributes is currently challenging.

### Your contribution

Possibly able to contribute

Comments:
Issue: Add R as a How to use from the Polars (R) Library as an option
State: open
Author: ran-codes

### Feature request

The boiler plate code to access a dataset via the hugging face file system is very useful. Please addd 


## Add Polars (R) option
The equivailent code works, because the [Polars-R](https://github.com/pola-rs/r-polars) wrapper has hugging faces funcitonaliy as well.

```r
library(polars)

df <- pl$read_parquet("hf://datasets/SALURBAL/core__admin_cube_public/core__admin_cube_public.parquet")
```

## Polars (python) option
![image](https://github.com/user-attachments/assets/8f1bcd19-e578-4b18-b324-7cc00b80ac0a)


## Libraries Currently

![image](https://github.com/user-attachments/assets/0cf50063-f9db-443c-97b4-3ef0664b6e6e)




### Motivation

There are many data/analysis/research/statistics teams (particularly in academia and pharma) that use R as the default language. R has great integration with most of the newer data techs (arrow, parquet, polars) and having this included could really help in bringing this community into the hugging faces ecosystem.

**This is a small/low-hanging-fruit front end change but would make a big impact expanding the community**

### Your contribution

I am not sure which repositroy this should be in, but I have experience in R, Python and JS and happy to submit a PR in the appropriate repository. 

Comments:
Issue: Huggingface GIT returns null as Content-Type instead of application/x-git-receive-pack-result
State: open
Author: padmalcom

### Describe the bug

We push changes to our datasets programmatically. Our git client jGit reports that the hf git server returns null as Content-Type after a push.

### Steps to reproduce the bug

A basic kotlin application:
```
   val person = PersonIdent(
        "padmalcom",
        "padmalcom@sth.com"
    )

    val cp = UsernamePasswordCredentialsProvider(
        "padmalcom",
        "mysecrettoken"
    )

    val git =
        KGit.cloneRepository {
            setURI("https://huggingface.co/datasets/sth/images")
            setTimeout(60)
            setProgressMonitor(TextProgressMonitor())
            setCredentialsProvider(cp)
        }

    FileOutputStream("./images/images.csv").apply { writeCsv(images) }
    git.add {
        addFilepattern("images.csv")
    }

    for (i in images) {
        FileUtils.copyFile(
            File("./files/${i.id}"),
            File("./images/${i.id + File(i.fileName).extension }")
        )
        git.add {
            addFilepattern("${i.id + File(i.fileName).extension }")
        }
    }

    val revCommit = git.commit {
        author = person
        message = "Uploading images at " + LocalDateTime.now()
            .format(DateTimeFormatter.ISO_DATE_TIME)
        setCredentialsProvider(cp)
    }

    val push = git.push {
        setCredentialsProvider(cp)
    }
```

### Expected behavior

The git server is expected to return the Content-Type _application/x-git-receive-pack-result_.

### Environment info

It is independent from the datasets library.

Comments:
Issue: Fallback to arrow defaults when loading dataset with custom features that aren't registered locally
State: open
Author: alex-hh

### Describe the bug

Datasets allows users to create and register custom features.

However if datasets are then pushed to the hub, this means that anyone calling load_dataset without registering the custom Features in the same way as the dataset creator will get an error message.

It would be nice to offer a fallback in this case.

### Steps to reproduce the bug

```python
load_dataset("alex-hh/custom-features-example")
```
(Dataset creation process - must be run in separate session so that NewFeature isn't registered in session in which download is attempted:)

```python
from dataclasses import dataclass, field
import pyarrow as pa
from datasets.features.features import register_feature

from datasets import Dataset, Features, Value, load_dataset
from datasets import Feature

@dataclass
class NewFeature(Feature):
    _type: str = field(default="NewFeature", init=False, repr=False)
    def __call__(self):
        return pa.int32()

def examples_generator():
    for i in range(5):
        yield {"feature": i}

ds = Dataset.from_generator(examples_generator, features=Features(feature=NewFeature()))
ds.push_to_hub("alex-hh/custom-features-example")
register_feature(NewFeature, "NewFeature")
```



### Expected behavior

It would be nice, and offer greater extensibility, if there was some kind of graceful fallback mechanism in place for cases where user-defined features are stored in the dataset but not available locally.

### Environment info

3.0.2

Comments:
Issue: add CustomFeature base class to support user-defined features with encoding/decoding logic
State: closed
Author: alex-hh

intended as fix for #7220 if this kind of extensibility is something that datasets is willing to support!


```python
from datasets.features.features import CustomFeature


class ListOfStrs(CustomFeature):
    requires_encoding = True
    def _encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
feats.encode_example({"strlist": "a"})["strlist"] == feats["strlist"].encode_example("a")
```

Comments:
@lhoestq would you be open to supporting this kind of extensibility?

I suggested a fix in https://github.com/huggingface/datasets/issues/7220 that would not necessarily require a parent class for custom features, lmk what you think