Sulak2020/github-issues-multirepo-datasets-Sulakshana huggingface dataset:
Title: Add nifti support
State: closed

Add support for NIfTI.

supports #7804

This PR follows https://github.com/huggingface/datasets/pull/7325 very closely

I am a bit unsure what we need to add to the `document_dataset.mdx` and `document_load.mdx`. I should probably create a dataset on the hub first to create this guide instead of copy+pasting from PDF. 

Open todos:
 - [x] create nifti dataset on the hub
 - ~[ ] update `document_dataset.mdx` and `document_load.mdx`~
 
 EDIT:
 I tested with two datasets I created on the hub: 
  - https://huggingface.co/datasets/TobiasPitters/test-nifti-unzipped
  - https://huggingface.co/datasets/TobiasPitters/test-nifti
  
 for zipped (file extension `.nii.gz` and unzipped `.nii`) files and both seem to work fine. Also tested loading locally and that seems to work as well.
Here is the scriptsthat I ran against the hub:

```python
from pathlib import Path

from datasets import load_dataset
import nibabel as nib


dataset = load_dataset(
        "TobiasPitters/test-nifti-unzipped",
        split="test"  # Load as single Dataset, not DatasetDict
)

print("length dataset unzipped:", len(dataset))
for item in dataset:
    isinstance(item["nifti"], nib.nifti1.Nifti1Image)

dataset = load_dataset(
        "TobiasPitters/test-nifti",
        split="train"  # Load as single Dataset, not DatasetDict
)
print("length dataset zipped:", len(dataset))
for item in dataset:
    isinstance(item["nifti"], nib.nifti1.Nifti1Image)
```

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7815). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.Btw I couldn't resist but share your PR with the community online on twitter already, I hope this is fine !> Btw I couldn't resist but share your PR with the community online on twitter already, I hope this is fine !

Wow, that was quick! Thanks, already liked your comment, I appreciate it!NIfTI support is out in `datasets==4.4.0` ! 🥳 

Btw do you know a good NIfTI vizualizer in HTML/JS or using python ? We could add something like `.to_html()` (or equivalent) to view data in a notebook and enable the Dataset Viewer on HF if it can be useful

cc @cfahlgren1 @georgiachanning for viz
Title: Allow streaming hdf5 files
State: closed

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Parquet: add `on_bad_file` argument to error/warn/skip bad files
State: closed

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Less api calls when resolving data_files
State: closed

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: [Docs] Missing documentation for `Dataset.from_dict`
State: open

Documentation link: https://huggingface.co/docs/datasets/en/package_reference/main_classes

Link to method (docstring present): https://github.com/huggingface/datasets/blob/6f2502c5a026caa89839713f6f7c8b958e5e83eb/src/datasets/arrow_dataset.py#L1029

The docstring is present for the function, but seems missing from the official documentation for the `Dataset` class on HuggingFace.

The method in question:
```python
    @classmethod
    def from_dict(
        cls,
        mapping: dict,
        features: Optional[Features] = None,
        info: Optional[DatasetInfo] = None,
        split: Optional[NamedSplit] = None,
    ) -> "Dataset":
        """
        Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].

        Important: a dataset created with from_dict() lives in memory
        and therefore doesn't have an associated cache directory.
        This may change in the future, but in the meantime if you
        want to reduce memory usage you should write it back on disk
        and reload using e.g. save_to_disk / load_from_disk.

        Args:
            mapping (`Mapping`):
                Mapping of strings to Arrays or Python lists.
            features ([`Features`], *optional*):
                Dataset features.
            info (`DatasetInfo`, *optional*):
                Dataset information, like description, citation, etc.
            split (`NamedSplit`, *optional*):
                Name of the dataset split.

        Returns:
            [`Dataset`]
        """
```

Comments: I'd like to work on this documentation issue.Hi I'd like to work on this. I can see the docstring is already in the code. 
Could you confirm:
1. Is this still available?
2. Should I add this to the main_classes.md file, or is there a specific 
   documentation file I should update?
3. Are there any formatting guidelines I should follow?

I'm new to contributing but eager to learn!
Title: Add parquet scan options and docs
State: closed

I added scan options, useful to control buffering and caching when streaming and docs, including how to select a subset of columns and apply filters

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Audio dataset is not decoding on 4.1.1
State: open

### Describe the bug

The audio column remain as non-decoded objects even when accessing them.

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```

Works fine with `datasets==3.6.0`

Followed the docs in 

- https://huggingface.co/docs/datasets/en/audio_load

### Steps to reproduce the bug

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```


### Expected behavior

It should decode when accessing the elemenet

### Environment info

4.1.1
ubuntu 22.04


Related

- https://github.com/huggingface/datasets/issues/7707

Comments: Previously (datasets<=3.6.0), audio columns were decoded automatically when accessing a row. Now, for performance reasons, audio decoding is lazy by default: you just see the file path unless you explicitly cast the column to Audio.

Here’s the fix (following the current [datasets audio docs](https://huggingface.co/docs/datasets/en/audio_load)
):

```
from datasets import load_dataset, Audio

dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly decode the audio column
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset[0]["audio"])
# {'path': '...', 'array': array([...], dtype=float32), 'sampling_rate': 16000}
```@haitam03-yo's comment is right that the data is not decoded by default anymore indeed, but here is how it works in practice now:

From `datasets` v4, audio data are read as [AudioDecoder](https://meta-pytorch.org/torchcodec/0.4/generated/torchcodec.decoders.AudioDecoder.html) objects from torchcodec. This doesn't decode the data by default, but you can call `audio.get_all_samples()` to decode the audio.

See the documentation on how to process audio data here: https://huggingface.co/docs/datasets/audio_processTo resolve this, you need to explicitly cast the audio column to the Audio feature. This will decode the audio data and make it accessible as an array. Here is the corrected code snippet


from datasets import load_dataset, Audio

# Load your dataset
dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly cast the 'audio' column to the Audio feature
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

# Now you can access the decoded audio array
print(dataset[0]["audio"])

By adding the cast_column step, you are telling the datasets library to decode the audio data with the specified sampling rate, and you will then be able to access the audio array as you were used to in previous versions.
Title: Datasets: Add WMT21 & WMT22 loaders (basic TSV loaders, sample data, tests)
State: closed

- Implemented TSV-based dataset loaders:
   - WMT21Dataset (local_datasets/wmt21/wmt21_dataset.py)
   - WMT22Dataset (local_datasets/wmt22/wmt22_dataset.py)
   These classes load source-target pairs from .tsv files for train, validation, and test splits.
- Created sample dummy data for both datasets:
   - dummy_data/train.tsv, dummy_data/validation.tsv, dummy_data/test.tsv
   - Includes a few realistic example lines to allow CI and local tests to pass without downloading full datasets.
- Added automated tests for robust validation:
   - tests/test_wmt21.py and tests/test_wmt22.py
   - Checks that all splits load correctly, empty lines are ignored, and the number of examples matches the number of lines in the .tsv files.
   - Edge cases handled: empty lines, malformed lines, extra tabs.
- Added README.md files for both datasets:
   - Provides dataset structure, usage instructions, and placeholders for citation & license information.
   - Ensures that other developers and reviewers can understand dataset usage immediately.
- Ensured easy local testing:
   - Load datasets programmatically using WMT21Dataset / WMT22Dataset.
   - Verified train/validation/test splits are correctly returned as Python dictionaries of Dataset objects.


-Provides initial support for WMT21 and WMT22 NLP/translation experiments.
-Allows contributors and reviewers to test dataset loading locally or in CI without downloading large datasets.
-Serves as a template to extend to other WMT datasets in the future.

Testing instructions:
```
# Activate your environment
pytest tests/test_wmt21.py -v
pytest tests/test_wmt22.py -v
```
```
from local_datasets.wmt21.wmt21_dataset import WMT21Dataset
from local_datasets.wmt22.wmt22_dataset import WMT22Dataset

# WMT21
wmt21 = WMT21Dataset("local_datasets/wmt21/dummy_data")
ds21 = wmt21.load()
print(ds21["train"][0])
print(ds21["validation"][0])
print(ds21["test"][0])

# WMT22
wmt22 = WMT22Dataset("local_datasets/wmt22/dummy_data")
ds22 = wmt22.load()
print(ds22["train"][0])
print(ds22["validation"][0])
print(ds22["test"][0])
```

Comments: closing since datasets should be added on https://huggingface.co directly
Title: Add pyarrow's binary view to features
State: closed

Basically https://github.com/huggingface/datasets/pull/7718 just for binary view instead of string view

Comments: @lhoestq 🙏 The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix: add `num_proc` argument to `Dataset.to_sql`
State: open

**Task Done:** 

- Resolve issue #7788 : Add the missing argument mapping in Dataset.to_sql (`src/datasets/arrow_dataset.py`)




Comments: Hi ! could you also write a test to make sure this works fine ?

(in case there needs to be a special logic to handle the concurrent writes to the database)Hi @lhoestq 
Done! Let me know if more is needed :)
Title: `Dataset.to_sql` doesn't utilize `num_proc`
State: open

The underlying `SqlDatasetWriter` has `num_proc` as an available argument [here](https://github.com/huggingface/datasets/blob/5dc1a179783dff868b0547c8486268cfaea1ea1f/src/datasets/io/sql.py#L63) , but `Dataset.to_sql()` does not accept it, therefore it is always using one process for the SQL conversion.
Title: Fix Audio docstring by removing unsupported mono argument
State: closed

This PR fixes issue #7745.

Who can review:
@lhoestq 

Comments: I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method> I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

Thank you @lhoestq. I will add torch.mean().fixed by #7840
Title: Support huggingface_hub v0.x and v1.x
State: closed

Related to https://github.com/huggingface/huggingface_hub/issues/3340.

This PR adapts `datasets` to be compatible with both huggingface_hub v0.x and v1.x. 

In practice nothing else should change (I've checked the codebase). The `HfHubHTTPError` is a base error defined in `huggingface_hub` that inherits from `requests.HTTPError` in v0.x and will inherit from `httpx.HTTPError` in v1.x. It has been introduced ~2 years ago so it's fine to use it right now (i.e. no need to wait for v1.x release or bump minimal version).

Most of the changes have been around the test suite to make sure that tests are passing with both `requests` and `httpx` backends. Mid-term it would be good to completely remove the `requests` dependency from `datasets` but that's an orthogonal topic. 

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.@lhoestq could you have a look at this PR please? It makes `datasets` compatible with the upcoming `huggingface_hub` v1.0 release while staying compatible with 0.x. Look at PR description for more details.

The CI is currently failing because of 429 rate limit errors but otherwise everything should be fine (made extensive local and ci tests to ensure that). Let me know if you notice anything weird. PR is ready to be merged "as-is" in my opinion.
Title: BIGPATENT dataset inaccessible (deprecated script loader)
State: closed

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.



Comments: Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !The dataset now works with `datasets` v4 ! closing this issue
Title: [FIX] force spawning pool for MacOS
State: open

This PR gets multiprocessing to work on mac os:

```python
from datasets import load_dataset

ds = load_dataset("fka/awesome-chatgpt-prompts", split="train").take(100)

ds = ds.map(lambda x: x, num_proc=4)

ds.push_to_hub("burtenshaw/dataset-test", num_proc=4)
```

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.After more discussions on slack, we can switch the default to spawn.

Let's use multiprocess instead of multiprocessing and maybe add a check to apply this only on Macos ?
Title: push_to_hub not overwriting but stuck in a loop when there are existing commits
State: closed

### Describe the bug

`get_deletions_and_dataset_card` stuck at error a commit has happened error since push to hub for http error 412 for tag 4.1.0. The error does not exists in 4.0.0. 

### Steps to reproduce the bug

Create code to use push_to_hub, ran twice each time with different content for datasets.Dataset.
The code will stuck in time.sleep loop for `get_deletions_and_dataset_card`. If error is explicitly printed, the error is HTTP 412.

### Expected behavior

New datasets overwrite existing one on repo.

### Environment info

datasets 4.1.0

Comments: HTTP 412 means a commit happened in the meantime, so `get_deletions_and_dataset_card` has to retry to get the latest version of the dataset card and what files to delete based on the latest version of the dataset repository

Are you running other operations in the dataset repo for your push_to_hub ?There was only a map() followed by a push_to_hub(). The repo had one prior commit also by using push_to_hub(). The error disappeared when I downgraded datasets to 4.0.0.It is reproducible if you use finegrained token with Read+Write (Open pull request) access to only that repo.Ah it was due to the use of requests_cache with POST methods, closing this.
Title: [docs] Fix broken WebDataset link on “Create a video dataset” page
State: closed

### What
Fix the "WebDataset documentation" link on the Create a video dataset page to point
to the correct section on the video load guide.

### Why
The link currently points to an external repo, but the Hugging Face docs
have an internal "WebDataset" section under video_load.

### How
- docs/source/video_dataset.mdx: updated link to
  `https://huggingface.co/docs/datasets/main/en/video_load#webdataset`

### Issue
Fixes #7699
Title: Error processing scalar columns using tensorflow.
State: open

`datasets==4.0.0`
```
columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']
train_ds.set_format(type='tf', columns=columns_to_return)
```
`train_ds`:
```
train_ds type: <class 'datasets.arrow_dataset.Dataset'>, shape: (1000, 9)
columns: ['question', 'sentences', 'answer', 'str_idx', 'end_idx', 'input_ids', 'attention_mask', 'start_positions', 'end_positions']
features:{'question': Value('string'), 'sentences': Value('string'), 'answer': Value('string'), 'str_idx': Value('int64'), 'end_idx': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8')), 'start_positions': Value('int64'), 'end_positions': Value('int64')}
```
`train_ds_tensor = train_ds['start_positions'].to_tensor(shape=(-1,1))` hits the following error:
```
AttributeError: 'Column' object has no attribute 'to_tensor'
```
`tf.reshape(train_ds['start_positions'], shape=[-1,1])` hits the following error:
```
TypeError: Scalar tensor has no `len()`
```

Comments: Using tf.convert_to_tensor works fine:

```
import tensorflow as tf

start_pos = tf.convert_to_tensor(train_ds['start_positions'], dtype=tf.int64)
start_pos = tf.reshape(start_pos, [-1, 1])
```


Alternatively, using the built-in to_tf_dataset also avoids the issue:

```
train_tf = train_ds.to_tf_dataset(
    columns=['input_ids','attention_mask'],
    label_cols=['start_positions','end_positions'],
    shuffle=True,
    batch_size=32
)
``````
    start_pos = tf.convert_to_tensor(self._train_ds['start_positions'], dtype=tf.int64)
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/constant_op.py", line 108, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: TypeError: Scalar tensor has no `len()`
Traceback (most recent call last):

  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/ops.py", line 361, in __len__
    raise TypeError("Scalar tensor has no `len()`")

TypeError: Scalar tensor has no `len()`
```

`to_tf_dataset` works perfectly.
Title: Add support for arrow iterable when concatenating or interleaving
State: closed

Fixes a case when concatenating or interleaving datasets with `with_format(...)` call was slower.
Details here: https://github.com/huggingface/datasets/issues/6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.

Comments: Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

```
1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])
```
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci? @lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?They seem unrelated to your changes. Merging :)
Title: Fix: Correct float feature generation in `generate_examples` 
State: open

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.


**Key changes include:**

* Added explicit handling for `float` features using `np.random.rand` to generate continuous values.
* Introduced fail-fast type checks for unsupported dtypes to improve robustness.
* Added validation for sequence features to ensure `seq_shapes` is provided.

### Before Fix

Float features were generated incorrectly as integers cast to float:

```text
- Example 0:
- int_feature: 0
- float_feature: 9.0  <-- Incorrect: An integer disguised as a float
- string_feature: The small grey turtle was surprisingly fast...
- seq_feature: [0.3048 0.4291 0.4283]
```

### After Fix

Float features are now correctly generated as continuous numbers in the range [0, 1):
```text
+ Example 0:
+ int_feature: 0
+ float_feature: 0.0183  <-- Correct: A true random float
+ string_feature: The small grey turtle was surprisingly fast...
+ seq_feature: [0.9237 0.7972 0.8526]
```
#### Note: This PR is a follow-up/fix of the previously closed PR #7769 for clarity and context.


Comments: Hi @lhoestq, just a gentle follow-up on this PR.
Title: Fix: Correct float feature generation in `generate_examples`
State: closed

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.

**Key changes include:**
1.  Added explicit handling for float features using `np.random.rand` to generate continuous values.
2.  Introduced fail-fast type checks for unsupported dtypes to improve robustness.
3.  Added validation for sequence features to ensure `seq_shapes` is provided.


### Before Fix
Float features were generated incorrectly as integers cast to float:

```text
Example 0:
int_feature: 0
float_feature: 9.0  <-- Incorrect: An integer disguised as a float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.3048 0.4291 0.4283]
```
### After Fix
Float features are now correctly generated as continuous numbers in the range [0, 1):
```text

Example 0:
int_feature: 0
float_feature: 0.0183  <-- Correct: A true random float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.9237 0.7972 0.8526]