noamaanMulla-03/datasets-issues huggingface dataset:
Preserve float columns when JSON loader uses `field=`

Closes #6937.

When `load_dataset("json", data_files=..., field="data", ...)` is used, columns whose values are all integer-valued floats (`[0.0, 1.0, 2.0]`) get silently coerced to `int64`. Repro:

```python
import tempfile, json
from datasets import load_dataset

with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
    json.dump({"data": [{"col": v} for v in [0.0, 1.0, 2.0]]}, f)

ds = load_dataset("json", data_files=f.name, field="data", split="train")
print(ds.features)  # before: {'col': Value('int64')}
                    # after:  {'col': Value('float64')}
```

The underlying cause is `pd.read_json(..., dtype_backend="pyarrow")` (tracked upstream at pandas-dev/pandas#58866). The path that hits it is the one introduced in #6914 to preserve column insertion order. The dataset-viewer statistics regression in the issue is a direct consequence.

This PR replaces the `pandas.read_json` -> `pa.Table.from_pandas` path in the `field=` branch with a small `_arrow_table_from_field` helper that builds the Arrow table directly from the already-parsed Python object. PyArrow's own JSON inference preserves `float64`, and CPython dict iteration order gives us the `#6914` column-insertion-order invariant for free (no pandas roundtrip needed).

The helper handles:

- list of dicts (the common case): collect keys in insertion order, build a column-major dict
- list of scalars: wrap in a single-column table named after the configured feature, falling back to "text" (matches the prior `df.columns == [0]` rename)
- dict of lists (column-major payload): pass through to `pa.Table.from_pydict`
- empty list with features supplied: emit empty columns matching the configured feature names so downstream `_cast_table` aligns

The other JSON loading path (raw JSON Lines via `paj.read_json`) does not have this bug and is unchanged.

There is an older dormant PR for this issue (#7635 from June 2025, no reviews). I went a different direction because that PR scans the resulting DataFrame and re-casts float-looking int columns back to float after the fact. By that point pandas has already converted the values to Python ints, so the `isinstance(x, float)` check there does not actually detect them, and the scan adds an O(rows) Python pass to a hot path. Sidestepping pandas entirely is simpler and faster.

Tests in `tests/packaged_modules/test_json.py`:

- `test_json_field_path_preserves_float_columns`: the exact #6937 repro, asserts `col` stays float64
- `test_json_field_path_preserves_float_columns_alongside_ints`: list-of-dicts payload with mixed int / float / int-valued-float columns, each type preserved
- `test_json_field_path_preserves_float_columns_with_dict_of_lists`: dict-of-lists field payload
- `test_json_field_path_preserves_column_order_with_list_of_dicts`: column insertion order check (sanity vs the #6914 invariant)

`pytest tests/packaged_modules/test_json.py` shows 33 passed, 0 failed (29 pre-existing + 4 new).
Fix parquet columns arg

Fix the error message "The columns and features argument must contain the same columns, but got.." when passing `columns=` to load_dataset on a streaming parquet dataset (with features declared in the README.md)

useful e.g. for https://huggingface.co/datasets/jasperai/monet to load the thumbnails without the embeddings


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8210). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix docs: use AutoImageProcessor for vision model examples

swaps AutoFeatureExtractor for AutoImageProcessor in the image preprocessing section of use_dataset.mdx — aligns with the transformers library direction where vision feature extractor references were already replaced in their own docs (see PRs #20111 and #20501). audio section is untouched since AutoFeatureExtractor is still correct there.



Thanks for the clear explanation! Good catch keeping AutoFeatureExtractor 
in the audio section since it's still valid there. The vision section 
update looks correct to me.
replace AutoFeatureExtractor with AutoImageProcessor in docs

Closes #8199

## What does this PR do?

Replaces `AutoFeatureExtractor` with `AutoImageProcessor` in the image augmentation example in `docs/source/use_dataset.mdx`. For vision models, `AutoImageProcessor` is the recommended API and better reflects current `transformers` conventions.

This is consistent with the direction taken in the `transformers` library itself — see [PR #20111](https://github.com/huggingface/transformers/pull/20111) and [PR #20501](https://github.com/huggingface/transformers/pull/20501), where vision feature extractor references were systematically replaced with image processors across the `transformers` codebase and documentation. The `datasets` docs were not in scope for that PR, making this a natural follow-up.
Something is wrong with WMT 19 kk-en dataset

The translation in the `train` set does not look right:

```
>>>import nlp
>>>from nlp import load_dataset
>>>dataset = load_dataset('wmt19', 'kk-en')
>>>dataset["train"]["translation"][0]
{'kk': 'Trumpian Uncertainty', 'en': 'Трамптық белгісіздік'}
>>>dataset["validation"]["translation"][0]
{'kk': 'Ақша-несие саясатының сценарийін қайта жазсақ', 'en': 'Rewriting the Monetary-Policy Script'}
```

Labels: name: dataset bug

Hi! Triaging older issues — I think this one can be closed as out-of-scope for the current `datasets` repo.

- The script-based `wmt19` loader that this issue targets is no longer hosted in this repo (the old `datasets/wmt19/` folder is gone — script-based loaders were migrated out, see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md)).
- Current WMT datasets live as Hub-hosted Parquet datasets (e.g. [`wmt/wmt19`](https://huggingface.co/datasets/wmt/wmt19) on the Hub). Any data-quality issue with that specific upload should be reported in the **Community** tab of that Hub dataset rather than here, since this repo no longer owns the data.

Would you mind closing this out? If you can reproduce a bug in the `datasets` library code itself (e.g. in [`src/datasets/`](https://github.com/huggingface/datasets/tree/main/src/datasets)) rather than in the dataset content, happy to take another look.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
New Datasets: IWSLT15+, ITTB

**Links:**
[iwslt](https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/datasets/iwslt.html)
Don't know if that link is up to date.

[ittb](http://www.cfilt.iitb.ac.in/iitb_parallel/)
**Motivation**: replicate mbart finetuning results (table below)
![image](https://user-images.githubusercontent.com/6045025/88490093-0c1c8c00-cf67-11ea-960d-8dcaad2aa8eb.png)


For future readers, we already have the following language pairs in the wmt namespaces:

```
wmt14: ['cs-en', 'de-en', 'fr-en', 'hi-en', 'ru-en']
wmt15: ['cs-en', 'de-en', 'fi-en', 'fr-en', 'ru-en']
wmt16: ['cs-en', 'de-en', 'fi-en', 'ro-en', 'ru-en', 'tr-en']
wmt17: ['cs-en', 'de-en', 'fi-en', 'lv-en', 'ru-en', 'tr-en', 'zh-en']
wmt18: ['cs-en', 'de-en', 'et-en', 'fi-en', 'kk-en', 'ru-en', 'tr-en', 'zh-en']
wmt19: ['cs-en', 'de-en', 'fi-en', 'gu-en', 'kk-en', 'lt-en', 'ru-en', 'zh-en', 'fr-de']
```

Labels: name: dataset request

Thanks Sam, we now have a very detailed tutorial and template on how to add a new dataset to the library. It typically take 1-2 hours to add one. Do you want to give it a try ?
The tutorial on writing a new dataset loading script is here: https://huggingface.co/nlp/add_dataset.html
And the part on how to share a new dataset is here: https://huggingface.co/nlp/share_dataset.html

Hi @sshleifer, I'm trying to add IWSLT using the link you provided but the download urls are not working. Only `[en, de]` pair is working. For others language pairs it throws a `404` error.



Hi! Triaging older issues this week — I think this one can be closed as out-of-scope for the current `datasets` repo.

- Script-based dataset loaders (the `datasets/wmt14/`, `datasets/iwslt/`, etc. folders that this issue was implicitly targeting) are no longer hosted in this repo. The current contribution model is "add the dataset directly to the Hub": see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md), which now redirects to https://huggingface.co/datasets and the [upload docs](https://huggingface.co/docs/datasets/upload_dataset).
- A quick search of the Hub already returns community uploads matching the request: see e.g. [IWSLT2017 on the Hub](https://huggingface.co/datasets?search=iwslt) — anyone wanting `iwslt15` / `ittb` can upload them as a regular Hub dataset and they will be loadable via `load_dataset("user/iwslt15")` without any change to this repo.

Would you mind closing this out? If I missed something that genuinely requires repo-side changes (rather than a Hub upload), point me at the specific piece and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
[Suggestion] Glue Diagnostic Data with Labels 

Hello! First of all, thanks for setting up this useful project!

I've just realised you provide the the [Glue Diagnostics Data](https://huggingface.co/nlp/viewer/?dataset=glue&config=ax) without labels, indicating in the `GlueConfig` that you've only a test set.

Yet, the data with labels is available, too (see also [here](https://gluebenchmark.com/diagnostics#introduction)):

https://www.dropbox.com/s/ju7d95ifb072q9f/diagnostic-full.tsv?dl=1 

Have you considered incorporating it?

Labels: name: Dataset discussion

Hi! Triaging older issues this week — I think this one can be closed as out-of-scope for the current `datasets` repo.

- Script-based dataset loaders (the `datasets/glue/` folder this issue was implicitly targeting) are no longer hosted in this repo. The current model is "add the dataset directly to the Hub": see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md), which redirects users to https://huggingface.co/datasets and the [upload guide](https://huggingface.co/docs/datasets/upload_dataset).
- The labeled GLUE diagnostic data referenced here can be uploaded by anyone as a Hub dataset (e.g. `username/glue-ax-labeled`) and loaded with `load_dataset("username/glue-ax-labeled")` — no repo change required.

Would you mind closing this out? If something specific still needs a code change in this repo rather than a Hub upload, point me at it and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
add MLDoc dataset

Hi,

I am recommending that someone add MLDoc, a multilingual news topic classification dataset.

- Here's a link to the Github: https://github.com/facebookresearch/MLDoc
- and the paper: http://www.lrec-conf.org/proceedings/lrec2018/pdf/658.pdf

Looks like the dataset contains news stories in multiple languages that can be classified into four hierarchical groups: CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). There are 13 languages: Dutch, French, German, Chinese, Japanese, Russian, Portuguese, Spanish, Latin American Spanish, Italian, Danish, Norwegian, and Swedish

Labels: name: dataset request

Any updates on this?

This request is still an open issue waiting to be addressed by any community member, @GuillemGSubies.

Hi! Triaging older issues this week — I think this one can be closed as out-of-scope for the current `datasets` repo.

- Script-based dataset loaders are no longer added to this repo. The current model is "upload directly to the Hub": see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md), which now redirects to https://huggingface.co/datasets and the [upload guide](https://huggingface.co/docs/datasets/upload_dataset).
- Anyone wanting MLDoc available via `load_dataset(...)` can publish it as a Hub dataset (subject to its license) and it becomes loadable as `load_dataset("user/mldoc")` without any change to this repo.

Would you mind closing this out? If you think MLDoc needs something specific from the loader side (rather than a Hub upload), let me know and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
Add learningq dataset

Hi, 

Thank you again for this amazing repo. 

Would it be possible for y'all to add the LearningQ dataset - https://github.com/AngusGLChen/LearningQ ? 



Labels: name: dataset request

Hi! Triaging older issues this week — I think this one can be closed as out-of-scope for the current `datasets` repo.

- Script-based dataset loaders are no longer added to this repo. The current model is "upload directly to the Hub": see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md) and the [upload guide](https://huggingface.co/docs/datasets/upload_dataset).
- Anyone interested in LearningQ being available via `load_dataset(...)` can publish it as a Hub dataset (subject to its license) and load with `load_dataset("user/learningq")` without any change required here.

Would you mind closing this out? Happy to be redirected if I missed something specific that needs repo-side work.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
Read cached dataset_info.json to populate config_names offline

## What

`get_dataset_config_names` with `HF_DATASETS_OFFLINE=1` returns `['default']` even when the cache contains multiple config subdirectories for the dataset. For example, after caching `cais/mmlu`:

```
$ HF_DATASETS_OFFLINE=1 python -c "import datasets; print(datasets.get_dataset_config_names('cais/mmlu'))"
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub (offline mode is enabled).
['default']
```

This is because `CachedDatasetModuleFactory.get_module` returns a `DatasetModule` without `builder_configs_parameters`, so `get_dataset_config_names` falls back to `default`.

## Fix

Walk the cached repo subdirectories, parse each `dataset_info.json`, and populate `builder_configs_parameters` with the discovered config names. Malformed or unreadable `dataset_info.json` files are skipped silently so a single bad cache entry does not break the lookup.

With this change:

```
$ HF_DATASETS_OFFLINE=1 python -c "import datasets; print(datasets.get_dataset_config_names('cais/mmlu'))"
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub (offline mode is enabled).
['abstract_algebra', 'all', 'anatomy', ...]
```

## Test

Adds `test_CachedDatasetModuleFactory_offline_populates_config_names` in `tests/test_load.py` that builds a fake cache layout with three configs, calls the factory with `HF_HUB_OFFLINE` patched to `True`, and asserts every cached config name is returned. No network access required.

Fixes #7947. Supersedes the abandoned #7977.
docs: surface `repository_structure` earlier in the Get Started flow

> **Dear maintainer** — this PR has a permanent home with methodology + [opt-out](https://github.com/adv0r/tokens-for-good/blob/main/MAINTAINER_REMOVAL.md) at [tokens-for-good](https://github.com/adv0r/tokens-for-good). A one-line "no thanks" → auto-close + blacklist. Sorry for the notification this edit caused.
>
> ---

Closes #5971

## What

The `repository_structure` doc page describes the simplest path to
creating a dataset (no dataset script required) but is currently
buried in the side navigation, so newcomers don't find it. This PR
repositions the entry in `docs/source/_toctree.yml` to surface it
earlier in the "Get started" / "Create a dataset" sequence.

## Why

The reporter on #5971 observed that the easiest onboarding flow is
not visible until users dig several layers into the docs.

## AI disclosure

Per project norms — this PR is AI-generated and unattended, written by
a Cursor Anthropic Claude Opus 4.7 agent under @adv0r as a personal
token-burn initiative before my Cursor billing cycle ends. Opened as
**draft** for review. Happy to iterate (in particular if maintainers
prefer a different placement or a cross-link instead).

Made with [Cursor](https://cursor.com)
Parallel downloads progress bar flickers

When there are parallel downloads using the download manager, the tqdm progress bar flickers since all the progress bars are on the same line.

To fix that we could simply specify `position=i` for i=0 to n the number of files to download when instantiating the tqdm progress bar. 

Another way would be to have one "master" progress bar that tracks the number of finished downloads, and then one progress bar per process that show the current downloads.


Hi! Triaging older issues this week — I think this one has shipped and can be closed.

The exact fix suggested in the original report (assign `position=i` to each parallel tqdm bar so they don't share a line) is in place on current `main` (4.8.6.dev0):

- `src/datasets/utils/py_utils.py:401` — the per-worker tqdm bar is opened with `position=rank` inside `_single_map_nested`, which is the helper that drives parallel work in `map_nested` (used by the download manager and other internal parallelization).
- `src/datasets/utils/py_utils.py:395` — there's an extra notebook-multiprocessing guard referencing [tqdm#485](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) so progress bars stay readable when `num_proc > 1` in notebooks too.

Together these are the two mechanisms one would land to fix the "all bars on the same line / flicker" symptom this issue describes.

Would you mind closing this out? If something specific is still flickering on a recent `datasets` release, point me at a repro (command + `num_proc`) and I'll dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
How to update the "wino_bias" dataset

Hi all,

Thanks for the efforts to collect all the datasets! But I think there is a problem with the wino_bias dataset. The current link is not correct. How can I update that?

Thanks!


Hi @JieyuZhao !

You can edit the dataset card of wino_bias to update the URL via a Pull Request. This would be really appreciated :)

The dataset card is the README.md file you can find at https://github.com/huggingface/datasets/tree/master/datasets/wino_bias
Also the homepage url is also mentioned in the wino_bias.py so feel free to update it there as well.

You can create a Pull Request directly from the github interface by editing the files you want and submit a PR, or from a local clone of the repository.

Thanks for noticing !

Hi! Triaging older issues — I think this one is out of scope for the `huggingface/datasets` repo on the current architecture.

Since the script-based dataset loaders were removed, `datasets` no longer ships individual dataset Python files. The `wino_bias` script that lived in this repo is gone and `load_dataset("wino_bias", …)` now resolves to whatever is hosted at https://huggingface.co/datasets/wino_bias on the Hub — so the right place to update broken links / data files is now the Hub repo, not this codebase.

- `src/datasets/load.py:1661-1666` — current `load_dataset` rejects `trust_remote_code` and explicitly tells users: "If the dataset is based on a loading script, please ask the dataset author to remove it and convert it to a standard format like Parquet." That's the migration path for any script-era dataset.
- The `datasets/` directory that used to contain the per-dataset Python scripts no longer exists at the repo root (only `src/`, `tests/`, `docs/`, `benchmarks/`, etc.).

If the issue is specifically about the Hub-hosted `wino_bias` data, opening a discussion on https://huggingface.co/datasets/wino_bias is the path that reaches the dataset author. Otherwise I'd suggest closing this out — happy to be wrong if I missed something.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)
XSum dataset missing spaces between sentences

I noticed that the XSum dataset has no space between sentences. This could lead to worse results for anyone training or testing on it. Here's an example (0th entry in the test set):

`The London trio are up for best UK act and best album, as well as getting two nominations in the best song category."We got told like this morning 'Oh I think you're nominated'", said Dappy."And I was like 'Oh yeah, which one?' And now we've got nominated for four awards. I mean, wow!"Bandmate Fazer added: "We thought it's best of us to come down and mingle with everyone and say hello to the cameras. And now we find we've got four nominations."The band have two shots at the best song prize, getting the nod for their Tynchy Stryder collaboration Number One, and single Strong Again.Their album Uncle B will also go up against records by the likes of Beyonce and Kanye West.N-Dubz picked up the best newcomer Mobo in 2007, but female member Tulisa said they wouldn't be too disappointed if they didn't win this time around."At the end of the day we're grateful to be where we are in our careers."If it don't happen then it don't happen - live to fight another day and keep on making albums and hits for the fans."Dappy also revealed they could be performing live several times on the night.The group will be doing Number One and also a possible rendition of the War Child single, I Got Soul.The charity song is a  re-working of The Killers' All These Things That I've Done and is set to feature artists like Chipmunk, Ironik and Pixie Lott.This year's Mobos will be held outside of London for the first time, in Glasgow on 30 September.N-Dubz said they were looking forward to performing for their Scottish fans and boasted about their recent shows north of the border."We just done Edinburgh the other day," said Dappy."We smashed up an N-Dubz show over there. We done Aberdeen about three or four months ago - we smashed up that show over there! Everywhere we go we smash it up!"`


Hi! Triaging older issues — I think this one can be closed as out-of-scope for the current `datasets` repo.

- The script-based `xsum` loader is no longer hosted in this repo (script-based loaders were migrated out, see [`ADD_NEW_DATASET.md`](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md) — content moved to the Hub).
- XSum now lives on the Hub (e.g. [`EdinburghNLP/xsum`](https://huggingface.co/datasets/EdinburghNLP/xsum)). Any content-formatting issue (missing spaces between sentences in the source text) is a property of the Hub upload, not the `datasets` library code, and should be raised in the **Community** tab of the relevant Hub dataset.

Would you mind closing this out? If you can reproduce a problem in `datasets` library code (e.g. in [`src/datasets/`](https://github.com/huggingface/datasets/tree/main/src/datasets)), happy to dig further.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)


Sure, thanks for the explanation!