TydiQA dataset is mixed and is not split per language 

Hi @lhoestq 
Currently TydiQA is mixed and user can only access the whole training set of all languages:
https://www.tensorflow.org/datasets/catalog/tydi_qa

for using this dataset, one need to train/evaluate in each separate language, and having them mixed, makes it hard to use this dataset. This is much convenient for user to have  them split and I appreciate your help on this. 

Meanwhile, till hopefully this is split per language, I greatly appreciate telling me how I can preprocess and get data per language. thanks a lot 


You can filter the languages this way:
```python
tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
```

Otherwise maybe we can have one configuration per language ?
What do you think of this for example ?

```python
load_dataset("tydiqa", "primary_task.en")
```

Hi
thank you very much for the great response, this will be really wonderful
to have one configuration per language, as one need the dataset in majority
of case per language for cross-lingual evaluations.
This becomes also then more close to TFDS format, which is separated per
language https://www.tensorflow.org/datasets/catalog/tydi_qa which will be
really awesome to have.
thanks

On Mon, Mar 29, 2021 at 6:17 PM Quentin Lhoest ***@***.***>
wrote:

> You can filter the languages this way:
>
> tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
>
> Otherwise maybe we can have one configuration per language ?
> What do you think of this for example ?
>
> load_dataset("tydiqa", "primary_task.en")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2132#issuecomment-809516799>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXPW2PWSQ2RHG73O7TTGCY4LANCNFSM4Z7ER7IA>
> .
>


@lhoestq  I greatly appreciate any updates on this. thanks a lot

Hi! Triaging older issues — I think this one is out of scope on the current architecture.

The "TydiQA is mixed, not split per language" behavior was baked into the old `tydiqa/tydiqa.py` loading script in this repo. Script-based loading has been removed:

- `src/datasets/load.py:1661-1666` — `load_dataset` no longer supports loading scripts; the error message points users to datasets "in a standard format like Parquet."
- There's no `datasets/tydiqa/` directory in this repo anymore; TydiQA now lives on the Hub (https://huggingface.co/datasets/tydiqa) and any per-language config has to be added on that Hub repo (typically by re-uploading Parquet shards partitioned by language as separate configs).

Until someone re-shards on the Hub, a workable client-side workaround is to load the mixed dataset and filter by the `id` prefix (TydiQA encodes the language in the example id, e.g. `english--…`, `finnish--…`):
```python
ds = load_dataset("tydiqa", "primary_task", split="train")
ds_en = ds.filter(lambda ex: ex["id"].startswith("english"))
```

Would you mind closing this out? If you want per-language configs on the Hub, opening a discussion on https://huggingface.co/datasets/tydiqa is the right path.

(Disclosure: I drafted this comment with help from Claude Code while triaging stale issues; the references above were verified manually against the current `main` branch at commit 19302564.)