SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> To get a visual idea of these first exchanges, I coded a Streamlit app that I put online on Spaces: https://huggingface.co/spaces/lbourdois/Language-tags-demo.
> The code is in Python so I don't know if it can be used by HF who seems to need something in Node.js but it serves as a proof of concept. The advantage is also that you can directly test ideas by enter things in a search bar and see what comes up.

This is really great. You're doing a fantastic job. I love watching the creative process evolve. It is exciting. Let me provide some links to some search interfaces for further inspiration. I always find it helpful to know how others have approached a problem when figuring out my approach. I will link to three examples Glottolog, r12a's language sub-tag chooser, and the FLEx project builder wizard. The first two are online, but the last one is in an application which must be downloaded and works only on windows or linux. I have placed some notes on each of the screenshots.

* **[Glottolog](https://glottolog.org/)** | [Search Query](https://glottolog.org/glottolog?name=en&namequerytype=part&multilingual=on#2/20.9/150.0) 

![Glottolog1](https://user-images.githubusercontent.com/40230/188494425-84ee6ecf-6868-4684-a4ae-008973f3b367.png)
![Glottolog2](https://user-images.githubusercontent.com/40230/188494426-fc1c225c-f99a-46b5-a1aa-950cf7912ce3.png)


* **[r12a language sub-tag chooser](https://r12a.github.io/app-subtags/)** | [Code on github](https://github.com/r12a/app-subtags)

![r12a1](https://user-images.githubusercontent.com/40230/188495349-8e53be68-8433-46ff-a0c7-c2f6e25458b6.png)


* **FLEx Language Chooser** | [application page](https://software.sil.org/fieldworks/)
![FLEx1](https://user-images.githubusercontent.com/40230/188499742-82c5601e-7e37-4863-bd63-8bff8c0694e3.png)
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> In practice, what I do is if we enter `fr-CA` is that the letters before the `-` refer to a language in the `Name` column for a `639-1` == `fr` (`639-3` for `fra` or `fre`) in the base of my image above. Then I look at the letters after the `-` which refers to a territory. It comes out `French (Canada)`. I used https://cldr.unicode.org/translation/displaynames/languagelocale-name-patterns for the pattern that came up.

What you are doing is looking at the algorithm for Locale generation rather than BCP-47's original documentation. I'm not sure there are difference, there might be. I know that locale IDs generally follow BCP-47 But I think there are some differences such as the use of `_` vs. `-`.
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> A first question: based on the Glottocode, would it be possible to grab the closest ISO639-3 code? In case there is no match for the exact language variety, one needs to explore the higher-level groupings, level by level. For this language (Japhug), the information provided in the extracted CSV file (`glottolog-languoids-v4.6.csv`) is: `sino1245/burm1265/naqi1236/qian1263/rgya1241/core1262/jiar1240` One need not look further than the first higher-level grouping, [`jiar1240`](https://glottolog.org/resource/languoid/id/jiar1240), to get an ISO639-3 code, namely `jya`.
> 
> Thus users searching by language names would get ISO639-3 (often less fine-grained than Glottolog) as a bonus. It might be possible to ask the Glottolog team to provide this piece of information as part of an export from their database.

This is logical, but the fine grained assertions are not the same. That is just because they are in a hierarchical structure today doesn't mean they will be tomorrow. In some cases the glottolog is clearly referring to sub-language variants which will never receive full language status, whereas in other cases glottolog is referencing to unequal entities one or more of which should be a language. Many of the codes in glottolog have no associated documentation indicating what sort of speech variety they are.
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
@lbourdois 
> * Since Glottolog can be trust, I will add a condition to the code that if there is no ISO 639-3 code in the "official" database (https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab), look for it in the "ISO639P3code" column of Glottolog.

I'm confused here... if there is no ISO639-3 code in the official database from the registrar, why would you look for an "unofficial" code from someone else? What is the use case here?
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> For the point of adding the closest ISO 639-3 code for a Glottolog code, what convention should be adopted for the output? Just the ISO 639-3 code, or the ISO 639-3 code - Glottolog code, or the ISO 639-3 code - language name?
To use the example of Japhug , should it be just jya, or jya-japh1234 or jya-Japhug?

(answer edited in view of [Benjamin Galliot's comment](https://github.com/huggingface/datasets/issues/4881#issuecomment-1237420600) 
Easy part of the answer first: jya-Japhug is out, because, as @BenjaminGalliot pointed out above, mixing language names with language codes will make trouble. For Japhug, `jya-Japhug` looks rather good: the pair looks nice, the one (`jya`) packed together, the other (`Japhug`) good and complete while still pretty compact.  But think about  languages like 'Yongning Na' or 'Yucatán Maya': a code with a space in the middle, like `nru-Yongning Na`, is really unsightly and unwieldy, not?

Some [principles for language naming in English](http://hdl.handle.net/10125/24725) have been put forward, with some linguistic arguments, but always supposing that such standardization is desirable, actual standardization of language names in English may well never happen.

As for `jya-japh1234`: again, at first sight it seems cute, combining two fierce competitors (Ethnologue and Glottolog) into something that gets the best of both worlds. 
But @HughP has a point: _adding additional codes is a waste of time both for HF devs and for future users who get a confusing idea about language tagging_ Strong wording, for an important comment: better stick with BCP 47. 

So the solution pointed out by Benjamin, from Frances Gillis-Webber and Sabine Tittel, looks attractive: 
jya-x-japh1234

On the other hand, if the idea for HF Datasets is simply to add the closest ISO 639-3 code for a Glottolog code, maybe it could be provided simply in three letters: providing the 'raw' ISO 639-3 code `jya`. Availability of 'straight' ISO 639-3 codes could save trouble for some users, and those who want more detail could look at the rest of the metadata and general information associated with datasets.
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
The problem seems to have already been raised here: https://drops.dagstuhl.de/opus/volltexte/2019/10368/pdf/OASIcs-LDK-2019-4.pdf

An example can be seen here :

> 3.1.2 The use of privateuse sub-tag
In light of unambiguous language codes being available for the two Khoisan varieties, we
propose to combine the ISO 639-3 code for the parent language N‖ng, i.e., ‘ngh’, with the
privateuse sub-tag ‘x-’ and the respective Glottocodes stated above.
The language tags for N|uu and ‖’Au can then be defined accordingly:
N|uu: ngh-x-nuuu1242
‖’Au: ngh-x-auni1243

By the way, while searching for this, I came across this application: https://huggingface.co/spaces/cdleong/langcode-search
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> > * Since Glottolog can be trust, I will add a condition to the code that if there is no ISO 639-3 code in the "official" database (https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab), look for it in the "ISO639P3code" column of Glottolog.
> 
> I'm confused here... if there is no ISO639-3 code in the official database from the registrar, why would you look for an "unofficial" code from someone else? What is the use case here?

Hi @HughP, I'm happy to clear what confusion may exist here :innocent: Here is the use case. 
Guillaume Jacques (@rgyalrong) put together a sizeable corpus of the Japhug language. It is up on HF Datasets ([here](https://huggingface.co/datasets/Lacito/pangloss/viewer/japh1234)) as well as on Zenodo. 

Zenodo is an all-purpose repository without adequate domain-specific metadata ("[métadonnées métier](https://www.cines.fr/archivage/des-expertises/les-metadonnees/metadonnees-metier/)"), and the deposits in there are not easy to locate. The Zenodo deposit is intended for a highly specific user case: someone reads about the dataset in a paper, goes to the address on Zenodo and grabs the dataset at one go. 

HF Datasets, on the other hand, allows users to look around among corpora. The Japhug corpus needs proper tagging so that HF Datasets users can find out about it. 
Japhug has an entry of its own in Glottolog, whereas it lacks an entry of its own in Ethnologue. Hence the practical usefulness of Glottolog. Ethnologue pools together, under the code `jya`, three different languages (Japhug, Tshobdun `tsho1240` and Zbu `zbua1234`). 

I hope that this helps.
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
> By the way, while searching for this, I came across this application: https://huggingface.co/spaces/cdleong/langcode-search

Really relevant Space, so tagging its author @cdleong, just in case!
Issue: Language names and language codes: connecting to a big database (rather than slow enrichment of custom list)

Comment:
@cdleong A one-stop shop for language codes: terrific!
How do you feel about the use of Glottocodes? When searching the language names 'Japhug' and 'Yongning Na' (real examples, related to a HF Datasets deposit & various research projects), the relevant Glottocodes are retrieved, and that is great (and not that easy, notably with the space in the middle of 'Yongning Na'). But this positive result is 'hidden' in the results page. Specifically: 

- for Japhug: when searching by language name ('Japhug'), the result in big print is 'Failure', even though there is an available Glottocode (at bottom).
![image](https://user-images.githubusercontent.com/6072524/188604619-a5032f53-6d2c-4751-b83b-bf70a5bf3b22.png)
When searching by Glottocode (japh1234), same outcome: 'Result: failure!' (even though this _is_ the right Glottocode
When searching by x-japh1234 (Glottocode encapsulated in BCP 47 syntax), one gets the message 

> ''x-japh1234' parses meaningfully as a language tag according to IANA"

but there is paradoxically no link provided to Glottolog: the 'Glottolog' part of the results page is empty
![image](https://user-images.githubusercontent.com/6072524/188605698-91a39982-ae70-4c48-94ae-cceeb06c25f5.png)

- Yongning Na: the correct code is identified (yong1288) but instead of foregrounding this exact match, the first result that comes up is a completely different language, called 'Yong'. 

Trying to formulate a conclusion (admittedly, this note is not based on intensive testing, it is just feedback on initial contact): from a user perspective, it seems that the tool could make more extensive use of Glottolog. `langcode-search` does a great job querying Glottolog, why not make more extensive use of that information? (including: to arrive at the nearest ISO 639-3 code)
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
Love this in principle 🚀 

Let's keep in mind users might rely on `dataset_infos.json` already.

I'm not convinced by the two-syntax solution, wouldn't it be simpler to have only one syntax with a `default` config for datasets with only one config? ie, always having the `configs` field. This makes parsing the metadata easier IMO.

Might also be good to wrap the tags under a `datasets_info` tag as follows:

```yaml
description: ...
citation: ...
dataset_infos:
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  configs:
    - ...
[...]
```

Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> Let's keep in mind users might rely on dataset_infos.json already.

Yea we'll full full backward compatibility

> Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.

The main things that may use or ingest these data IMO are:
- users in the UI or IDE
- `datasets` to populate `DatasetInfo` python object
- moon landing which is already parsing YAML

Am I missing something ? If not I think it's ok to use YAML

> Might also be good to wrap the tags under a datasets_info tag as follows:

Maybe one single syntax like this then ?
```yaml
dataset_infos:
- config: unlabeled
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  splits:
  - name: train
    num_examples: 10000
  features:
  - name: text
    dtype: string
- config: labeled
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  splits:
  - name: train
    num_examples: 100
  features:
  - name: text
    dtype: string
  - name: label
    dtype: ClassLabel
    names:
    - negative
    - positive
```
and when you have only one config
```yaml
dataset_infos:
- config: default
  splits:
  - name: train
    num_examples: 10000
  features:
  - name: text
    dtype: string
```
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
love the idea, and the trend in general to move more things (like tasks) to a single place (YAML).

also, if you browse files on a dataset's page (in "Files and versions"), raw `README.md` files looks nice and readable, while `.json` files are just one long line that users need to scroll. 

> Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.

do users often parse `datasets_infos.json` file themselves?
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> do users often parse datasets_infos.json file themselves?

Not AFAIK, but I'm sure there should be a few users.
Users that access these info via the `DatasetInfo` from `datasets` won't see the change though e.g.
```python
>> from datasets import get_datasets_infos
>>> get_datasets_infos("squad")
{'plain_text': DatasetInfo(description='Stanford Question Answering Dataset...
```
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> Maybe one single syntax like this then ?

LGTM!

> The main things that may use or ingest these data IMO are:
> - users in the UI or IDE
> - datasets to populate DatasetInfo python object
> - moon landing which is already parsing YAML

Fair point!

Having dataset info in the README's YAML is great for API / `huggingface_hub` consumers as well as it will be inserted in the  `cardData` field out of the box 🔥
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
Very supportive of this!

Nesting an array of configs inside `dataset_infos: ` sounds good to me. One small tweak is that `config: default` can be optional for the default config (which can be the first one by convention)

We'll be able to implement metadata validation on the Hub side so we ensure that those metadata are always in the right format (maybe for @coyotte508 ? cc @Pierrci). From a quick glance the `features` might be the harder part to validate here, any doc will be welcome.

### Other high-level points:
- as we move from mostly academic datasets to *all* datasets (which include the data inside the repos), my intuition is that more and more datasets (Hub-stored) are going to be **single-config**
- similarly, less and less datasets will have a loading script, **just the data + some metadata**
- to lower the barrier to entry to contribution, in the long term users shouldn't need to compute/update this data via a command line. It could be filled automatically on the Hub through a "bot" inside Discussions & Pull requests for instance.
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
re: `config: default`

Note also that the default config is not named `default`, afaiu, but create from the repo name, eg: https://huggingface.co/datasets/nbtpj/bionlp2021SAS default config is `nbtpj--bionlp2021SAS` (which is awful)
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> Note also that the default config is not named default, afaiu, but create from the repo name, eg: https://huggingface.co/datasets/nbtpj/bionlp2021SAS default config is nbtpj--bionlp2021SAS (which is awful)

We can change this to `default` I think or something else
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> From a quick glance the features might be the harder part to validate here, any doc will be welcome.

I dug into features validation, see:

- the OpenAPI spec: https://github.com/huggingface/datasets-server/blob/main/chart/static-files/openapi.json#L460-L697
- the node.js code: https://github.com/huggingface/moon-landing/blob/upgrade-datasets-server-client/server/lib/datasets/FeatureType.ts
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> We can change this to default I think or something else

I created https://github.com/huggingface/datasets/issues/4902 to discuss that
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
> Note also that the default config is not named `default`, afaiu, but create from the repo name

in case of single-config you can even hide the config name from the UI IMO

> I dug into features validation, see: the OpenAPI spec

in moon-landing we use [Joi](https://joi.dev/api/) to validate metadata so we would need to generate from Joi code from the OpenAPI spec (or from somewhere else) but I guess that's doable – or just rewrite it manually, as it won't change often
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
I remember there was an ongoing discussion on this topic:
- #3507

I recall some of the concerns raised on that discussion:
- @lhoestq:  Tensorflow Datasets catalog includes a community catalog where you can find and use HF datasets. They are using the exported dataset_infos.json files from github to get the metadata: [#3507 (comment)](https://github.com/huggingface/datasets/issues/3507#issuecomment-1056997627)
- @severo: [#3507 (comment)](https://github.com/huggingface/datasets/issues/3507#issuecomment-1042779776)
  - the metadata header might be very long, before reaching the start of the README/dataset card. 
  - It also somewhat prevents including large strings like the checksums
  - two concepts are mixed in the same file (metadata and documentation). This means that if you're interested only in one of them, you still have to know how to parse the whole file. 
- @severo: the future "datasets server" could be in charge of generating the dataset-info.json file: [#3507 (comment)](https://github.com/huggingface/datasets/issues/3507#issuecomment-1033752157)
Issue: Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`

Comment:
Thanks for bringing these points up !

> @lhoestq: Tensorflow Datasets catalog includes a community catalog where you can find and use HF datasets. They are using the exported dataset_infos.json files from github to get the metadata: https://github.com/huggingface/datasets/issues/3507#issuecomment-1056997627

The TFDS implementation is not super advanced, so it's ok IMO as long as we don't break all the dataset scripts. Note that users can still use `to_tf_dataset`.

We had a chance to discuss the two nexts points with @julien-c as well:

> @severo: https://github.com/huggingface/datasets/issues/3507#issuecomment-1042779776
the metadata header might be very long, before reaching the start of the README/dataset card.

If we don't add the checksums we should be fine. We can also set a maximum number of supported configs in the README to keep it readable.

> @severo: the future "datasets server" could be in charge of generating the dataset-info.json file: https://github.com/huggingface/datasets/issues/3507#issuecomment-1033752157

I guess the "HF Hub actions" could open PRs to do the same in the YAML directly
Issue: `_resolve_features` ignores the token

Comment:
Hi ! Your HF_ENDPOINT seems wrong because of the extra "/"
```diff
- os.environ["HF_ENDPOINT"] = "https://hub-ci.huggingface.co/"
+ os.environ["HF_ENDPOINT"] = "https://hub-ci.huggingface.co"
```

can you try again without the extra "/" ?
Issue: `_resolve_features` ignores the token

Comment:
Oh, yes, sorry, but it's not the issue.

In my code, I set `HF_ENDPOINT=https://hub-ci.huggingface.co`. I added `os.environ["HF_ENDPOINT"] = "https://hub-ci.huggingface.co/"` afterward just to indicate that we had to have this env var and made a mistake there
Issue: `_resolve_features` ignores the token

Comment:
I can't reproduce on my side. I tried using a private dataset repo with a CSV file on hub-ci

What's your version of `huggingface_hub` ?
Issue: `_resolve_features` ignores the token

Comment:
I can't reproduce either... Not sure what has occurred, very sorry to have made you lost your time on that
Issue: `_resolve_features` ignores the token

Comment:
I got something similar in https://github.com/huggingface/datasets-server/pull/608. Look how changing the order of the tests (https://github.com/huggingface/datasets-server/pull/608/commits/2c50fe833323de3dfdc76c5cd68639279e0887f8) change the result, which means that something has a side-effect:
- https://github.com/huggingface/datasets-server/actions/runs/3264636253/jobs/5365612918 works
- https://github.com/huggingface/datasets-server/actions/runs/3264651839/jobs/5365654924 does not work

I still couldn't reproduce it with a simpler script...
Issue: `_resolve_features` ignores the token

Comment:
The issue happens because `extend_module_for_streaming` can't be used several times on packaged builders like `csv` to apply a new auth token. Indeed `extend_module_for_streaming` only applies authentication once, and on subsequent calls does nothing:

https://github.com/huggingface/datasets/blob/07b7c38d9e9c72c74b02524c432ca64d0d3738f4/src/datasets/streaming.py#L62-L64

This behavior exists because the authenticatoin wrapper only supports one token. This is an issue for packaged builders which can be used to load several datasets, so it may require several tokens.

This can be fixed by storing a dict `token_per_repo_id` instead of `use_auth_token` in the authentication wrapper, and by making it possible to update the authentication wrapper with a new token
Issue: `_resolve_features` ignores the token

Comment:
> This can be fixed by storing a dict token_per_repo_id instead of use_auth_token in the authentication wrapper, and by making it possible to update the authentication wrapper with a new token

If I call the module on the same repo twice: first with authentication, then without authentication, would the second call use authentication anyway? It sounds like a bug: the argument passed to the function would be silently ignored.