Repository: nltk/nltk
Pull request: Support both iso639-3 codes and BCP-47 language tags
Fix #3058 by adding a function to obtain language names, using their respective iso639-3 language codes.

This PR adds a langname.py module in the nltk directory. I plan to use it with wordnet.py, to obtain the names of the multilingual wordnets from OMW.

```
>>>from nltk.langnames import langname
>>>for code in ["cat", "wln"]:
>>>    print(f"{code}: {langname(code)}")
```

> cat: Catalan
> wln: Walloon
> 

The language codes were obtained from https://iso639-3.sil.org/code_tables/download_tables
and since it is not clear whether this use is authorized,  it seems safer to ask them:

> For any questions about whether a particular use is covered by these guidelines, contact the Registration Authority at ISO639-3@sil.org.

Problem statement:
Translating between language names and iso-639 codes
Some NLTK libraries (tag, crubadan, wordnet) use iso-639 language codes from the International Organization for Standardization (ISO). Iso-639 represents language names using either a 2-letter or 3-letter code that is not always very obvious: for ex. it may be difficult to guess that  _el_ or _ell_ are the codes for "Modern Greek".

In particular, the OMW data uses iso-639-3, though NLTK does not provide any clue about which languages the codes represent.

Wouldn't it be relevant to have functions available, to translate between language names and the corresponding language codes?

Hints from issue discussion:
_nltk_data/corpora/crubadan/table.txt_ seems to provide the necessary data in 3 columns, and _crubadan.py_ includes functions to translate between columns 1 and 2 from that file. To solve the present issue, it is easy to implement a translation between columns 2 and 3.
However the iso-639 codes in crubadan's _table.txt_ don't always seem very accurate, since it lists 'fra' as the code for Walloon, which should be 'wln'.

It would be better to use an authoritative source. For ISO 639-3, the registration authority is sil.org, and they make the relevant data package freely available at
https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3_Code_Tables_20220311.zip
However, they don't allow redistribution of this data.
An alternative could be to extract a table from
https://en.wikipedia.org/w/index.php?title=Wikipedia:WikiProject_Languages/List_of_ISO_639-3_language_codes_(2019)

This yields 7859 unique iso639-3 codes with their corresponding (multiple) language names.

NB https://iso639-3.sil.org/

@stevenbird, here is the iso639-3.sil.org policy. The last clause clearly means that they don't allow distributing their tables verbatim as a corpus in nltk_data. But the preamble indicates that NLTK might be allowed to incorporate the language codes in a .py file, for ex. as a Python dictionary, though the safest would be to ask them.

```
ISO 639-3 Code Tables Terms of Use

   The ISO 639-3 code set may be downloaded and incorporated into software
   products, web-based systems, digital devices, etc., either commercial
   or non-commercial, provided that:
     * attribution is given www.iso639-3.sil.org as the source of the
       codes;
     * the identifiers of the code set are not modified or extended except
       as may be privately agreed using the Private Use Area (range qaa to
       qtz), and then such extensions shall not be distributed publicly;
     * the product, system, or device does not provide a means to
       redistribute the code set.

```
Also see https://pypi.org/project/langcodes/ which does just about anything you need with language codes (BCP-47), although one notable omission is the ability to map to ISO639-3 (it can map _from_ ISO639-3, and can map to ISO639-2, however).
Thanks @goodmami. Ideally, NLTK would support both iso639-3 and bcp-47. A great advantage with bcp-47 is that the data can be freely redistributed (unlike iso639-3), so it would be possible to have a package in nltk_data, and process it with a CorpusReader. 
But bcp-47 is more difficult to deal with, because it sometimes includes several language  names for the same language code, and the data records don't indicate which one is the preferred name, while iso639-3 offers a neat 1:1 bijective correspondence between codes and names. In particular, it is problematic that bcp-47 overloads the 'ca' code with both "Catalan" and "Valencian", although they are different (though related) languages. 
So, for the moment, iso639-3 appears more linguistically satisfying.
> it is problematic that bcp-47 overloads the 'ca' code with both "Catalan" and "Valencian"

That is not quite right. Whereas both ISO639-1's `ca` and ISO639-3's [`cat`](https://iso639-3.sil.org/code/cat) conflate Catalan and Valencian, BCP-47 (which encompasses both code systems and more with a syntax for assembling tags at various levels of specificity) is able to distinguish them with a subtag: `ca-valencia`.

BCP-47 can be as specific as you want it to be (`ca-Latn-ES-valencia` would be equivalent but redundant), but this open-ended nature can make it less useful when you want a simple and consistent mapping. I'm not advocating for either system. The descriptive power of BCP-47 is offset by its complexity.
Thanks @goodmami, I have often seen those tags that you refer to, where a "territory" is appended to a language code, but I was not aware that they are an inherent part of BCP-47. My first impression was based on the much simpler official BCP-47 subtag registry file at
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry

It includes records like:

> %%
> Type: language
> Subtag: ca
> Description: Catalan
> Description: Valencian
> Added: 2005-10-16
> Suppress-Script: Latn
> %%
 
The tag extension mechanism is more complex, but since it is a part of the BCP-47 specification, and very widely used, it could also be nice to support. 
An alternative could be to start by supporting the most used bcp47 tag combinations, as in this project:

https://github.com/highfestiva/bcp47.py

Commit message: Use !r for repr formatting in warnings

Patch to review:
diff --git a/nltk/corpus/__init__.py b/nltk/corpus/__init__.py
index 5b0c666a90..c7d1f7b727 100644
--- a/nltk/corpus/__init__.py
+++ b/nltk/corpus/__init__.py
@@ -72,6 +72,7 @@
     encoding=[("science", "latin_1"), ("rural", "utf8")],
 )
 alpino = LazyCorpusLoader("alpino", AlpinoCorpusReader, tagset="alpino")
+bcp47 = LazyCorpusLoader("bcp47", Bcp47CorpusReader, r"(cldr|iana)/*")
 brown = LazyCorpusLoader(
     "brown",
     CategorizedTaggedCorpusReader,
diff --git a/nltk/corpus/reader/__init__.py b/nltk/corpus/reader/__init__.py
index 22310c4621..8f1d501e1d 100644
--- a/nltk/corpus/reader/__init__.py
+++ b/nltk/corpus/reader/__init__.py
@@ -105,6 +105,7 @@
 from nltk.corpus.reader.comparative_sents import *
 from nltk.corpus.reader.panlex_lite import *
 from nltk.corpus.reader.panlex_swadesh import *
+from nltk.corpus.reader.bcp47 import *
 
 # Make sure that nltk.corpus.reader.bracket_parse gives the module, not
 # the function bracket_parse() defined in nltk.tree:
@@ -181,4 +182,5 @@
     "UnicharsCorpusReader",
     "MWAPPDBCorpusReader",
     "PanlexSwadeshCorpusReader",
+    "Bcp47CorpusReader",
 ]
diff --git a/nltk/corpus/reader/bcp47.py b/nltk/corpus/reader/bcp47.py
new file mode 100644
index 0000000000..fa37214bb7
--- /dev/null
+++ b/nltk/corpus/reader/bcp47.py
@@ -0,0 +1,216 @@
+# Natural Language Toolkit: BCP-47 language tags
+#
+# Copyright (C) 2022 NLTK Project
+# Author: Eric Kafe <kafe.eric@gmail.com>
+# URL: <https://www.nltk.org/>
+# For license information, see LICENSE.TXT
+
+import re
+from warnings import warn
+from xml.etree import ElementTree as et
+
+from nltk.corpus.reader import CorpusReader
+
+
+class Bcp47CorpusReader(CorpusReader):
+    """
+    Parse BCP-47 composite language tags
+
+    Supports all the main subtags, and the 'u-sd' extension:
+
+    >>> from nltk.corpus import bcp47
+    >>> bcp47.name('oc-gascon-u-sd-fr64')
+    'Occitan (post 1500): Gascon: Pyrénées-Atlantiques'
+
+    Can load a conversion table to Wikidata Q-codes:
+    >>> bcp47.load_wiki_q()
+    >>> bcp47.wiki_q['en-GI-spanglis']
+    'Q79388'
+
+    """
+
+    def __init__(self, root, fileids):
+        """Read the BCP-47 database"""
+        super().__init__(root, fileids)
+        self.langcode = {}
+        with self.open("iana/language-subtag-registry.txt") as fp:
+            self.db = self.data_dict(fp.read().split("%%\n"))
+        with self.open("cldr/common-subdivisions-en.xml") as fp:
+            self.subdiv = self.subdiv_dict(
+                et.parse(fp).iterfind("localeDisplayNames/subdivisions/subdivision")
+            )
+        self.morphology()
+
+    def load_wiki_q(self):
+        """Load conversion table to Wikidata Q-codes (only if needed)"""
+        with self.open("cldr/tools-cldr-rdf-external-entityToCode.tsv") as fp:
+            self.wiki_q = self.wiki_dict(fp.read().strip().split("\n")[1:])
+
+    def wiki_dict(self, lines):
+        """Convert Wikidata list of Q-codes to a BCP-47 dictionary"""
+        return {
+            pair[1]: pair[0].split("/")[-1]
+            for pair in [line.strip().split("\t") for line in lines]
+        }
+
+    def subdiv_dict(self, subdivs):
+        """Convert the CLDR subdivisions list to a dictionary"""
+        return {sub.attrib["type"]: sub.text for sub in subdivs}
+
+    def morphology(self):
+        self.casing = {
+            "language": str.lower,
+            "extlang": str.lower,
+            "script": str.title,
+            "region": str.upper,
+            "variant": str.lower,
+        }
+        dig = "[0-9]"
+        low = "[a-z]"
+        up = "[A-Z]"
+        alnum = "[a-zA-Z0-9]"
+        self.format = {
+            "language": re.compile(f"{low*3}?"),
+            "extlang": re.compile(f"{low*3}"),
+            "script": re.compile(f"{up}{low*3}"),
+            "region": re.compile(f"({up*2})|({dig*3})"),
+            "variant": re.compile(f"{alnum*4}{(alnum+'?')*4}"),
+            "singleton": re.compile(f"{low}"),
+        }
+
+    def data_dict(self, records):
+        """Convert the BCP-47 language subtag registry to a dictionary"""
+        self.version = records[0].replace("File-Date:", "").strip()
+        dic = {}
+        dic["deprecated"] = {}
+        for label in [
+            "language",
+            "extlang",
+            "script",
+            "region",
+            "variant",
+            "redundant",
+            "grandfathered",
+        ]:
+            dic["deprecated"][label] = {}
+        for record in records[1:]:
+            fields = [field.split(": ") for field in record.strip().split("\n")]
+            typ = fields[0][1]
+            tag = fields[1][1]
+            if typ not in dic:
+                dic[typ] = {}
+            subfields = {}
+            for field in fields[2:]:
+                if len(field) == 2:
+                    [key, val] = field
+                    if key not in subfields:
+                        subfields[key] = [val]
+                    else:  # multiple value
+                        subfields[key].append(val)
+                else:  # multiline field
+                    subfields[key][-1] += " " + field[0].strip()
+                if (
+                    "Deprecated" not in record
+                    and typ == "language"
+                    and key == "Description"
+                ):
+                    self.langcode[subfields[key][-1]] = tag
+            for key in subfields:
+                if len(subfields[key]) == 1:  # single value
+                    subfields[key] = subfields[key][0]
+            if "Deprecated" in record:
+                dic["deprecated"][typ][tag] = subfields
+            else:
+                dic[typ][tag] = subfields
+        return dic
+
+    def val2str(self, val):
+        """Return only first value"""
+        if type(val) == list:
+            #            val = "/".join(val) # Concatenate all values
+            val = val[0]
+        return val
+
+    def lang2str(self, lg_record):
+        """Concatenate subtag values"""
+        name = f"{lg_record['language']}"
+        for label in ["extlang", "script", "region", "variant", "extension"]:
+            if label in lg_record:
+                name += f": {lg_record[label]}"
+        return name
+
+    def parse_tag(self, tag):
+        """Convert a BCP-47 tag to a dictionary of labelled subtags"""
+        subtags = tag.split("-")
+        lang = {}
+        labels = ["language", "extlang", "script", "region", "variant", "variant"]
+        while subtags and labels:
+            subtag = subtags.pop(0)
+            found = False
+            while labels:
+                label = labels.pop(0)
+                subtag = self.casing[label](subtag)
+                if self.format[label].fullmatch(subtag):
+                    if subtag in self.db[label]:
+                        found = True
+                        lang[label] = self.val2str(
+                            self.db[label][subtag]["Description"]
+                        )
+                        break
+                    elif subtag in self.db["deprecated"][label]:
+                        found = True
+                        note = f"The {subtag!r} {label} code is deprecated"
+                        if "Preferred-Value" in self.db["deprecated"][label][subtag]:
+                            prefer = self.db["deprecated"][label][subtag][
+                                "Preferred-Value"
+                            ]
+                            note += f"', prefer '{self.val2str(prefer)}'"
+                        lang[label] = self.val2str(
+                            self.db["deprecated"][label][subtag]["Description"]
+                        )
+                        warn(note)
+                        break
+            if not found:
+                if subtag == "u" and subtags[0] == "sd":  # CLDR regional subdivisions
+                    sd = subtags[1]
+                    if sd in self.subdiv:
+                        ext = self.subdiv[sd]
+                    else:
+                        ext = f"<Unknown subdivision: {ext}>"
+                else:  # other extension subtags are not supported yet
+                    ext = f"{subtag}{''.join(['-'+ext for ext in subtags])}".lower()
+                    if not self.format["singleton"].fullmatch(subtag):
+                        ext = f"<Invalid extension: {ext}>"
+                        warn(ext)
+                lang["extension"] = ext
+                subtags = []
+        return lang
+
+    def name(self, tag):
+        """
+        Convert a BCP-47 tag to a colon-separated string of subtag names
+
+        >>> from nltk.corpus import bcp47
+        >>> bcp47.name('ca-Latn-ES-valencia')
+        'Catalan: Latin: Spain: Valencian'
+
+        """
+        for label in ["redundant", "grandfathered"]:
+            val = None
+            if tag in self.db[label]:
+                val = f"{self.db[label][tag]['Description']}"
+                note = f"The {tag!r} code is {label}"
+            elif tag in self.db["deprecated"][label]:
+                val = f"{self.db['deprecated'][label][tag]['Description']}"
+                note = f"The {tag!r} code is {label} and deprecated"
+                if "Preferred-Value" in self.db["deprecated"][label][tag]:
+                    prefer = self.db["deprecated"][label][tag]["Preferred-Value"]
+                    note += f", prefer {self.val2str(prefer)!r}"
+            if val:
+                warn(note)
+                return val
+        try:
+            return self.lang2str(self.parse_tag(tag))
+        except:
+            warn(f"Tag {tag!r} was not recognized")
+            return None
diff --git a/nltk/langnames.py b/nltk/langnames.py
new file mode 100644
index 0000000000..278bb03e12
--- /dev/null
+++ b/nltk/langnames.py
@@ -0,0 +1,730 @@
+# Natural Language Toolkit: Language Codes
+#
+# Copyright (C) 2022 NLTK Project
+# Author: Eric Kafe <kafe.eric@gmail.com>
+# URL: <https://www.nltk.org/>
+# For license information, see LICENSE.TXT
+#
+# iso639-3 language codes (C) https://iso639-3.sil.org/
+
+"""
+Translate between language names and language codes.
+
+The iso639-3 language codes were downloaded from the registration authority at
+https://iso639-3.sil.org/
+
+The iso639-3 codeset is evolving, so retired language codes are kept in the
+"iso639retired" dictionary, which is used as fallback by the wrapper functions
+"langname" and "langcode", in order to support the lookup of retired codes.
+
+The "langcode" function returns the current iso639-3 code if there is one,
+and falls back to the retired code otherwise. As specified by BCP-47,
+it returns the shortest (2-letter) code by default, but 3-letter codes
+are also available:
+
+    >>> import nltk.langnames as lgn
+    >>> lgn.langname('fri')          #'fri' is a retired code
+    'Western Frisian'
+
+    The current code is different from the retired one:
+    >>> lgn.langcode('Western Frisian')
+    'fy'
+
+    >>> lgn.langcode('Western Frisian', typ = 3)
+    'fry'
+
+"""
+
+import re
+from warnings import warn
+
+from nltk.corpus import bcp47
+
+codepattern = re.compile("[a-z][a-z][a-z]?")
+
+
+def langname(tag, typ="full"):
+    """
+    Convert a composite BCP-47 tag to a language name
+
+    >>> from nltk.langnames import langname
+    >>> langname('ca-Latn-ES-valencia')
+    'Catalan: Latin: Spain: Valencian'
+
+    >>> langname('ca-Latn-ES-valencia', typ="short")
+    'Catalan'
+    """
+    tags = tag.split("-")
+    code = tags[0].lower()
+    if codepattern.fullmatch(code):
+        if code in iso639retired:  # retired codes
+            return iso639retired[code]
+        elif code in iso639short:  # 3-letter codes
+            code2 = iso639short[code]  # convert to 2-letter code
+            warn(f"Shortening {code!r} to {code2!r}", stacklevel=2)
+            tag = "-".join([code2] + tags[1:])
+        name = bcp47.name(tag)  # parse according to BCP-47
+        if typ == "full":
+            return name  # include all subtags
+        elif name:
+            return name.split(":")[0]  # only the language subtag
+    else:
+        warn(f"Could not find code in {code!r}", stacklevel=2)
+
+
+def langcode(name, typ=2):
+    """
+    Convert language name to iso639-3 language code. Returns the short 2-letter
+    code by default, if one is available, and the 3-letter code otherwise:
+
+    >>> from nltk.langnames import langcode
+    >>> langcode('Modern Greek (1453-)')
+    'el'
+
+    Specify 'typ=3' to get the 3-letter code:
+
+    >>> langcode('Modern Greek (1453-)', typ=3)
+    'ell'
+    """
+    if name in bcp47.langcode:
+        code = bcp47.langcode[name]
+        if typ == 3 and code in iso639long:
+            code = iso639long[code]  # convert to 3-letter code
+        return code
+    elif name in iso639code_retired:
+        return iso639code_retired[name]
+    else:
+        warn(f"Could not find language in {name!r}", stacklevel=2)
+
+
+# =======================================================================
+# Translate betwwen Wikidata Q-codes and BCP-47 codes or names
+# .......................................................................
+
+
+def tag2q(tag):
+    """
+    Convert BCP-47 tag to Wikidata Q-code
+
+    >>> tag2q('nds-u-sd-demv')
+    'Q4289225'
+    """
+    return bcp47.wiki_q[tag]
+
+
+def q2tag(qcode):
+    """
+    Convert Wikidata Q-code to BCP-47 tag
+
+    >>> q2tag('Q4289225')
+    'nds-u-sd-demv'
+    """
+    return wiki_bcp47[qcode]
+
+
+def q2name(qcode, typ="full"):
+    """
+    Convert Wikidata Q-code to BCP-47 (full or short) language name
+
+    >>> q2name('Q4289225')
+    'Low German: Mecklenburg-Vorpommern'
+
+    >>> q2name('Q4289225', "short")
+    'Low German'
+    """
+    return langname(q2tag(qcode), typ)
+
+
+def lang2q(name):
+    """
+    Convert simple language name to Wikidata Q-code
+
+    >>> lang2q('Low German')
+    'Q25433'
+    """
+    return tag2q(langcode(name))
+
+
+# ======================================================================
+# Data dictionaries
+# ......................................................................
+
+
+def inverse_dict(dic):
+    """Return inverse mapping, but only if it is bijective"""
+    if len(dic.keys()) == len(set(dic.values())):
+        return {val: key for (key, val) in dic.items()}
+    else:
+        warn("This dictionary has no bijective inverse mapping.")
+
+
+bcp47.load_wiki_q()  # Wikidata conversion table needs to be loaded explicitly
+wiki_bcp47 = inverse_dict(bcp47.wiki_q)
+
+iso639short = {
+    "aar": "aa",
+    "abk": "ab",
+    "afr": "af",
+    "aka": "ak",
+    "amh": "am",
+    "ara": "ar",
+    "arg": "an",
+    "asm": "as",
+    "ava": "av",
+    "ave": "ae",
+    "aym": "ay",
+    "aze": "az",
+    "bak": "ba",
+    "bam": "bm",
+    "bel": "be",
+    "ben": "bn",
+    "bis": "bi",
+    "bod": "bo",
+    "bos": "bs",
+    "bre": "br",
+    "bul": "bg",
+    "cat": "ca",
+    "ces": "cs",
+    "cha": "ch",
+    "che": "ce",
+    "chu": "cu",
+    "chv": "cv",
+    "cor": "kw",
+    "cos": "co",
+    "cre": "cr",
+    "cym": "cy",
+    "dan": "da",
+    "deu": "de",
+    "div": "dv",
+    "dzo": "dz",
+    "ell": "el",
+    "eng": "en",
+    "epo": "eo",
+    "est": "et",
+    "eus": "eu",
+    "ewe": "ee",
+    "fao": "fo",
+    "fas": "fa",
+    "fij": "fj",
+    "fin": "fi",
+    "fra": "fr",
+    "fry": "fy",
+    "ful": "ff",
+    "gla": "gd",
+    "gle": "ga",
+    "glg": "gl",
+    "glv": "gv",
+    "grn": "gn",
+    "guj": "gu",
+    "hat": "ht",
+    "hau": "ha",
+    "hbs": "sh",
+    "heb": "he",
+    "her": "hz",
+    "hin": "hi",
+    "hmo": "ho",
+    "hrv": "hr",
+    "hun": "hu",
+    "hye": "hy",
+    "ibo": "ig",
+    "ido": "io",
+    "iii": "ii",
+    "iku": "iu",
+    "ile": "ie",
+    "ina": "ia",
+    "ind": "id",
+    "ipk": "ik",
+    "isl": "is",
+    "ita": "it",
+    "jav": "jv",
+    "jpn": "ja",
+    "kal": "kl",
+    "kan": "kn",
+    "kas": "ks",
+    "kat": "ka",
+    "kau": "kr",
+    "kaz": "kk",
+    "khm": "km",
+    "kik": "ki",
+    "kin": "rw",
+    "kir": "ky",
+    "kom": "kv",
+    "kon": "kg",
+    "kor": "ko",
+    "kua": "kj",
+    "kur": "ku",
+    "lao": "lo",
+    "lat": "la",
+    "lav": "lv",
+    "lim": "li",
+    "lin": "ln",
+    "lit": "lt",
+    "ltz": "lb",
+    "lub": "lu",
+    "lug": "lg",
+    "mah": "mh",
+    "mal": "ml",
+    "mar": "mr",
+    "mkd": "mk",
+    "mlg": "mg",
+    "mlt": "mt",
+    "mon": "mn",
+    "mri": "mi",
+    "msa": "ms",
+    "mya": "my",
+    "nau": "na",
+    "nav": "nv",
+    "nbl": "nr",
+    "nde": "nd",
+    "ndo": "ng",
+    "nep": "ne",
+    "nld": "nl",
+    "nno": "nn",
+    "nob": "nb",
+    "nor": "no",
+    "nya": "ny",
+    "oci": "oc",
+    "oji": "oj",
+    "ori": "or",
+    "orm": "om",
+    "oss": "os",
+    "pan": "pa",
+    "pli": "pi",
+    "pol": "pl",
+    "por": "pt",
+    "pus": "ps",
+    "que": "qu",
+    "roh": "rm",
+    "ron": "ro",
+    "run": "rn",
+    "rus": "ru",
+    "sag": "sg",
+    "san": "sa",
+    "sin": "si",
+    "slk": "sk",
+    "slv": "sl",
+    "sme": "se",
+    "smo": "sm",
+    "sna": "sn",
+    "snd": "sd",
+    "som": "so",
+    "sot": "st",
+    "spa": "es",
+    "sqi": "sq",
+    "srd": "sc",
+    "srp": "sr",
+    "ssw": "ss",
+    "sun": "su",
+    "swa": "sw",
+    "swe": "sv",
+    "tah": "ty",
+    "tam": "ta",
+    "tat": "tt",
+    "tel": "te",
+    "tgk": "tg",
+    "tgl": "tl",
+    "tha": "th",
+    "tir": "ti",
+    "ton": "to",
+    "tsn": "tn",
+    "tso": "ts",
+    "tuk": "tk",
+    "tur": "tr",
+    "twi": "tw",
+    "uig": "ug",
+    "ukr": "uk",
+    "urd": "ur",
+    "uzb": "uz",
+    "ven": "ve",
+    "vie": "vi",
+    "vol": "vo",
+    "wln": "wa",
+    "wol": "wo",
+    "xho": "xh",
+    "yid": "yi",
+    "yor": "yo",
+    "zha": "za",
+    "zho": "zh",
+    "zul": "zu",
+}
+
+
+iso639retired = {
+    "fri": "Western Frisian",
+    "auv": "Auvergnat",
+    "gsc": "Gascon",
+    "lms": "Limousin",
+    "lnc": "Languedocien",
+    "prv": "Provençal",
+    "amd": "Amapá Creole",
+    "bgh": "Bogan",
+    "bnh": "Banawá",
+    "bvs": "Belgian Sign Language",
+    "ccy": "Southern Zhuang",
+    "cit": "Chittagonian",
+    "flm": "Falam Chin",
+    "jap": "Jaruára",
+    "kob": "Kohoroxitari",
+    "mob": "Moinba",
+    "mzf": "Aiku",
+    "nhj": "Tlalitzlipa Nahuatl",
+    "nhs": "Southeastern Puebla Nahuatl",
+    "occ": "Occidental",
+    "tmx": "Tomyang",
+    "tot": "Patla-Chicontla Totonac",
+    "xmi": "Miarrã",
+    "yib": "Yinglish",
+    "ztc": "Lachirioag Zapotec",
+    "atf": "Atuence",
+    "bqe": "Navarro-Labourdin Basque",
+    "bsz": "Souletin Basque",
+    "aex": "Amerax",
+    "ahe": "Ahe",
+    "aiz": "Aari",
+    "akn": "Amikoana",
+    "arf": "Arafundi",
+    "azr": "Adzera",
+    "bcx": "Pamona",
+    "bii": "Bisu",
+    "bke": "Bengkulu",
+    "blu": "Hmong Njua",
+    "boc": "Bakung Kenyah",
+    "bsd": "Sarawak Bisaya",
+    "bwv": "Bahau River Kenyah",
+    "bxt": "Buxinhua",
+    "byu": "Buyang",
+    "ccx": "Northern Zhuang",
+    "cru": "Carútana",
+    "dat": "Darang Deng",
+    "dyk": "Land Dayak",
+    "eni": "Enim",
+    "fiz": "Izere",
+    "gen": "Geman Deng",
+    "ggh": "Garreh-Ajuran",
+    "itu": "Itutang",
+    "kds": "Lahu Shi",
+    "knh": "Kayan River Kenyah",
+    "krg": "North Korowai",
+    "krq": "Krui",
+    "kxg": "Katingan",
+    "lmt": "Lematang",
+    "lnt": "Lintang",
+    "lod": "Berawan",
+    "mbg": "Northern Nambikuára",
+    "mdo": "Southwest Gbaya",
+    "mhv": "Arakanese",
+    "miv": "Mimi",
+    "mqd": "Madang",
+    "nky": "Khiamniungan Naga",
+    "nxj": "Nyadu",
+    "ogn": "Ogan",
+    "ork": "Orokaiva",
+    "paj": "Ipeka-Tapuia",
+    "pec": "Southern Pesisir",
+    "pen": "Penesak",
+    "plm": "Palembang",
+    "poj": "Lower Pokomo",
+    "pun": "Pubian",
+    "rae": "Ranau",
+    "rjb": "Rajbanshi",
+    "rws": "Rawas",
+    "sdd": "Semendo",
+    "sdi": "Sindang Kelingi",
+    "skl": "Selako",
+    "slb": "Kahumamahon Saluan",
+    "srj": "Serawai",
+    "suf": "Tarpia",
+    "suh": "Suba",
+    "suu": "Sungkai",
+    "szk": "Sizaki",
+    "tle": "Southern Marakwet",
+    "tnj": "Tanjong",
+    "ttx": "Tutong 1",
+    "ubm": "Upper Baram Kenyah",
+    "vky": "Kayu Agung",
+    "vmo": "Muko-Muko",
+    "wre": "Ware",
+    "xah": "Kahayan",
+    "xkm": "Mahakam Kenyah",
+    "xuf": "Kunfal",
+    "yio": "Dayao Yi",
+    "ymj": "Muji Yi",
+    "ypl": "Pula Yi",
+    "ypw": "Puwa Yi",
+    "ywm": "Wumeng Yi",
+    "yym": "Yuanjiang-Mojiang Yi",
+    "mly": "Malay (individual language)",
+    "muw": "Mundari",
+    "xst": "Silt'e",
+    "ope": "Old Persian",
+    "scc": "Serbian",
+    "scr": "Croatian",
+    "xsk": "Sakan",
+    "mol": "Moldavian",
+    "aay": "Aariya",
+    "acc": "Cubulco Achí",
+    "cbm": "Yepocapa Southwestern Cakchiquel",
+    "chs": "Chumash",
+    "ckc": "Northern Cakchiquel",
+    "ckd": "South Central Cakchiquel",
+    "cke": "Eastern Cakchiquel",
+    "ckf": "Southern Cakchiquel",
+    "cki": "Santa María De Jesús Cakchiquel",
+    "ckj": "Santo Domingo Xenacoj Cakchiquel",
+    "ckk": "Acatenango Southwestern Cakchiquel",
+    "ckw": "Western Cakchiquel",
+    "cnm": "Ixtatán Chuj",
+    "cti": "Tila Chol",
+    "cun": "Cunén Quiché",
+    "eml": "Emiliano-Romagnolo",
+    "eur": "Europanto",
+    "gmo": "Gamo-Gofa-Dawro",
+    "hsf": "Southeastern Huastec",
+    "hva": "San Luís Potosí Huastec",
+    "ixi": "Nebaj Ixil",
+    "ixj": "Chajul Ixil",
+    "jai": "Western Jacalteco",
+    "mms": "Southern Mam",
+    "mpf": "Tajumulco Mam",
+    "mtz": "Tacanec",
+    "mvc": "Central Mam",
+    "mvj": "Todos Santos Cuchumatán Mam",
+    "poa": "Eastern Pokomam",
+    "pob": "Western Pokomchí",
+    "pou": "Southern Pokomam",
+    "ppv": "Papavô",
+    "quj": "Joyabaj Quiché",
+    "qut": "West Central Quiché",
+    "quu": "Eastern Quiché",
+    "qxi": "San Andrés Quiché",
+    "sic": "Malinguat",
+    "stc": "Santa Cruz",
+    "tlz": "Toala'",
+    "tzb": "Bachajón Tzeltal",
+    "tzc": "Chamula Tzotzil",
+    "tze": "Chenalhó Tzotzil",
+    "tzs": "San Andrés Larrainzar Tzotzil",
+    "tzt": "Western Tzutujil",
+    "tzu": "Huixtán Tzotzil",
+    "tzz": "Zinacantán Tzotzil",
+    "vlr": "Vatrata",
+    "yus": "Chan Santa Cruz Maya",
+    "nfg": "Nyeng",
+    "nfk": "Shakara",
+    "agp": "Paranan",
+    "bhk": "Albay Bicolano",
+    "bkb": "Finallig",
+    "btb": "Beti (Cameroon)",
+    "cjr": "Chorotega",
+    "cmk": "Chimakum",
+    "drh": "Darkhat",
+    "drw": "Darwazi",
+    "gav": "Gabutamon",
+    "mof": "Mohegan-Montauk-Narragansett",
+    "mst": "Cataelano Mandaya",
+    "myt": "Sangab Mandaya",
+    "rmr": "Caló",
+    "sgl": "Sanglechi-Ishkashimi",
+    "sul": "Surigaonon",
+    "sum": "Sumo-Mayangna",
+    "tnf": "Tangshewi",
+    "wgw": "Wagawaga",
+    "ayx": "Ayi (China)",
+    "bjq": "Southern Betsimisaraka Malagasy",
+    "dha": "Dhanwar (India)",
+    "dkl": "Kolum So Dogon",
+    "mja": "Mahei",
+    "nbf": "Naxi",
+    "noo": "Nootka",
+    "tie": "Tingal",
+    "tkk": "Takpa",
+    "baz": "Tunen",
+    "bjd": "Bandjigali",
+    "ccq": "Chaungtha",
+    "cka": "Khumi Awa Chin",
+    "dap": "Nisi (India)",
+    "dwl": "Walo Kumbe Dogon",
+    "elp": "Elpaputih",
+    "gbc": "Garawa",
+    "gio": "Gelao",
+    "hrr": "Horuru",
+    "ibi": "Ibilo",
+    "jar": "Jarawa (Nigeria)",
+    "kdv": "Kado",
+    "kgh": "Upper Tanudan Kalinga",
+    "kpp": "Paku Karen",
+    "kzh": "Kenuzi-Dongola",
+    "lcq": "Luhu",
+    "mgx": "Omati",
+    "nln": "Durango Nahuatl",
+    "pbz": "Palu",
+    "pgy": "Pongyong",
+    "sca": "Sansu",
+    "tlw": "South Wemale",
+    "unp": "Worora",
+    "wiw": "Wirangu",
+    "ybd": "Yangbye",
+    "yen": "Yendang",
+    "yma": "Yamphe",
+    "daf": "Dan",
+    "djl": "Djiwarli",
+    "ggr": "Aghu Tharnggalu",
+    "ilw": "Talur",
+    "izi": "Izi-Ezaa-Ikwo-Mgbo",
+    "meg": "Mea",
+    "mld": "Malakhel",
+    "mnt": "Maykulan",
+    "mwd": "Mudbura",
+    "myq": "Forest Maninka",
+    "nbx": "Ngura",
+    "nlr": "Ngarla",
+    "pcr": "Panang",
+    "ppr": "Piru",
+    "tgg": "Tangga",
+    "wit": "Wintu",
+    "xia": "Xiandao",
+    "yiy": "Yir Yoront",
+    "yos": "Yos",
+    "emo": "Emok",
+    "ggm": "Gugu Mini",
+    "leg": "Lengua",
+    "lmm": "Lamam",
+    "mhh": "Maskoy Pidgin",
+    "puz": "Purum Naga",
+    "sap": "Sanapaná",
+    "yuu": "Yugh",
+    "aam": "Aramanik",
+    "adp": "Adap",
+    "aue": "ǂKxʼauǁʼein",
+    "bmy": "Bemba (Democratic Republic of Congo)",
+    "bxx": "Borna (Democratic Republic of Congo)",
+    "byy": "Buya",
+    "dzd": "Daza",
+    "gfx": "Mangetti Dune ǃXung",
+    "gti": "Gbati-ri",
+    "ime": "Imeraguen",
+    "kbf": "Kakauhua",
+    "koj": "Sara Dunjo",
+    "kwq": "Kwak",
+    "kxe": "Kakihum",
+    "lii": "Lingkhim",
+    "mwj": "Maligo",
+    "nnx": "Ngong",
+    "oun": "ǃOǃung",
+    "pmu": "Mirpur Panjabi",
+    "sgo": "Songa",
+    "thx": "The",
+    "tsf": "Southwestern Tamang",
+    "uok": "Uokha",
+    "xsj": "Subi",
+    "yds": "Yiddish Sign Language",
+    "ymt": "Mator-Taygi-Karagas",
+    "ynh": "Yangho",
+    "bgm": "Baga Mboteni",
+    "btl": "Bhatola",
+    "cbe": "Chipiajes",
+    "cbh": "Cagua",
+    "coy": "Coyaima",
+    "cqu": "Chilean Quechua",
+    "cum": "Cumeral",
+    "duj": "Dhuwal",
+    "ggn": "Eastern Gurung",
+    "ggo": "Southern Gondi",
+    "guv": "Gey",
+    "iap": "Iapama",
+    "ill": "Iranun",
+    "kgc": "Kasseng",
+    "kox": "Coxima",
+    "ktr": "Kota Marudu Tinagas",
+    "kvs": "Kunggara",
+    "kzj": "Coastal Kadazan",
+    "kzt": "Tambunan Dusun",
+    "nad": "Nijadali",
+    "nts": "Natagaimas",
+    "ome": "Omejes",
+    "pmc": "Palumata",
+    "pod": "Ponares",
+    "ppa": "Pao",
+    "pry": "Pray 3",
+    "rna": "Runa",
+    "svr": "Savara",
+    "tdu": "Tempasuk Dusun",
+    "thc": "Tai Hang Tong",
+    "tid": "Tidong",
+    "tmp": "Tai Mène",
+    "tne": "Tinoc Kallahan",
+    "toe": "Tomedes",
+    "xba": "Kamba (Brazil)",
+    "xbx": "Kabixí",
+    "xip": "Xipináwa",
+    "xkh": "Karahawyana",
+    "yri": "Yarí",
+    "jeg": "Jeng",
+    "kgd": "Kataang",
+    "krm": "Krim",
+    "prb": "Lua'",
+    "puk": "Pu Ko",
+    "rie": "Rien",
+    "rsi": "Rennellese Sign Language",
+    "skk": "Sok",
+    "snh": "Shinabo",
+    "lsg": "Lyons Sign Language",
+    "mwx": "Mediak",
+    "mwy": "Mosiro",
+    "ncp": "Ndaktup",
+    "ais": "Nataoran Amis",
+    "asd": "Asas",
+    "dit": "Dirari",
+    "dud": "Hun-Saare",
+    "lba": "Lui",
+    "llo": "Khlor",
+    "myd": "Maramba",
+    "myi": "Mina (India)",
+    "nns": "Ningye",
+    "aoh": "Arma",
+    "ayy": "Tayabas Ayta",
+    "bbz": "Babalia Creole Arabic",
+    "bpb": "Barbacoas",
+    "cca": "Cauca",
+    "cdg": "Chamari",
+    "dgu": "Degaru",
+    "drr": "Dororo",
+    "ekc": "Eastern Karnic",
+    "gli": "Guliguli",
+    "kjf": "Khalaj",
+    "kxl": "Nepali Kurux",
+    "kxu": "Kui (India)",
+    "lmz": "Lumbee",
+    "nxu": "Narau",
+    "plp": "Palpa",
+    "sdm": "Semandang",
+    "tbb": "Tapeba",
+    "xrq": "Karranga",
+    "xtz": "Tasmanian",
+    "zir": "Ziriya",
+    "thw": "Thudam",
+    "bic": "Bikaru",
+    "bij": "Vaghat-Ya-Bijim-Legeri",
+    "blg": "Balau",
+    "gji": "Geji",
+    "mvm": "Muya",
+    "ngo": "Ngoni",
+    "pat": "Papitalai",
+    "vki": "Ija-Zuba",
+    "wra": "Warapu",
+    "ajt": "Judeo-Tunisian Arabic",
+    "cug": "Chungmboko",
+    "lak": "Laka (Nigeria)",
+    "lno": "Lango (South Sudan)",
+    "pii": "Pini",
+    "smd": "Sama",
+    "snb": "Sebuyau",
+    "uun": "Kulon-Pazeh",
+    "wrd": "Warduji",
+    "wya": "Wyandot",
+}
+
+
+iso639long = inverse_dict(iso639short)
+
+iso639code_retired = inverse_dict(iso639retired)


Code review comments:
diff_hunk: @@ -0,0 +1,216 @@
+# Natural Language Toolkit: BCP-47 language tags
+#
+# Copyright (C) 2022 NLTK Project
+# Author: Eric Kafe <kafe.eric@gmail.com>
+# URL: <https://www.nltk.org/>
+# For license information, see LICENSE.TXT
+
+import re
+from warnings import warn
+from xml.etree import ElementTree as et
+
+from nltk.corpus.reader import CorpusReader
+
+
+class Bcp47CorpusReader(CorpusReader):
path: nltk/corpus/reader/bcp47.py
text: @user1:
```suggestion
class BCP47CorpusReader(CorpusReader):
```
It seems that BCP47 is more fitting.

@author:
I agree, but since all other corpus reader classes include CorpusReader in their name (see the \_\_all\_\_ list in nltk/corpus/reader/\_\_init\_\_.py), I prefer to do the same for the moment.

@user1:
Apologies, I meant `BCP47CorpusReader`, i.e. capitalising Bcp to BCP.

@author:
Oh, ok, we can do that, but it will require changes in 3 files.

@user1:
If you'd like, I can handle that, and then merge it afterwards. If you think we're ready for merging afterwards 😄 

@author:
Thanks @user1, I just made the change.