Package: hyperglot
Hyperglot – a database and tools for detecting language support in fontsHyperglot helps type designers answer a seemingly simple question of language support in fonts: When can I use font A to set texts in language B?  It takes a pragmatic answer by identifying a standard character set for each orthography used by a language. The database that currently contains information for over 640+ languages is a work in progress, designed to grow.We record a basic and any auxiliary character sets for each orthography of a language. Note that only actively used orthographies (their status is set toprimary) are used when detecting language support in a font. Other, secondary or historical, orthographies are displayed just for information purposes.Where relevant, we also provide a brief design note containing tips about shaping and positioning requirements that go beyond Unicode character code points. Hyperglot should only be used to detect whether a font can be considered for use with a particular language. It does not say anything about the quality of a font’s design.Hyperglot is a work in progress provided AS IS, and the validity of its language data varies. To help you assess the validity of the results you view, each language in the database comes with a label indicating the quality of the data we have for it (e.g. some are considereddrafts, some have beenverified). We have checked the information against various online and offline sources and we are committed to continually improve it. However, we admit that mapping all the languages of the world in this way is beyond our capacity – we need help from users of each respective language! So, if you spot an issue or notice your favourite language is altogether missing from the database, get in touch. We will happilyincorporate your feedback and credit you.Read more about Hyperglot on the web app about pageThe comparison of Hyperglot and the Unicode CLDRHow to useThere are several ways how to use the database:Hyperglot web app athttp://hyperglot.rosettatype.comcommand-line tool (pip install hyperglot, see usage notes below)python packagage (pip install hyperglot)access the YAML file with the database directly (database README)It is complicated (disclaimer)A few notes to illustrate why the question of language support is complicated:a single language can be written using different orthographies in one or more scripts,languages are not isolated, there are loan words, names etc. from other languages, thus finding what is an essential character set for a language is largerly a question of convention,what one person considers a dialect, is a language for someone else,different kinds of texts require different vocabulary and hence different characters.It is important to note thatthere is more to language support in fonts than supporting a set of code points. A font needs to include glyphs with acceptable/readable shapes of the characters for a particular language. Sometimes there are regional or language variations for the same code point which means that different languages pose different requirements on the shape of a character, but identical requirements on the code point of the character. Moreover, glyphs have to interact as expected by the convention of a particular script/orthography. For example, some languages/scripts require (or strongly expect) certain glyph combinations to form ligatures or some glyph combinations require additional spacing correction (kerning) to prevent clashes or gaps. Thus, the report produced by the Hyperglot tools should only be used to detect whether a font can be considered for use with a particular language. It does not say anything about the quality of the design.Read more about this on the web app about pageDetecting supportCharacters are represented usingUnicodecode points in digital texts, e.g. the Latin-script letterahas a code pointU+0061. Digital OpenType fonts map these code points to glyphs, visual representations of characters. In order to find whether one can use a font for texts in a particular language, one needs to know which character code points are required for the language. This is what the Hyperglot database is for.A list of codepoints is obtained from a font.The database can be accessed in two modes:By default combinations of a base character with marks are required as single code point where this exists (e.g. encodedä), codepoints for base characters and combining mark characters (e.g.aand combining¨) from these combinations are not required unless the combination has no encoded form or the--marksflag is used.Using the--decomposedflag fonts are required to contain the base character and combining marks for a language (e.g. languages withäwill match for fonts that only haveaand combining¨but notäas encoded glyph).Specifiedvaliditylevel is used to filter out language entries with a lower (meaning, more uncertain) validity.If requested,baseandaux(auxiliary) lists of codepoints are combined to achieve more strict criteria by using the--supportoption. Themarksin the data are required based on the--decomposedand--marksflags. Marks that only appear inauxcharacters will not be required forbasevalidity.When detecting language support, code points foranyprimary orthography for a given language are considered. Orthographies withhistoricalandsecondarystatus are ignored. If multiple orthographies have thepreferred_as_groupvalue they are considered as one orthography even if including several scripts.When detecting orthography support, use--include-all-orthographies, all orthographies for a given language are checked individually. Orthographies withsecondarystatus are included. Orthographies withhistoricalstatus are ignored.If the list of code points in the font includes all code points from the list of codepoints from points 5 or 6, the font is considered to support this language/orthography. Additionally, joining behaviour and mark attachment is validated and a language/orthography is only considered supported if the font shapes these correctly. In listings the supported languages are grouped by scripts.The language-orthography combination means that a language that has multiple orthographies using different scripts (e.g., Serbian or Japanese) is listed under all of these scripts in the tools’ output.Important note: the web app currently does not include the shaping checks!Command-line toolA simple CLI tool is provided to output language support data for a passed in font file.InstallationYou will need to have Python 3 installed. Install via pip:pipinstallhyperglotUsagehyperglotpath/to/font.otfor to check several fonts at once, or their combined coverage (with-m union)hyperglotpath/to/font.otfpath/to/anotherfont.otf...Additional options:-s, --support: Specify what level of support to check against (currently options are "base" (default if omitted) or "aux")-m, --marks: Flag to signal a font should also include all combining marks used for a language - by default only those marks are required which are not part of preencoded characters (default is False)-d, --decomposed: Flag to signal a font should be considered supporting a language as long as it has all base glyphs and marks to write a language - by default also encoded precomposed glyphs are required (default is False)-a, --autonyms: Output the language names in their native language and script--speakers: Ouput how many speakers each language has (where available)--sort: Specify "speakers" to sort by speakers (default is "alphabetic")--sort-dir: Specify "desc" to sort in descending order (default is "asc" for ascending order)-o, --output: Supply a file path to write the output to, in yaml format. For a single input font this will be a subset of the Hyperglot database with the languages and orthographies that the font supports. If several fonts are provided the yaml file will have a top level dict key for each file. If the-moption is provided the yaml file will contain the specific intersection or union result-c, --comparison: How to process input if several files are provided (currently options are "individual", "union" and "intersection")--include-all-orthographies: Check all orthographies of a language, not just its primary one(s)--validity: Specify to filter by the level of validity of the language data (default is "preliminary")--include-historical: Option to include languages and orthographies marked as historical (default is False)--include-constructed: Option to include languages and orthographies that are marked as constructed (default is False)--strict-iso: Display language names and macrolanguage data strictly according to ISO (default is False)-v, --verbose: More logging information (default is False)-V, --version: Print the version hyperglot version number (default is False)Installing the pip package also installed thehyperglot-validateandhyperglot-savecommands, which allow checking and saving the yaml data in a structured and compatible way.Findingalmostsupported languagesHyperglot comes with ahyperglot-reportcommand that takes all the same options
the mainhyperglotcommand (see above). It additionally takes these options to
output reporting about what characters or shaping is missing in order to support
languages detected as not supported:--report-missing: Parameter to report unmatched languages which are missingnor less characters. Ifnis 0 all languages with any number of missing characters are listed (default).--report-marks: Parameter to report languages which are missingnor less mark attachment sequences. Ifnis 0 all languages with any number any number of missing mark attachment sequences are listed (default).--report-joining: Parameter to report languages which are missingnor less joining sequences. Ifnis 0 all languages with any number of missing joining sequences are listed (default).--report-all: Parameter to set/overwrite all other--report-xxxparameters.Database and contributingThe data structure is described in a separate file together with guidelines for contributing.Updates are comitted/merged to thedevbranch with themasterbranch holding the latest released version.Database and contributingRoadmapinclude records to track language and script specific punctuation and numeralscomparison to Unicode CLDRimprove references for language data (use APA everywhere)export in a way that would be useful to submit to Unicode CLDRbasic analysis of shaping instructions provided by the font (GPOS and GSUB)analyse shaping during language detection: check whetherbase + markcombinations are affected by the font instructionscheck whether joining behaviour (aka presentation forms, e.g. Arabic or Syriac) is supported in the fontcheck whether character combinations are affected by the font instructionsan effective and scalable way to prescribe more complex character/mark combinations, e.g. for Arabic or Hindi/DevanagariAuthors and contributorsThe Hyperglot database and tools were originally developed byRosetta, world typography specialists, publishers, and makers of original fonts addressing the needs of global typography. Our goal is to enable people to read better in their native languages.Main contributors, so far:David Březinadavid@rosettatype.com@MrBrezina (concept, database structure, and main language data set)Johannes Neumeierhello@johannesneumeier.com@kontur (tool and tests)Sérgio Martins @sergiolmartins (major expansion and review of the data, ca 250+ languages)Toshi Omagari  @toshe (basic language data for many Cyrillic-script languages)Denis Moyogo Jacquerye @moyogoThe full list of contributorsSimilar projects and inspirationsUnicode CLDRUnderware Latin PlusAlphabet Type’ Charset Checker(uses Unicode CLDR)Typekit SpeakeasyAdobe spreadsheets for Latin and CyrillicWebINK character setsfont-config languages definitionsContext of diacritics