Package: unihan-etl
unihan-etl ·AnETLtool for the Unicode Han Unification (UNIHAN) database releases. unihan-etl is designed to fetch (download), unpack (unzip), and convert the database from the Unicode website into either a flattened, tabular format or a structured, hierarchical format.unihan-etl serves dual purposes: as a Python library offering anAPIfor accessing data as Python objects, and as a command-line interface (CLI) for exporting data into CSV, JSON, or YAML formats.This tool is a component of thecihaisuite of CJK related projects. For a similar tool, seelibUnihan.As of v0.31.0, unihan-etl is compatible with UNIHAN Version 15.1.0 (released on 2023-09-01, revision 35).The UNIHAN databaseTheUNIHANdatabase organizes data across multiple files, exemplified below:U+3400	kCantonese		jau1
U+3400	kDefinition		(same as U+4E18 丘) hillock or mound
U+3400	kMandarin		qiū
U+3401	kCantonese		tim2
U+3401	kDefinition		to lick; to taste, a mat, bamboo bark
U+3401	kHanyuPinyin		10019.020:tiàn
U+3401	kMandarin		tiànValues vary in shape and structure depending on their field type.kHanyuPinyinmaps Unicode codepoints toHànyǔ Dà Zìdiǎn, where10019.020:tiànrepresents
an entry. Complicating it further, more variations:U+5EFE	kHanyuPinyin		10513.110,10514.010,10514.020:gǒng
U+5364	kHanyuPinyin		10093.130:xī,lǔ 74609.020:lǔ,xīkHanyuPinyinsupports multiple entries delimited by spaces. ":" (colon) separate locations in the
work from pinyin readings. "," (comma) separate multiple entries/readings. This is just one of 90
fields contained in the database.Tabular, "Flat" outputCSV (default)$unihan-etlchar,ucn,kCantonese,kDefinition,kHanyuPinyin,kMandarin
㐀,U+3400,jau1,(same as U+4E18 丘) hillock or mound,,qiū
㐁,U+3401,tim2,"to lick; to taste, a mat, bamboo bark",10019.020:tiàn,tiànWith$ unihan-etl -F yaml --no-expand:-char:㐀kCantonese:jau1kDefinition:(same as U+4E18 丘) hillock or moundkHanyuPinyin:nullkMandarin:qiūucn:U+3400-char:㐁kCantonese:tim2kDefinition:to lick; to taste, a mat, bamboo barkkHanyuPinyin:10019.020:tiànkMandarin:tiànucn:U+3401To preview in the CLI, trytabvieworcsvlens.JSON$unihan-etl-Fjson--no-expand[{"char":"㐀","ucn":"U+3400","kDefinition":"(same as U+4E18 丘) hillock or mound","kCantonese":"jau1","kHanyuPinyin":null,"kMandarin":"qiū"},{"char":"㐁","ucn":"U+3401","kDefinition":"to lick; to taste, a mat, bamboo bark","kCantonese":"tim2","kHanyuPinyin":"10019.020:tiàn","kMandarin":"tiàn"}]Tools:View in CLI:python-fx,jlessorfx.Filter via CLI:jq,jql,gojq.YAML$unihan-etl-Fyaml--no-expand-char:㐀kCantonese:jau1kDefinition:(same as U+4E18 丘) hillock or moundkHanyuPinyin:nullkMandarin:qiūucn:U+3400-char:㐁kCantonese:tim2kDefinition:to lick; to taste, a mat, bamboo barkkHanyuPinyin:10019.020:tiànkMandarin:tiànucn:U+3401Filter via the CLI withyq."Structured" outputCodepoints can pack a lot more detail, unihan-etl carefully extracts these values in a uniform
manner. Empty values are pruned.To make this possible, unihan-etl exports to JSON, YAML, and python list/dicts.Why not CSV?Unfortunately, CSV is only suitable for storing table-like information. File formats such as JSON
and YAML accept key-values and hierarchical entries.JSON$unihan-etl-Fjson[{"char":"㐀","ucn":"U+3400","kDefinition":["(same as U+4E18 丘) hillock or mound"],"kCantonese":["jau1"],"kMandarin":{"zh-Hans":"qiū","zh-Hant":"qiū"}},{"char":"㐁","ucn":"U+3401","kDefinition":["to lick","to taste, a mat, bamboo bark"],"kCantonese":["tim2"],"kHanyuPinyin":[{"locations":[{"volume":1,"page":19,"character":2,"virtual":0}],"readings":["tiàn"]}],"kMandarin":{"zh-Hans":"tiàn","zh-Hant":"tiàn"}}]YAML$unihan-etl-Fyaml-char:㐀kCantonese:-jau1kDefinition:-(same as U+4E18 丘) hillock or moundkMandarin:zh-Hans:qiūzh-Hant:qiūucn:U+3400-char:㐁kCantonese:-tim2kDefinition:-to lick-to taste, a mat, bamboo barkkHanyuPinyin:-locations:-character:2page:19virtual:0volume:1readings:-tiànkMandarin:zh-Hans:tiànzh-Hant:tiànucn:U+3401Featuresautomatically downloads UNIHAN from the internetstrives for accuracy with the specifications described inUNIHAN's database designexport to JSON, CSV and YAML (requirespyyaml) via-Fconfigurable to export specific fields via-faccounts for encoding conflicts due to the Unicode-heavy contentdesigned as a technical proof for future CJK (Chinese, Japanese, Korean) datasetscore component and dependency ofcihai, a CJK librarydata packagesupportexpansion of multi-value delimited fields in YAML, JSON and python dictionariessupports >= 3.7 and pypyIf you encounter a problem or have a question, pleasecreate an issue.InstallationTo download and build your own UNIHAN export:$pipinstall--userunihan-etlor bypipx:$pipxinstallunihan-etlDevelopmental releasespip:$pipinstall--user--upgrade--preunihan-etlpipx:$pipxinstall--suffix=@next'unihan-etl'--pip-args'\--pre'--force// Usage: unihan-etl@next load yoursessionUsageunihan-etloffers customizable builds via its command line arguments.Seeunihan-etl CLI argumentsfor information
on how you can specify columns, files, download URL's, and output destination.To output CSV, the default format:$unihan-etlTo output JSON:$unihan-etl-FjsonTo output YAML:$pipinstall--userpyyaml$unihan-etl-FyamlTo only output the kDefinition field in a csv:$unihan-etl-fkDefinitionTo output multiple fields, separate with spaces:$unihan-etl-fkCantonesekDefinitionTo output to a custom file:$unihan-etl--destination./exported.csvTo output to a custom file (templated file extension):$unihan-etl--destination./exported.{ext}Seeunihan-etl CLI argumentsfor advanced
usage examples.Code layout#cachedir(Unihan.zipisdownloaded,contentsextracted){XDG cache dir}/unihan_etl/#outputdir{XDG data dir}/unihan_etl/unihan.jsonunihan.csvunihan.yaml   # (requires pyyaml)#packagedirunihan_etl/core.py    # argparse, download, extract, transform UNIHAN's dataoptions.py    # configuration objectconstants.py  # immutable data vars (field to filename mappings, etc)expansion.py  # extracting details baked inside of fieldstypes.py      # type annotationsutil.py       # utility / helper functions#testsuitetests/*APIThe package is python underneath the hood, you can utilize its fullAPI.
Example:>>>fromunihan_etl.coreimportPackager>>>pkgr=Packager()>>>hasattr(pkgr.options,'destination')TrueDeveloping$gitclonehttps://github.com/cihai/unihan-etl.git$cdunihan-etlBootstrap your environment and learn more about contributing. We use the same conventions / tools across all cihai projects:pytest,sphinx,mypy,ruff,tmuxp, and file watcher helpers (e.g.entr(1)).More information