Package: demicode
It's Not Just Unicode, It's Hemi-Semi-Demicode!Demicode is a Python command line tool to explore the current, broken state of
fixed-width rendering forUnicodein terminals and
code editors. However, because terminals support styling a program's output withANSI escape sequences, they
also are more amenable to helpful visualization than code editors.Fixed-Width Character BlotsFeaturesInstallationVersionsEtcFixed-Width Character BlotsDemicode's core functionality is thefixed-width character blot, which
visualizes a single grapheme cluster's fixed-width rendering. Since the current
state-of-the-art uses two fixed-width columns at most, each blot is one more
column, that is, three columns wide. That extra padding makes glaringly obvious
when theoretical and actual width diverge. For terminals, said padding comes in
two forms, with the first usingU+0020 space in a different color to
highlight any overlap and the second using█U+2588 full block to obstruct
those same bits.The following screenshot shows an example fordemicode's output--with-curationwhen running in Terminal.app on macOS. Out of the nine
terminals I have been testing—Alacritty,Hyper,iTerm2,Kitty,Rio,Terminal.app,Visual Studio
Code's terminal,Warp, andwezTerm—I find Terminal.app's and
iTerm2's handling of overly wide glyphs the least bad. However, even with
demicode using ANSI escape codes to line up columns, Terminal.app still manages
to distort the column grid, as the lines for the technologist, person: red hair,
and rainbow flag emoji in the screenshot below illustrate. I haven't found an
effective work-around, despite trying several alternatives such as rendering
character information first and blots second.FeaturesDemicode supports the following features:Display fixed-widthcharacter blots together with helpful metadataone
grapheme per line.Or,display--in-grid/-gto fit many more graphemes into the same
window, albeit without metadata.For code points that combine with variation selectors,automatically show
the code point without and with applicable variation selectors.Optionallydisplay blots--in-more-color/-cand--in-dark-mode/-d. The first option may be given twice for even more
color. The second option usually is superfluous because demicode
automatically detects dark mode. See screenshot below.Run--with-curationand--with-…othercarefully selected groups of
graphemes. Or provide your own graphemes as regular command line
arguments. Both literal strings and Unicode'sU+…notation are acceptable.
Quote severalU+…forms to group them into a grapheme.Automatically download necessary filesfrom theUnicode Character
Database(UCD) andCommon Locale Data
Repository(CLDR) and then cache them locally.Automatically detect the most recent version of the UCD and the CLDR.
Since CLDR data serves one, non-normative purpose only, emoji sequence
names, demicode always utilizes the latest version. But--ucd-versionlets
you pick older UCD versions at will.In interactive mode,page the output. Let user control whether togo
backward or forwardwhile alsoautomatically adjusting to terminal
window size.On Linux and macOS,page backward and forward with the left and right
arrow keys. On other operating systems, useborpfollowed by‹return›to page backward; just‹return›or alternativelyfornfollowed by‹return›to page forward; and just‹control-c›or
alternativelyqorxfollowed by‹return›to terminate demicode. All
of these, no‹return›required, work on Linux and macOS, too. Plus‹delete›or‹shift-tab›to page backward;‹space›or‹tab›to page
forward; and‹escape›to terminate. So which triple is yours?In batch mode, i.e., with standard in or out redirected,emit all
character blots at once and consecutively.InstallationDemicode is written in Python and distributed throughPyPI, the Python Packaging Index. Since it
utilizes recent language and library features, itrequires Python 3.11or
later. The best option for installing demicode is usingpipx. If you haven't installedpipxyet,brewmakes that easy on Linux or macOS:%brewinstallpipx==>Fetchingpipx==>Downloadinghttps://ghcr.io/v2/homebrew/core/pipx/manifests/1.2.0
...
🍺/usr/local/Cellar/pipx/1.2.0:885files,11.2MB==>Running`brewcleanuppipx`...
DisablethisbehaviorbysettingHOMEBREW_NO_INSTALL_CLEANUP.
HidethesehintswithHOMEBREW_NO_ENV_HINTS(see`manbrew`).
%Once you havepipxinstalled, installingdemicodeis trivial:%python--version
Python3.11.1
%pipxinstalldemicodeinstalledpackagedemicode0.5.0,installedusingPython3.11.5Theseappsarenowgloballyavailable-demicodedone!✨🌟✨
%demicode--with-curation
...The output of the last command should look something like the first screenshot.Versionsv1.3.0(2024/01/07):Automate running demicode in popular terminals on macOS and collecting
either performance data or screenshots.Fix crash when mirror directory has not yet been created. Thanks toyohann84Lfor reporting this bug.v1.2.0(2023/10/30):Mirroring of UCD files has been refactored and now uses anexplicit
manifestfor tracking locally available versions.--ucd-mirror-allcauses demicode to eagerly download all files for all
versions, enabling fully disconnected operation.--ucd-list-versionslists all versions contained in mirror directory.With--inspect-latency/-T, demicode now measures page rendering
latency.Initial resultssuggest that the nine measured
terminals are reasonable fast rendering styled text, taking between
4–9 ms on a four-year-old macOS laptop. But when demicode also
queries the terminal for the current column, the spread of average
latencies explodes to 10–946 ms.To better track provenance of experimental results, demicode gains the
ability to determine terminal name and version—based on environment
variables, ANSI escape codes, and, on macOS, bundle identifiers.Demicode now uses GitHub actions for CI.v1.1.0(2023/10/17):Improve terminal intput/output, notably by--incrementally/-idisplaying character blots, which is significantly slower but allows for
measuring the size of blots.Fix crashing bug in path handling for mirrored CLDR files.Make internal handling of UCD data more uniform, with an eye towards
evolving demicode's UCD abstractions into a more generally useful
library.Switch from mypy to pyright, address pyright's improved diagnostics,
integrate type checking intoruntest.py, and improve test script
output.v1.0.0(2023/09/19):Support grapheme cluster segmentation according to Unicode 15.1 and 15.0.Tabulate bit size of Unicode properties, alternative groups of required
properties for--stats.Update internal interface for UCD data to favor generic access to
properties.v1.0.0b1(2023/09/12):In interactive mode, render every page from scratch, taking terminal
size into account. This enables paging forwardandbackward. On Linux
and macOS, use left and right arrow keys to control paging.In batch mode, i.e., when standard input or output are redirected, emit
all character blots without paging.Test file loading and property look up for  every supported UCD version
to squash any remaining crashing bugs. Nonetheless, advise in tool help
that default, i.e., latest version produces best results.In preparation of Unicode 15.1, add support for the
Canonical_Combining_Class, Indic_Syllabic_Category, and Script
properties. Remove support for unused Dash, Noncharacter_Code_Point,
Variation_Selector, and White_Space properties again.Clean up UCD file loading. Eliminate most boilerplate and private helper
functions indemicode.ucd.Eliminate global instance ofUnicodeCharacterDatabase. Leverage
independent instance for collecting statistics, eliminating need for two
tool runs to collect all data.v0.7.0(2023/09/06) Clearly distinguish between user errors and
unexpected exceptions; print traceback only for the latter. Modularize test
script usingunittest. In preparation of Unicode 15.1, specify which
versions to use for code generation.v0.6.0(2023/09/05) Fix handling of emoji data for early versions of
Unicode. Suppress blot for unassigned code points or sequences that are more
than one grapheme cluster; add explanatory note.v0.5.0(2023/09/04) Optimize range-based Unicode data for space and
bisection speed. Improve built-in selections of graphemes; notably, the
Unicode version oracle now displays exactly one emoji per detectable Unicode
version.v0.4.0(2023/09/01) Fix bug in URL creation for UCD files and move local
cache to the OS-specific application cache directory. Restructure and
simplify code to computewidth(), renamed fromwcwidth()due to changes.v0.3.0(2023/09/01) Add support for grapheme clusters in addition to
individual code points; account for emoji when calculating width; expose
binary emoji properties; log server accesses; add tests; and improve
property count statistics.v0.2.0–0.2.3(2023/08/13) First advertised release, with more robust UCD
mirroring, more elaborate output, and support for dark mode. Alas,
screenshot links and README still needed some TLC.v0.1.0(2023/08/06) First, downlow releaseEtcTheproject name is a play on the name Unicode: Fixed-width rendering of
Unicode can't get by with a singleuni-column—from the Latinunusfor
one—but requires at the very least ademi-view—from the Latindimidiusfor
half via the Frenchdemialso for half. As so happens,hemiandsemimean
half as well, tracing back to Greek and Latin origin, respectively.Alas, the real question is whetherhemisemidemi-anything is cumulative, i.e.,1⁄8, or just reinforcing, i.e., still1⁄2.I amworking on a technical blog postto provide more on motivation,
technical background, and first findings after blotting far too many Unicode
code points. One unexpected outcome is a test that should identify the Unicode
version supported by a terminal just by displaying a bunch of emoji.  😳I 💖 Unicode!Demicode is © 2023Robert Grimmand has been released
under the Apache 2.0 license.