Bloat of Datain theUnicode Era

Behdad Esfahbod
behdad@behdad.org

The FarsiWeb Project
http://www.farsiweb.info/
Desktop Developers’ Conference ’05
July 18, 2005

Agenda• The Importance of Unicode• Unicode Character Database• Common Locale Data Repository• Future Plan

2

The Importance of Unicode• The Old DaysTM: Gazillions of 8-bit charactersets• ISO 10646: A unified character set• The Unicode Standard: And unified algorithms
to deal with these unified character set

3

Intro to Unicode• Currently at 4.1.0 release• Not 16-bit, 21-bit(16 + log2 17 = 20.087462841250339-bit)• A unique non-negative integer less than
1,114,112 assigned to each character

4

Intro to Unicode (continued)• Slightly less than 100,000 characters registered sofar• New scripts and characters are encoded with eachrelease• Major releases published as a book, with online
updates for minor releases
5Intro to Unicode (continued)• The book is available online as PDF files• The updates and other references are available inplain HTML• Data files as text files

6Architectural View of Unicode• The book: The Unicode encoding model,
Encoding model and issues for individual scripts• Key specifications: Standard Annex, Standard
Report, or Technical Report, algorithms for
rendering or otherwise dealing with text

• Data files: Unicode Character Database, text
files that define character properties and internalmappings
7

Key Specifications• Unicode Collation (UCA)• Bidirectional Algorithm (Bidi)• Normalization (NFC, NFD, . . . )8

Unicode Character Database• More than 70 character properties• The canonical character name, eg. U+0041 is
LATIN CAPITAL LETTER A• The most commonly used one is the General
Category, eg. U+0041 is Lu: Letter, upper case

• Mostly binary and enumerated properties
9

They Show Up Everywhere• Glibc character types: isalpha, isdigit, isprint,. . . (ctype.h)
Warning: The C standard limits the value ofsome of these functions• Convenience and module libraries: Glib has some,
Qt’s QChar class has some, Python’s unicodedata
module has the important ones, Perl supports allof them in regular expressions
10

And in (Some) Applications• Gucharmap uses them of course• Terminal emulators use the wcwidth function fromMarkus Kuhn

• But not much more

11

Where Else is it Useful?

• I want my editor to show the character names• Unicode regular expressions (PCRE)• Wherever a list of scripts is useful

12The Problem• Glibc is not available everywhere• The manual and Perl-script approaches, the2-year cycle, performance• Different versions of the data around : And old
Glibc, Glib, FriBidi in Pango, wcwidth ingnome-terminal, gucharmap, ...
13

The Problem (continued)• File formats, default values, etc, change. Can gounnoticed• High entry cost for getting the data in yourapplication• Support for new scripts is broken for years

14Ideally• A new approach to Unicode libraries: Only data,no converters, no algorithms• A central library exporting the UCD efficiently• Easier maintenance, easier update15

Ideally (continued)• Better memory overhead, more sharing• Problems in format change, etc have morechance to get noticed• Different versions of the UCD can live together(IDN requires 3.2)

16

Ideally (continued)• A runtime library that you can query properties
efficiently• A development kit that generates efficient
lookup-table code for pedantic projects• Central translation effort for property names,
script names, character names, etc
17Where are We Now?
• Planning• Got the name: gNUichar• A binding-friendly efficient design• Fetch compressor and bits from different projects• Release and advertise18

Localization• Much trickier than internationalization• O(n2 + k.n) where n is the number of languages
and k is the number of different atoms• More exposed to the end user: date formats,number formats, language names, country names,etc
19Locale Data
• Glibc has the basic functionality, but very limited• Evolution has a handful of date formats totranslate, other modules have too• Several projects maintain a list of language names
and countries, that get translated separately• Paper sizes, date formats, currency, timezone, etc
20The Problem• Again, Glibc is not available everywhere• If no Glibc locale, no support• Translating country names and language names isquite hard

• Maintenance is a nightmare
21

Common Locale Data Repository• A group effort coordinated by the UnicodeConsortium• Backed by companies like IBM, Sun, Apple, etc

22

CLDR Architecture• Current version is 1.3.0• Released as a set of XML files• Using inheritance to reduce the effort• XML and file-based inheritance, makes it hard touse
23

The Problem (continued)• The XML architecture makes it pretty hard touse CLDR in an application• Overlaps with the Glibc data

24

Ideally• A central library to export the CLDR efficiently• Qt has its own locale system, convert• GNOME doesn’t have a locale system, push in25

Currently• A new list created for discussion,
locale-list@gnome.org• ICU may be finally useful• A long way to go, help needed, in design andimplementation

26

In the Future• Get these two libraries released• Build a higher-level locale library for GNOME• Start cleaning up GNOME and KDE• What else? Questions?

27