The Properties and Promizes of UTF-8

M.J. Dürst

The Properties and Promizes of
UTF-8

Martin J. Dürst
University of Zurich
Switzerland
mduerst@ifi.unizh.ch
http://www.ifi.unizh.ch/staff/mduerst/

The author can be reached as follows:
Martin J. Dürst, Department of Computer Science, University of Zurich,
Winterthurerstrasse 190, CH-8057 Zurich, Switzerland;
email: mduerst@ifi.unizh.ch

References:[Alv97] Harald T. Alvestrand, IETF Policy on Character Sets and Languages,
currently under discussion on the mailing list ietf-charsets@innosoft.com.
[ftpint] B. Curtin, Internationalization of the File Transfer Protocol, Internet Draft
draft-ietf-ftpext-intl-ftp-02.txt (work in progress), June 1997.
[PTT94] R. Pike, K. Thomson, and H. Trickey, Unicode in Plan 9, Proc. Unicode
Implementers’ workshop 6, Unicode, Inc., San Jose, CA, 1994.
(continued on next page)

11th International Unicode Conference-1-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Overview
F Motivation

F What is UTF-8
F Properties of UTF-8
F Heuristic detection of UTF-8
F Where is UTF-8 used
F Conclusions

This presentation gives an overview over UTF-8, one of the three currently defined
UCS (Universal Character Set) Transformation Formats. The others are UTF-7 and
UTF-16, and at some point, there was also UTF-1. UTF-8 also has been called
UTF-FFS (for File Format Safe) and UTF-2 (as the second format after UTF-1).
All current UTFs use the number of bits per unit for their distinction.
The focus of this talk is on the various properties of UTF-8, and in particular on
the possibility and indeed very high chances of heuristic detection of UTF-8.
References (cont.):[RFC2044] François Yergeau, UTF-8, A Transform Format for Unicode and ISO
10646, Alis Technologies, October 1996 (update in preparation).
[RFC2047] K. Moore, MIME Part Three: Message Header Extensions for NonASCII Text, University of Tennessee, November 1996 (replaces RFC 1522).
[RFC2070] François Yergeau, Gavin Thomas Nicol, Glenn Adams, and Martin J.
Dürst, Internationalization of the Hypertext Markup Language, January 1997.
[Wei96] Chris Weider et al., The Report of the IAB Character Set Workshop, RFC
2130, April 1997.

11th International Unicode Conference-2-

San Jose, September 1997
3

The Properties and Promizes of UTF-8

M.J. Dürst

General Motivation
F UTF-8 is used or suggested for many

applications
F Understanding the properties of UTF-8 will
increase and improve its use
F History: First used in Plan 9 (Bell Labs’

UNIX successor), later adopted by X/Openand then Unicode and ISO 10646

The accronym UTF-8 appears in many specifications, and is proposed for many
more. It is therefore important that its properties are well understood, so that it canbe used most efficiently and in the appropriate places.
UTF-8 was invented at Bell Labs [PTT94] by the inventors of the Unix operating
system for its successor, called Plan 9. While its inventors wanted to use an
encoding that had the potential of transmitting all the characters of the world, they
didn’t want to depart from the concept of byte streams, and did not want toredefine a byte as 16 bits, because this induced the problem of different endianness(big-endian or little-endian) on different hardware.
UTF-8 was later adopted by X/Open as the multibyte form of Unicode/ISO 10646,
and then adopted by both of the later when its potential and advantages whencompared to other formats, such as UTF-1, became apparent.

11th International Unicode Conference-3-

San Jose, September 19972

The Properties and Promizes of UTF-8

M.J. Dürst

What is UTF-8
F UTF-8 is a multibyte encoding of Unicode/

ISO 10646– Multibyte: Not all characters are represented
with the same number of bytes
F UTF-8 strictly protects US-ASCII– US-ASCII: 7-bit control/graphic characters– An octet that looks like US-ASCII is always
US-ASCII– A character in US-ASCII is always represented
directly

UTF-8 is a multibyte encoding of Unicode/ISO 10646. This means that thecodepoints of the later, which can be seen as integers without any specificrepresentation on a computer, are mapped to sequences of bytes in a deterministic
and reversible way. In particular, not all characters are represented with the same
number of bytes.
UTF-8 strictly protects US-ASCII. US-ASCII here means the whole 7-bit range,
with the usual control characters in C0 and the IRV(international reference
version) 1991 of ISO 646 as the graphic characters. The correspondence between
UTF-8 and US-ASCII is very strong: An octet that looks like US-ASCII (i.e. has
the 8th bit set to 0) can only represent an US-ASCII character, and an US-ASCII
character is always represented with the same octet as in US-ASCII.

11th International Unicode Conference-4-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

UTF-8 Bit/Byte Patterns
F 0xxxxxxx(US-ASCII)

F 110xxxxx 10xxxxxx
F 1110xxxx 10xxxxxx 10xxxxxx
F 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
F 111111xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
F 1111111x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

10xxxxxx

The bit/byte patters range in length from 1 byte (octet) to 6 bytes. The 1s and 0s
represent those bits that define the structure of a pattern, the x define the payload,
i.e. the lower, significant bits from the UCS-4 (or UCS-2) number of the characterto be encoded
A very high regularity can easily be detected. With the exception of the pattern oflength 1, the number of leading 1s in the first byte of a sequence indicates the total
number of bytes in the sequence. The remaining bytes all start with 10, and have a
payload of 6 bits. By chance, but rather conveniently, 3 bytes are exactly enoughto encode the whole BMP (base multilingual plane), and six bytes are exactly
enough to encode all of UTF-4 (31 bits).
The easiest way to picture the pattern structure is by immagining each pattern to bea train of a different length. The initial railcars need bigger and bigger engines topull the rest of the train, and therefore can carry less and less payload. The
regularity of the patterns has many interesting properties which are discussed later.

11th International Unicode Conference-5-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

UTF-8 GraphicallyASCII(alone)two-bytes leading byte

2
3
4
5
6
7
8
9
A
B
C
D
E
F

leading four bits
0 1 2 3 4 5 6 7 8 9 A B C D E F

trailing bytes

trailing
four 0bits 1four-bytes leading byte

five-bytes leading byte
six-bytes leading byte
unusedthree-bytes leading byte

UTF-8 assigns each of the possible 8-bit bit combination (bytes/octets) a specificrole, which is shown above. Half of the combinations are reserved for ASCII, halfof the rest for trailing bytes (all but the first byte of a sequence), and the rest for
leading bytes, with clear separation of leading bytes for sequences of different
lengths.
This clear separation assures that UTF-8 can easily be parsed, that while
processing a text string, local orientation is possible, and that resynchronization ispossible in the case of transmission errors.

11th International Unicode Conference-6-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

What is not UTF-8
F UTF-16 surrogate pairs encoded as two

UTF-8 codepoints of three bytes each
F UTF-8 byte sequences longer thannecessary
F Misusing unused byte combinations

UTF-8 is very well designed, and addresses a very clearly defined purpose.
Unfortunately, sometimes ideas come up to change UTF-8 to make it supposedly
even better. Each of these ideas, however, endangers some properties of UTF-8
and opens the door to all kinds of proprietary variants that interfere with
communication. The Unicode Technical Commitee therefore has clearly expressed
itself againts any modifications or adaptions of UTF-8, and hopefully will firmly
keep this position in the future.
In particular, the following behaviour proposed or observed in some
implementations is not UTF-8 and should never be presented as such. First, UTF16 surrogate pairs could be encoded in UTF-8 as two sequences of 3 bytes each
(this behaviour is observed in some Java implementations). Because UTF-8
however is not limited to the BMP, this is forbidden, and the correct 4-bytesequence must be used.
Second, UTF-8 sequences could be prolonged with leading 0s as payload. This
usually doesn’t disturb decoders too much, but can confuse other processing, suchas searching, heavily. Third, it is possible to devise schemes that produce byte
sequences that cannot appear in UTF-8, and to assign some special control
functions to them. This is very dangerous, becaue decoding and processing can beaffected in arbitary, uncontrollable ways. As a conclusion, UTF-8 alwasy has to be
used exactly as defined.

11th International Unicode Conference-7-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

The Properties of UTF-8
F Covers full ISO 10646 UCS-4
F Binary sorting identical to UCS-4
F Easily parsable variable-length encoding
F Fully compatible with ASCIIF Reasonably compact for ASCII-heavy text
F Reasonable use of 8-bit channel
F High chance of heuristic identification

UTF-8 has a large number of very interesting and useful properties. It is difficult tosay which one of them is most important; it’s probably their combination thatmakes them so desirable. Some of them, in particular the full compatibility withASCII, have already been discussed. Others, in particular parsabilit, compression(or expansion) characteristics, and heuristic detectability will be discussed in detaillater.
The fact that UTF-8 covers the full 31-bit codespace of UCS-4 is important tosome for principles sake, but when compared againts Unicode/UTF-16, which canrepresent a codespace of 17 * 2^16, i.e. more than one million, codepoints, it is oflittle relevance, because although one has to be careful of “appetite comes witheating” effects in new codepoint alloctations, already the UTF-16 codespace islarge enough that it will suffice longer than the history of ASCII up to the presentday.
Tongue-in-cheek, the only chance that this is being filled quickly is that the earth
is invaded by some intelligence from outer space, but if this happens, they will
most probably bring with them their own computer technology and codespace :-).
When sorted byte by byte and bit by bit, UTF-8 strings also sort in the same
sequence as their UCS-4 equivalent.

11th International Unicode Conference-8-

San Jose, September 19978

The Properties and Promizes of UTF-8

M.J. Dürst

Parsing UTF-8
F Problem: Where are the characterboundaries in UTF-8
F Solution: Before bytes of the form
0xxxxxxx or 11xxxxxx [in C: ((b&0xC0)!=0x80)]
F No overlapping matches– No need to restart at start of text string– Search: No false positives; 8-bit structure
allows use of wellknown, fast algorithms(e.g. Boyer-Moore), with small tablesMultibyte character encodings in general are known to present all kinds ofdifficulties for parsing; indeed in the case of ISO 2022 based switching schemes,parsing usually is only possible from the start of a text string, which induceds a lotof complexity and overheads.
UTF-8 is clearly different. The experiences with parsing problems for othermultibyte encodings lead to a very clean design. It is always possible to detect
whether there is a character boundary between two bytes. Local processing istherefore always possible, with a backup of at most 5 bytes or at most one
character. The test for a character boundary is very simple, an examlpe for the
programming language C is shown.
The clear structure with easily detectable character boundaries is also responsible
for the fact that there are no overlapping matches, i.e. that it is not possible thattwo or more sequential octets that belong to different characters can look the sameas a correctly encoded UTF-8 character. This in particular means that whensearching for a character string, there is no danger of finding false positives. Also,
the 8-bit structure allows to use wellknown algorithms and implementations forfast search such as those of the Boyer-Moore variety. Because these algorithms
construct various tables that increase dramatically in size when used for 16 bit
entities, special care is necessary when they are applied to UCS-2 or UTF-16.

11th International Unicode Conference-9-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

ASCII Compatibility
F ASCII is ASCII is ASCIIF This property is shared with many othercharacter encodings (iso-8859-X, EUC-X,...)F Many protocols, formats, and programming
languages,... use only ASCII as syntactic
delimiters
F Main compatibility condition for those:
Pass 1xxxxxxx through as data octet

Absolute and complete ASCII compatibility is one of the core properties of UTF8. Because this is shared with many legacy character encodings, there is a large
repertoire of generic processing software that can be used without changes.
This in particular applies to protocols and formats that use only ASCII charactersas syntactically relevant entities, and reserve all the other characters for comments
and data such as text strings. This is true of a large number of protocols, inparticular the application level protocols of the internet (where unfortunately, 8 bit
transparency is not guaranteed in all cases, but is steadily increasing even for oldcornerstones of the 7 bit world such as electronic mail), as well as for a large
number of programming languages.

11th International Unicode Conference-10-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Compression/Expansion Factors:
Single Characters
F ASCII
F Up to U+07FF
F Alphabetic in BMP
F Ideographic in BMP
F Outside BMP

local Unicode UTF-8
1
2
1
1
2
2
1
2
3
2
2
3
1/2/3 4
4

UTF-8, because it is a variable length encoding, leads to different compression or
expansion factors for different classes of characters. ASCII is represented as 1byte, characters up to U+07FF (Arabic) are represented with two bytes, the rest of
the BMP, both alphabetic and ideographic charcters, are represented as three bytes,
and characters outside the BMP, but representable with UTF-16, need four bytes.
Because roughly, the characters have been allocated in ISO 10646 according tofrequency, UTF-8 can result in an overall compression.This is in particular true for
ASCII-heavy data streams; the optimal information density is reached in the casethat on average, every second byte (not character) is from the ASCII range. ASCIIheavy data streams are rather frequent; HTML with all tags in ASCII is a good
example.
Of course, not only the amount of ASCII in formats that as real text may notcontain a single ASCII character is of importance, but also the huge amount of
ASCII used in the English-speaking world and by all languages that use Latin.
It should be mentionned that even in rather inefficent cases, these inefficiencies
turn out to be mostly lower than those e.g. of RFC 2047 (formely RFC 1522), aformat for tagging character encodings in 7-bit message headers.

11th International Unicode Conference-11-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Compression Factors: Languages
local Unicode UTF-8
F Latin without accents 1
2
1
F Latin with accents1
2
1.1-1.4
F Gr, Cyr, Ar, Heb
1
2
2
F Indic, Georgian,...
1
2
3
F Ideographic in BMP 22
3
F Outside BMP
1/2/3 4
4

UTF-8 is reasonably compact, in particular for Latin-based languages, where baseletters need one octet and accented letters mostly can be dealt with using two
octets, and are not very frequent. Also, it is rather compact for texts from East
Asia, where native two-octet characters are expanded to three octets. It issomewhat less efficient for other alphabets, which depending on their location in
the BMP may need two or three octets per alphabetic character. Greek, Cyrillic,
Hebrew, and Arabic are the “lucky” alphabets that got away with two bytes.
Indic alphabets, South East Asian alphabets, Georgian, Tibetan and others suffer
most because they get expanded to three bytes. Considering worldwide overall
efficiency, this may well be justified in terms of coding efficiency by the fact thatthose languages are currently not that heavily computerized. However, the fact thatthese regions can afford large memories and transmission capacities much lessthan the very affluent West that needs barely more than 1 byte per character carriessome potential for conflicts. It has however to be noted that generic compression
methods, currently present in most storage and transmission devices, easily reduce
the three bytes of UTF-8 back to close to one byte per character for theselanguages.

11th International Unicode Conference-12-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Heuristic Identification: Motivation
F UTF-8 is an excellent solution in many cases
F Unfortunately, some places are alreadyoccupied (in protocols, formats)
F Replace previous encodings with UTF-8?
F To avoid confusion, we have to:– Identify encoding (advantages of UTF-8 are lost!)– Rely on detection/guessing (-> heuristics)

UTF-8 is the encoding of choice to use in existing 8-bit-clean, ASCII-heavy
systems. Existing protocols and formats have however already otherwise used the
8-bit channel, either by default or by mutual agreement between involved parties(e.g. for national or regional contexts). It seems difficult to replace these by UTF8. Labeling UTF-8 only is better than identifying all encodings, because it reflects
the special role of UTF–8. But labeling the future instead of the past is not reallydesirable.
Fortunately, it turns out that UTF-8 in some way labels itself. Its regular patterns
rarely if every turn up in other encodings. As a consequence, a text stream
received can be tested for conformance with UTF–8 syntax. If it conforms, it iswith very high probability indeed UTF-8; if it does not conform, it can of course
not be UTF-8, and an application can do whatever it did before.
Because probability to detect UTF-8 is high, but not 100%, this is a heuristic
method. In the following, we present a methodology and results that allow toassess the probability of such heuristics in various cases.

11th International Unicode Conference-13-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Heuristic Identification:
Experiment Methodology
F Work out scenario– What has to be identified against what odds
F Collect/generate data

– Dictionaries, file names, random generators...
F Perform experiments
F Evaluate Results

When plannig to introduce UTF-8 for certain elements of a protocol or format
where previously, another default or various unlabeled encodings were used, theprobability of correct identification of UTF-8 should be assessed to balance itagainst the accuracy needs of the protocol.
This usually should be done by the following methodology. First, a scenario isworked out, which determines what kinds of UTF-8 strings have to be identified
against what kinds of legacy strings. Second, data has to be collected or generated
which appropriately reflects the scenario. This can include sample texts, world lists
from dictionaries, file names or other identifiers. In some cases, generating data
randomly or generating all possibilities according to a specific pattern should also
be considered. Third, the actual experiments should be conducted, involving
conversion between various encodings, checking of canditate strings against
possible patterns, and collection of statistics or individual cases.
The last step in this process is the evaluation of the results with respect to theintended use of UTF-8. Basically, only 100% correct identification is acceptable.
If this is not achieved, the result can be improved in several ways.

11th International Unicode Conference-14-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

How to Improve Heuristics
F Decrease granularity
F Increase test precision
F Add test for actual data
F Allow user overriding

There are various ways to improve the heuristics if the performance as evaluated is
not satisfactory. In most cases, precision increases exponentially with the length of
the text(s) used for guessing. If for example guessing on individual filenames is
not satisfactory, performance can be improved by decreasing the granularity, inthis examlpe by e.g. using all the filenames in a directory for guessing, assuming
they all use the same encoding.
In other cases, it may be possible to increase test precision. The simplest test for
UTF-8 conformance does not include patterns that have shorter encodings (see p.
7). If higher precision is necessary, a test for the actual occurence of a codepoint inUnicode can be introduced, although this is not forward-compatible. In somecases, additional plausibility checks such as testing for sequences of charactersfrom the same script are possible.
In other cases, it is possible to add tests for actual data. This makes it possible forexample to avoid having two filenames in a specific directory that may beinterpreted as the same character sequence when using different encodings.
Another possibility is to add some plausibility checking by the end user. Even
syntactically valid byte sequences on misinterpretation usually lead to character
strings that don’t represent any kind of meaningful words. In this case, the user
just may be offered to select another encoding.

11th International Unicode Conference-15-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Scenario: Random Strings
F What is the probability of an arbitrary octet string toconform to UTF-8, even if it is not UTF-8?
F Probability varies with length
F Calculation using random number generator
Length
1
3
5
7
9
12

Probability
0.000000
0.037098
0.018777
0.006910
0.002397
0.000410

Length
2
4
6
8
10
15

Probability
0.035971
0.027588
0.011753
0.004001
0.001314
0.000081

The probabilities above have been calculated by not producing byte values below
24 (0x17) to account for the usually low probability of control codes in the C0
range. The probabilities of all other octets are the same. This therefore represents
text strings with slightly more true 8-bit octets than ASCII octets. It is assumed
that comparison is against an encoding the preserves ASCII in the same way as
this is done by UTF-8. Therefore, the probabilities don’t include the cases where astring was detected to be pure ASCII (but they included mixed ASCII/UTF-8
strings). As a consequence, the probability for strings of length 1 is 0, because
there is no true (non-ASCII) UTF-8 string of length 1.The increase in probability
from strings of length2 to thoseof length 3 is explained on the next page.

11th International Unicode Conference-16-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Heuristic Identification:
The Knappsack EffectF Non-ASCII UTF-8 sequence patterns– Length 1: always ASCII– Length 2: 2
– Length 3: 3
– Length 4: 22 (or 4, but no codepoints yet)
– Length 5: 23 or 32– Length 6: 222 or 33– Length 7: 223 or 232 or 322– Length 8: 2222 or 233 or 323 or 332– Length 9: 2223 or 2232 or 2322 or 3222 or 333

The initial increase and later decrease of the probability of a random byte sequence
to be UTF-8 with increasing length is due to the fact that UTF-8 sequences have to
fit in full into the byte sequence of the given length. Extracting an arbitrary
substring out of an UTF-8 string does in no way guarantee that the substring is also
UTF-8.
The string boundaries therefore restrict the possible patterns heavily for shorter
strings; for longer strings, the effect of the boundaries decreases.

11th International Unicode Conference-17-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Scenario: HTTP Resource Names
F HTTP resource names (e.g. filenames) arein a single native encoding
F Resource names already exist on the server– This excludes query parts (form submission)
F Client sends names native or in UTF-8(allows upgrading from native to UTF-8)
F What is the chance for confusion?

The second heuristic UTF-8 detection scenario is closer to a practical application.
Currently, HTTP resource names (HTTP URLs) are in ASCII for ASCII, but canbe in any local encoding whatsoever, without any identification, for the charactersand octets beyond ASCII. It would be very convenient if it were possible to change
this to use UTF-8 uniquely. The question is whether in a transitory period, with aserver accepting both the previous legacy encoding and UTF-8 to account for older
and newer browsers, these two encodings can reliably be distinguished. If this canbe done, this means that there is a viable upgrade path.
The problem is examined for resource names that already exist on the server. Thisin particular excludes query parts. The big difference between real resource names
and query parts is that the namespace of the former is usually very sparsely
populated and easily accessible, whereas the namespaceof the later is much more
densely populated and not directly accessible (e.g. one has to pass the queryarguments to a script and try to detect whether they are accepted or not, which may
be very difficult).

11th International Unicode Conference-18-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

HTTP Resource Names: Methodology
F

Ignore ASCII (always identical)
F Generate all possible octet combinations of length X (X=1,2,3,4,…)
F For each length and encoding to be tested:– Assume octets are native, eliminate impossible ones(e.g. C1 for ISO-8859-X)– Assume octets are UTF-8, convert to nativeu Eliminate combinations that are not UTF-8 patternsu Eliminate combinations that do not represent a characterfrom the native repertoire– Eliminate combinations that have lost octets– Show UTF-8 and native representation side-by-side

For the scenario in question, ASCII is not a problem, it will usually be identical(exceptions include cases such as shift_jis). To be able to assess the actual
potential of confusion, it is important that all possible octet combinations aretested, within the limits of computational power and the repetitions that result in
longer strings.
The generated octet combinations are carefully culled in various ways. Octet
combinations that cannot appear in the native encoding can be eliminated. Even if
they are valid UTF-8, they will never match a local resource name directly andtherefore cannot cause confusion. Also, octet combinations that are not valid UTF8 can be eliminated because they can only be interpreted as native. But what ismore, octet combinations that are valid UTF-8, but don’t represent charcters in
UTF-8 that can also appear in the native encoding can also be eliminated; thisusually results in a drastic reduction. Any strings that lost any octet can beeliminated immediately or at the end of the process.

11th International Unicode Conference-19-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Encodings with no possibility ofconfusion with UTF-8
F KOI-8 (Cyrillic, without additions of KOI-8R)
F ISO-8859-8 (Hebrew)
F EBCDIC (base version)
F CP 866 (Russian MS-DOS)

There are a number of well-known legacy encodings that don’t pose any
possibility for confusion. These include the ones listed above, and possibly more.
In the case of KOI-8, the reason for this is easy to see; there are no UTF-8 trailingbytes in KOI-8, and therefore no non-ASCII UTF-8 octet combinations that arealso KOI-8.
In the other cases, the fact that there is no possibility for confusion is not
immediately obvious; it results both from the fact that the theoretical number ofcodepoints is not fully used and from the fact that the mappings of the charactersin these encodings to UTF-8 all contain octets that are not characters in theseencodings. In the case that more codepoints are filled up (as is the case e.g. with
Microsoft codepages based on the ISO 8859 series), some possibilities forconfusion will appear sooner or later.

11th International Unicode Conference-20-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Latin-1: Confusion Possibilities
FF

64 sequences of two
octets with twointerpretations,shown side-to-side
Heuristic fails if two
resouce names are
so close that they
can be changed into
each other by one ormore pair exchange
Examples:F

Judge for yourself!

F

UTF-8 Latin-1

UTF-8 Latin-1

UTF-8 Latin-1

UTF-8 Latin-1

For the very widely used Latin-1 (ISO-8859-1), the list of possible confusions islonger. Indeed, all non-ASCII characters of Latin-1 expressed in UTF-8 can be
confused with a two-letter combination of such characters expressed in Latin-1.
This completeness is due to the fact that Latin-1 is directly part of Unicode.
But despite this full list of possible confusions, the potential for actual confusion isextremely low. Many of the two-letter combinations on the right side of the table
look completely strange, and the probability that they ever appear in a resource
name is extremely low. For this, the fact that Latin-1 contains only special
characters in the area of UTF-8 trailing octets is partially responsible.
The possibility for such character combinations to appear is truely low, but it turns
out that we are not done yet. To produce an actual case of confusion, not only dothese letter combinations have to appear, but there have to be two resource namesthat are identical except for the complete exchange of all letters in the first column
by the corresponding combination in the second column. The reader is asked tojudge the probability for this him/herself.

11th International Unicode Conference-21-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

F Latin-2(iso-8859-2)

F Latin-3(iso-8859-3)

Other encodings than Latin-1 present similar situations. Because in particular in
the case of Latin-2, there are quite some true letter combinations, the probabilities
are more difficult to assess in general. The tables provide however a good base toasses the danger of confusion for specific languages or language combinations. If a
specific language does not use all three letters (a single letter on the left and thecorresponding two letters on the right), then this combination presents no danger.
Further checks can then be made with a dictionary, although there is the problem
that a dictionary never contains all possible words, and that of course resource
names don’t necessarily have to be words.
The difference in fonts used for the different characters is due to the fallback
mechanisms utilized in the framework [???] whith which these charts have been
produced.

11th International Unicode Conference-22-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

F Latin-4(iso-8859-4)

F Cyrillic(iso-8859-5)

The two encodings presented on this page are similar to those presented earlier,and to many others. In the case of Cyrillic, it is interesting to note that a lot of twoletter combinations can be rejected easily because of their unconventional
capitalization.
The author intends to document confusion possibilities in the way shown above formany more encodings in the form of a web site which will contain the tables bothas GIFs and as UTF-8 text. In order to be able to produce a table for a given
encoding, two things are necessary: The first is a converter between UTF-8 andthe encoding in question in both ways, and in both cases eliminates impossible
characters or octets (or replaces them by something that can easily be eliminated in
an additonal step). The second is the possibility to display all the characters inorder to produce a GIF. Help on both points is greatly appreciated.

11th International Unicode Conference-23-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

FTP Internationalization
F Officially, file names are restricted to ASCIIF 8-bit implementations and ad-hoc use common
F Draft recommends UTF-8 as preferred encoding

for file names
F Implementation possibilities:– Server/client converts filenames– Local filenames become UTF-8– UTF-8 links to local filenames– Client displays what it can recognize as UTF-8– Implicit “agreements” for other encodings still possible
F Same can work for HTTP!!!

The current version of FTP (the File Transfer Protocol) officially restricts filenames to ASCII, but 8-bit-transparent implementations based on octet identity arevery widespread. Octet identity of course cannot assure that the characters of thefilenames can be identified; in practice, this is done with extra knowledge of theclient about the server. This is clearly unsatisfactory.
The proposal for ftp internationalization [ftpint] is based on the recommendationof UTF-8 as the preferred encoding of file names in the protocol. This does notaffect the current practice because UTF-8 can be seen as just one more character
encoding that could be used anyway. But it allows implementations and users tosteadily proceed towards a more consistent solution.
The fact that UTF-8 can in most cases be distinguished from legacy encodings
allows the server-wise, directory-wise, or file-wise identification of UTF-8, andthe treatment of UTF-8 filenames by character identity, while legacy filenames aretreated by octet identity. Both server and client can convert filenames to their local
file system character set. The fact that some of the filenames cannot be converted
is not new to ftp which has to allow other, more severe, restrictions (e.g. caseindifferent 8.3 filenames).
[ftpint] contains an extensive discussion of this and several useful code fragments.

11th International Unicode Conference-24-

San Jose, September 1997
14

The Properties and Promizes of UTF-8

M.J. Dürst

UTF-8 in the Internet
F HTML: UTF-8 supported by V4 browsers
F LDAP (leightweight directory access protocol):Transition to UTF-8
F ACAP: fully UTF-8
F IMAP: modified UFT-7 (UTF-8 for URLs!)
F FTP: UTF-8 for internationalized filenames
F Usenet News: Moving towards UTF-8
F URNs: Syntax based on UTF-8
F New policy: Every IETF protocol text should
support UTF-8; UTF-8 as a default is preferred

Besides its early (and continued) use in the Plan 9 operating system, UTF-8 hasrecently become the preferred character encoding of the Internet community,
specifically the IETF (Internet Engineering Task Force) and the IESG (Internet
Engineering Strearing Group). A new policy that factually requires every IETF
protocol that transports text to support UTF-8, either as the only encoding or as
one of several encodings is close to being finalized [Alv97] based on the results of[Wei97]. In the realm of the IETF, UTF-8 is documented in [RFC 2044].

11th International Unicode Conference-25-

San Jose, September 1997

The Properties and Promizes of UTF-8

M.J. Dürst

Conclusions
F UTF-8 has many very interesting propertiesF UTF-8 can be used without much adaption

in many existing systems
F Careful examination of UTF-8 properties
leads to even better uses

The talk should have shown that UTF-8 is a multibyte encoding with an amazing
number of very interesting and useful properties, which allow it to be introduced inexisting systems without too much changes. While some of the properties, inparticular parsability and ASCII compatibility, have been designed into it from thestart, the ease and high success rate for heuristic detection in various scenarios, andthe high potential this brings for various applications, have not been discovered
until recently and are not yet fully understood, but extremely promizing.
The fact that UTF-8 has all these useful properties should not only be a motivation
for its use, but in particular for its correct use. Any kind of error or so-called“variant” immediately hampers several of the properties.

11th International Unicode Conference-26-

San Jose, September 1997