Internationalizing Web Addresses
Objectives• Brief overview of web addresses,• Architecture for domain names and URI• Recent recommendations for use of nonEnglish characters,

Internationalizing Web
Addresses• How Unicode fits in• Current state of the art.

Tex Texin
Internationalization Architect, Yahoo!
Internationalizing Web Addresses

What is a Web Address?

IUC27 – Berlin, Germany – April 2005

Resource• Web Address – lay term for URI• Anything that has identity• URI – Uniform Resource Identifier• Conceptual mapping to an entity (or set ofentities)• Two types of URI
URL – Uniform Resource Locator– Not necessarily the entity

URN – Uniform Resource Name

– Not necessarily network retrievable– Providing conceptual mapping is unchanged:

What are they? What is a resource?

• resource does not need to physically exist

And what are IDNA and IRI?
Internationalizing Web Addresses• content can change

IUC27 – Berlin, Germany – April 2005

3

Internationalizing Web Addresses

Resource Examples– A name• Databases, Database contents– A locator• Applications, Services– Both• People, Corporations• Books, DVDsm
IUC27 – Berlin, Germany – April 20054

• An object that can act as a reference tosomething that has identity• Devices (e.g. printers, …)Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

Resource Identifier• Files of all types (e.g. *.txt, .jpg, .htm, …)• etc.

2

ea
or

tra
bs

ct

5

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

1

IUC27 – Berlin, Germany – April 2005

6

Berlin, Germany, April 2005

Internationalizing Web Addresses
URL and URN

Uniform (or Universal)• Uniform Resource Locator (URL)• Naming scheme that supports different
types of identifiers– Subset of URI that identify resources via a
representation of their primary accessmechanism (e.g., their network “location”)– in the same context, and in many contexts• common syntactic conventions• Uniform Resource Name (URN)• consistent semantic interpretation• independent of access mechanism– Subset of URI that remain globally uniqueand persistent even when the resourceceases to exist or becomes unavailable– extensible• new types do not break existing uses

– URNs are not necessarily retrievable
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

7

Internationalizing Web Addresses

Examples Of Uniformity

IUC27 – Berlin, Germany – April 2005

8

Uniform Resource Identifier• Different identifier types• Achieving Uniformity– http://www.yahoo.com/– Characters required for Transcribability• Napkin-compatible– https://calendar.yahoo.com/textexin• Memorable– mailto:textexin@yahoo.com– Common syntax across both schemes andcontexts– ftp://ftp.yahoo.com/public/– http://search.yahoo.com/search?p=texin• Implies syntax restrictions, and– file:///D:/tex/index.html#toc• Character escape mechanisms– urn:example:animal:ferret:nose
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

9

Internationalizing Web Addresses

Uniform Resource Identifier

IUC27 – Berlin, Germany – April 200510

URI Character Representation
Transcribable URI
Scheme-dependent
mapping

Transcribable charactersconforming to a restricted syntax
used for uniformly identifying
an abstract or physical resource

Excluded and Non-ASCII
characters are escaped (%HH)

URI as a sequence of characters with syntax restrictions
Usable: [a-zA-Z0-9] | "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"URI Delimiters: ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | ","URI mapped to octets
Potentially map octets to original characters(requires encoding knowledge)
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

11

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

2

IUC27 – Berlin, Germany – April 2005

12

Berlin, Germany, April 2005

Internationalizing Web Addresses
URI Syntax

URI Components• <scheme>:<scheme-specific-part>– Scheme: method to access the resource• <scheme>://<authority><path>?<query>– Authority (Domain Name or IP Address)• Name of the machine hosting the resource• New in RFC 3986 (replaces RFC 2396)– Path: resource name, given as a path<scheme>://<authority><path>?<query>#<fragment>– Query: Info. interpreted by the resource– Fragment– Note: with key=value pairs, value can be URI• indirect identification of a secondary resource by reference
to a primary resource and additional identifying information

http://search.yahoo.com/search?p=http://www.brad-pitt.com

Each part has its own syntax!
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

13

Internationalizing Web Addresses

Schemes• acap, cid, crid, data, dav, dict, fax,
file, ftp, go, gopher, h323, http, https,
im, imap, ipp, iris.beep, ldap, mailto,
mid, modem, mtqp, mupdate, news,
nfs, nntp, pres, prospero, rtsp, service,snmp, soap.beep, soap.beeps, tag, tel,telnet, tftp, tip, pop, opaquelocktoken,
sip, sips, urn, vemmi, wais, xmlrpc.beep,
xmlrpc.beeps, z39.50r, z39.50s• Defines the syntax and semantics of therest of the URI<scheme>:<scheme-specific-part>• Definitions are in IETF RFCs
• Scheme registry is at:– www.iana.org/assignments/uri-schemes/
IUC27 – Berlin, Germany – April 2005

14

Schemes• Declares the type of resource and theaccess method.

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

15

Internationalizing Web Addresses

Authority

IUC27 – Berlin, Germany – April 2005

16

Path and Query<scheme>:<scheme-specific-part><scheme>://<authority><path>?<query>#<frag.><scheme>://<authority><path>?<query>#<frag.>• path is specific to the authority (orscheme, if no authority), and identifies theresource within the scope of that scheme
and authority• authority = server | reg_name• server = [ [ userinfo "@" ] host [ ":" port ] ]• host = hostname | IP address• hostname=*(domainlabel ".") toplabel ["."]

path = segment *( "/" segment )– Labels consist of Letters, Digits andHyphen (LDH)
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

• query is a string of information to beinterpreted by the resource
17

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

3

IUC27 – Berlin, Germany – April 2005

18

Berlin, Germany, April 2005

Internationalizing Web Addresses
Fragment<scheme>://<auth.><path>?<query>#<fragment>

fragment= *( pchar / "/" / "?" )– indirect identification of a secondary
resource by reference to a primary resource
and additional identifying information– can be a portion or view of the resource ora reference to another resource– semantics depends on the primary resource,
it’s media type and is independent of theschemeInternationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

Internationalizing Web
Addresses
Tex Texin
Internationalization Architect, Yahoo!
19Internationalizing Web Addresses

Internationalizing Schemes• Native names are needed for usability<scheme>://<authority><path>?<query>#<fragment>• Many people don’t know Latin characters• International scheme names notstrongly needed– This makes it difficult to:• Input, write, spell, pronounce,• recognize, remember, and comprehend– Branding, image association desired• But backward compatibility is key!
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

21

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

22

IDNA Goals• Provide international standard• Backward compatible– Existing DNS and application protocols
continue

International Domain
Names in Applications• One architecture worldwide– Independent of region, country and language

Tex Texin
Internationalization Architect, Yahoo!
Internationalizing Web Addresses

27th Internationalization and Unicode Conference

4

IUC27 – Berlin, Germany – April 2005

24

Berlin, Germany, April 2005

Internationalizing Web Addresses
IDNA Design

Domain Name Systems• No changes to existing DNS architecture• DNS name is hierarchical, friendly
identifier for computer IP address• Applications and/or protocols compensate– The name character repertoire is expanded– e.g. search.yahoo.com, www.kelkoo.co.uk– A mapping to the old syntax is definedwww.kelkoo.de, 123.145.167.189• DNS name is different from Hostname– New names can only be used where theapplication or protocol has been upgraded– DNS allows any octet, case-sensitive– Application (http, srv, etc.) can restrict further• Hostname restricts to ASCII, case-insensitive

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

25

Internationalizing Web Addresses

DNS Name Resolution

IUC27 – Berlin, Germany – April 2005

DNS Name Resolution• DNS information maintained as a vast
distributed databaseyahoo.com?

• Name resolved to IP address by lookup

123.234.012.245– Client accesses name server(s)NameServers– Name servers access other name servers– Name servers retrieve, share and update
Domain Name and IP Address informationwww.yahoo.com
123.234.012.245• IDNA introduces a layer over DNS
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

27

Internationalizing Web Addresses

DNS Name Resolution
日本語.jp?

26

admin

IUC27 – Berlin, Germany – April 2005

28

IDNA Architecture

ASCIIequivalent

123.234.012.245•

International Domain Name entered•

Conversion to Unicode, if needed•

Nameprep (DNA profile of Stringprep)
1. Characters are folded or removed

Name
Servers

2. NFKC Normalization is applied (UTR15)
3. Prohibited characters removed•

http://日本語.jp
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

29

27th Internationalization and Unicode Conference

Unicode to ACE (ASCII-Compatible
Encoding) Conversion

Internationalizing Web Addresses

5

IUC27 – Berlin, Germany – April 2005

30

Berlin, Germany, April 2005

Internationalizing Web Addresses
IDNA Architecture

Nameprep Character Folding

http://日本語.jp

1. Case folding to lower case (UAX 21)
2. Additional folding

Convert to Unicode– Certain Greek characters

Nameprep

DNS Servers

Case fold, Mapping, NFKC, Removal– Symbols which include latin characters

ACE (Punycode, profile of Bootstring)– b = NFKC(a); c = NFKC(b);If c <> b then add a map a => c

Convert to ASCII, Prepend “xn- -”ResolverInternationalizing Web Addresses

http://xn--wgv71a119e.jp

3. Reduce typographic variations
Application– line spacing, variant selectors… e.g. zwsp

Servers

IUC27 – Berlin, Germany – April 2005

31

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

Normalization (UAX 15)

Canonical Equivalences• Equivalent strings are put into asingle standardized form NFKC

• Composed vs. Combining characters– “Å”

U+00C532(A-ring pre-composed)– “A+˚ ” U+0041,U+030A (A+combining ring above)– Allows fast binary comparison• Singletons– Reduces visual ambiguity– “Å”• Unicode defines two equivalences

U+212B(Angstrom)– Canonical and Compatibility– NFKC normalization standardizes bothInternationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

33

Internationalizing Web Addresses

Compatibility Equivalence• Width (ｶ カ)• Ligature (ﬁ)• Font variants (ℌ)• Breaking differences (-)• Cursive forms (‫)ن ﻧ ﻨ ﻦ‬

IUC27 – Berlin, Germany – April 2005

34

Prohibited Characters• Circled (⑪)• Characters prohibited before IDNA

• Size, rotated (﹠ ︷)• Space, replacement & control characters• Super/subscripts (₉ ⁹)• Private use characters• Squared (㌀)• Non-character and surrogate code points• Fractions (⅚)• Inappropriate characters (not for plain
text, display variants)• Others (ǆ)

interlinear annotation, ideographic description, leftto-right mark, activate arabic form shaping,ideographic full stop
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 200535

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

6

IUC27 – Berlin, Germany – April 2005

36

Berlin, Germany, April 2005

Internationalizing Web Addresses
Classes of Characters

Nameprep Versioning• Based on Unicode 3.2• AO – Allowed characters• MN – Characters Mapped to Nothing or
normalized away
• D – Disallowed Characters (prohibited)• U - Unassigned code points• Unassigned code points become AO, MN
or D when assigned & Nameprep isupdated.
• Applications treat unassigned code points asallowed• Only allowed code points in name servers– Names are not registered until IDNA andservers are updated– Assumes additional case folding is minimal

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

37

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

38

Punycode

ASCII Compatible Encoding (ACE)• Compression algorithm.

• ACE maps large character set to ASCII– Extract characters in ascending codepoint order• Bootstring algorithm minimizes length– Encode difference of codepoint from previous
characters and position in an integer.

– DNS labels are 63 bytes, max. 255 bytes– Extract Letters, Digits and Hyphen as bootstring.

• Approximately 16 ideographs/63 bytes.

• ASCII conversion algorithm.

• Punycode parameterization for DNS– Introduces ‘Generalized variable-length integers’.

– ASCII unchanged– BASE36 (A-Z, 0-9).

– Non- ASCII mapped to: a-z, 0-9, hyphen• Prefix chosen to identify IDN: “xn - -”Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

39

Internationalizing Web Addresses

日本語.ＪＰ

•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Convert to Unicode
65E5 672C 8A9E 002E 006A 0070

… FF2A FF30

Nameprep“xn- -” + Punycode

xn--wgv71a119e.jp
ICU Demo: www.ibm.com/software/globalization/icu/demo/domain
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

40

ExamplesInternational Domain Names

IDNA Architecture Example
日本語.jp

IUC27 – Berlin, Germany – April 2005

41

27th Internationalization and Unicode Conference

afghanistan
algeria
austriabangladesh
belarusbelgiumbulgariachad
china
comoroscyprusczechrepublic
egypt
greecehungary

Internationalizing Web Addresses

7

http://‫اﻓﻐﺎﻧﺴﺘﺎ‬.icom.museum
http://‫اﻟﺠﺰاﺋﺮ‬.icom.museum
http://österreich.icom.museum
http://বাংলাদেশ.icom.museum
http://беларусь.icom.museum
http://belgië.icom.museum
http://българия.icom.museum
http://‫ﺗﺸﺎدر‬.icom.museum
http://中国.icom.museum
http://‫اﻟﻘﻤﺮ‬.icom.museum
http://κυπρος.icom.museum
http://českárepublika.icom.museum
http://‫ﻣﺼﺮ‬.icom.museum
http://ελλάδα.icom.museum
http://magyarország.icom.museum
IUC27 – Berlin, Germany – April 2005

42

Berlin, Germany, April 2005

Internationalizing Web Addresses
ExamplesInternational Domain Names
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

icelandindia
iranirelandisraeljapan
jordankazakhstan
korea
kyrgyzstan
laoslebanonmacedonia
mauritania
mexicoInternationalizing Web Addresses

ExamplesInternational Domain Names

http://ísland.icom.museum
http://भारत.icom.museum
http://‫اﻳﺮان‬.icom.museum
http://éire.icom.museum
http://‫ישראל‬.‫איקו״ם‬.museum
http://日本.icom.museum
http://‫اﻷردن‬.icom.museum
http://қазақстан.icom.museum
http://한국.icom.museum
http://кыргызстан.icom.museum
http://.icom.museum
http://‫ﻟﺒﻨﺎن‬.icom.museum
http://македонија.icom.museum
http://‫ﻣﻮرﻳﺘﺎﻧﻴﺎ‬.icom.museum
http://méxico.icom.museum
IUC27 – Berlin, Germany – April 2005

•
•
•
•
•
•
•
•

mongolia
http://монголулс.icom.museum
moroccohttp://‫اﻟﻤﻐﺮب‬.icom.museum
nepal
http://नेपाल.icom.museumoman
http://‫ﻋﻤﺎن‬.icom.museum
qatar
http://‫ﻗﻄﺮ‬.icom.museum
romania
http://românia.icom.museum
russiahttp://россия.иком.museum
serbia montenegro
http://србијаицрнагора.иком.museum• sri lanka
http://இலங்ைக.icom.museum• spainhttp://españa.icom.museum• thailand
http://ไทย.icom.museum• tunisia
http://‫ﺗﻮﻧﺲ‬.icom.museum• turkey
http://türkiye.icom.museum• ukraine
http://украина.icom.museum•
vietnam
http://việtnam.icom.museum
Internationalizing Web Addresses
IUC27 – Berlin, Germany – April 2005

43

ExamplesInternational Domain Names

44

IDNA Issues

The previous IDNA examples are courtesyof:• Mapping Traditional-Simplified Chinese
Characters• Cary Karp, President,• Multiscript spoofs– www.PAYPAL.com using U+0391 Greek “A”• Museum Domain Management
Association, Sweden– Recommendation for registry restrictions• http://musedoma.museum/• From his presentation (session A1) at
IUC27 (http://www.global-conference.com/iuc27/program.html)
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

45

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

46

URI Path<scheme>://<authority><path>?<query>#<fragment>• URI path is ASCII-based• %HH encoding for non-ASCII characters– Character encoding information is lost
– so restoring original characters is risky

International Resource
Identifiers• No restrictions on equivalences– normalization, case folding• Bidirectional scripts are problematic

Tex Texin
Internationalization Architect, Yahoo!

Internationalizing Web Addresses

27th Internationalization and Unicode Conference8

IUC27 – Berlin, Germany – April 2005

48

Berlin, Germany, April 2005

Internationalizing Web Addresses
Internationalized Resource
IdentifiersInternationalized Resource
Identifiers• IRI requirements• Solution: RFC 3987
– Similar to IDNA, create a new construct
Internationalized Resource Identifiers (IRI)– Distinguish IRI objects from URI– Distinct from URI, with a mapping to URI– Don’t use IRI where URI hasn’t upgraded– Support large character set– base encoding on UTF-8• e.g. XML “anyURI”– www.w3.org/TR/xmlschema-2/#anyURI– Leave URI untouched and define where IRI
can be used and when conversion to URI
occurs.

• for all URI,
• scheme-specific URI,
• URI component (e.g. fragment), or• specific URI within a scheme– Maintains backward compatibility
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

49

Internationalizing Web Addresses– Not in existing schemes, except by design– Newly designed elements– Presentation equivalents of existing
protocols– When used for retrieval, URI is generated<scheme>://<authority><path>?<query>#<fragment>• IRI Path Component– Unicode NFC (not NFKC) Normalization• Canonical equivalence only
• Applied to legacy encodings only

• Scheme may have additional syntax restrictions• Validating URI eliminates defining equivalent IRI
validation• Verify URI retrieval location– Transcode to UTF-8 before %hh encoding• Steps can also apply to othercomponents– Identification usage does not need URI
IUC27 – Berlin, Germany – April 2005

50

Internationalized Resource
Identifiers

IRI Usage

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

51

Internationalizing Web Addresses

Example Scenarios

IUC27 – Berlin, Germany – April 2005

52

Example Scenarios• http://www.w3.org/People/Dürst/• http://日本語.jp/Dürst/– Web Server using UTF-8• IRI: http://www.w3.org/People/Dürst/– The entire (UTF-8) string can be %hh
encoded• URI: http://www.w3.org/People/D%C3%BCrst/– Then the domain name mapping applied– Web Server using ISO 8859-1– Or the order can be reversed.

• URI:http://www.w3.org/People/D%FCrst/• Normalization sequence of domain (NFKC) vs IRI(NFC) is independent…

http://xn--wgv71a119e.jp/D%C3%BCrst/
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

53

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

9

IUC27 – Berlin, Germany – April 2005

54

Berlin, Germany, April 2005

Internationalizing Web Addresses
IRI Query, Fragment

Bidirectional IRI<scheme>://<authority><path>?<query>#<fragment>• Apply, except query, fragment– Backward compatibility limits the restrictions
that can be imposed• Use logical order– Rendering– The resource itself may impose restrictions:• Unicode bidirectional algorithm• E.g., query may be processed by cgi and a
database based on ISO-8859-1• Present as if embedded Left-To-Right– Host names• Fragment may reference a Japanese label in an
euc-jis resource• Labels should not mix LTR and RTL chars– If all components can be represented as utf8, then IRI. If any component is not, the URI.
Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

• Labels with RTL characters should start and end
with RTL characters
55

Internationalizing Web Addresses• The IRI is UTF-8 based, but the filesystem may not be.

• Martin Dürst created a patch for Apache.
– Encodings of files are named in .htaccessfile. Web server can then map IRI to filenameof each file.

– e.g. Unix/Linux file system is just bytes.
– File names are in the user’s locale/encoding.

www.w3.org/2003/06/mod_fileiri/– Therefore each web resource name may use
a different (user’s) encoding on disk.

www.w3.org/2003/Talks/0904-IUC-IRI/slide19-0.html• If Unix (or other) file system is UTF-8,
conversion is not needed.

• A different mapping may be required from
IRI path to each filename’s encoding.
IUC27 – Berlin, Germany – April 2005

• IIS and Apache 2 work as-is on Win 2000/XP
59

Internationalizing Web Addresses

Support

IUC27 – Berlin, Germany – April 200560

Detection• IDNA• How to detect IRI vs. URI?

– Mozilla, Opera, Safari;• Two possibilities:– IE only with plugin– Generally assume if byte pattern fits UTF-8,
it is likely UTF-8.

• IRI• Not reliable for short strings, esp. Chinese, or
when listing large numbers of URI as Yahoo!
does.

– IE, Mozilla can configure to use UTF-8– Opera and Safari– Convert address to escaped form both ways.
(UTF-8, native encoding). Do server
requests.

• IBM ICU, open source: uidna_• Verisign list of supporting productsInternationalizing Web Addresses

58

Example Solution
Apache mod_fileiri

IRI and Web Servers

Internationalizing Web Addresses

IUC27 – Berlin, Germany – April 2005

IUC27 – Berlin, Germany – April 2005

61

27th Internationalization and Unicode Conference

Internationalizing Web Addresses

10

IUC27 – Berlin, Germany – April 2005

62

Berlin, Germany, April 2005

Internationalizing Web Addresses
References• RFC 3490 IDNA

• RFC 3986 URI• RFC 3454 Stringprep• RFC 3987 IRI• RFC 3491 Nameprep• IDN and URI [PDF],
Michel Suignard• RFC 3492 Punycode

session://iuc27#Questions?

• Intro. to Multilingual
Web Addresses• www.dns.net/dnsrd/rfc/

Tex Texin
Internationalization Architect, Yahoo!

Internationalizing Web Addresses

27th Internationalization and Unicode Conference

11• W3C Character
Model, Resource
Identifiers• Numerous papers at
Unicode Conferences

IUC27 – Berlin, Germany – April 2005

64

Berlin, Germany, April 2005