Honey, My Unicode Disk Storage Went into the Circular File

Honey, My Unicode Data Disk Went
into the Circular File!

• This session will present some of the difficulties ofproviding a common international interface to fileservices on different operating systems.
• Although Unicode supports all the necessary
characters, identifying the set of characters that arelegitimate on any OS can be difficult, and rules forcase-insensitivity, normalization, etc. vary, and may
evenee varya y by useuser.
• The presentation describes the problem space and a
proposal for standardizing filename conventions.

Tex Texin
Xen Master, XenCraft
Copyright 2009 Tex Texin. All rights reserved.

IUC 33, Oct. 2009

A simple request: file interchange• Physical/Direct Access- Memory stick- Zip File, tar, archives• Backup and Restore- Restore precisely the names that are backed up• Network or Web access (FTP, Http, etc.)• Logical- “Type in this name: ‘xyz.txt’ ”- It’s written on this napkin

Copyright 2009 Tex Texin. All rights reserved.

Abstract3

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Copyright 2009 Tex Texin. All rights reserved.

2IUC 33, Oct. 2009

Naming requirement: interoperability• Operating systems• Platforms (Java,(Java .Net,Net etcetc.))• Protocols (CIFS, NFS, AFS, etc.)• Devices (many disk types, sizes, etc.)• Pseudo/Virtual devices (clusters, etc)
• Storage Architectures (FAT, LFN, etc.)
S stem Architecture
Architect re (Big/Little Endian- System
Endian, 8/16/32 etcetc.))- Archive formats (tape, zip, tar, et al)• Networks (SAN, NAS, etc.)• Locales (Language, location)
Copyright 2009 Tex Texin. All rights reserved.

4IUC 33, Oct. 2009Page 1

Honey, My Unicode Disk Storage Went into the Circular File

User locale affect’s name

File system bingo• HFS, FAT16, HPFS, ISO 9660:1988, JFS1,日本語(Shift-JIS)

日本語(EUC-JIS)\x93\xFA\x96\x7B\x8C\xEA\xC6\xFC\xCB\xDC\xB8\xEC

Copyright 2009 Tex Texin. All rights reserved.

5

IUC 33, Oct. 2009

NTFS, Joliet (CDFS),NTFS(CDFS) QFS,QFS UDF,UDF FAT32
FAT32,GPFS, HFS+, NSS, ISO 9660:1999, Lustre,GFS, zFS, FATX, UFS2, OCFS, VMFS2,Fossil, Google File System, ZFS, OCFS2,VMFS3, GFS2, exFAT, Btrfs, HAMMER, andmanyothersa y otesSee wiki:
Comparison_of_file_systems
Copyright 2009 Tex Texin. All rights reserved.

Typical variations• Filename length bytes--Byte-oriented file systems

entert andd recognize
i names as charactersht

256, 65,534, unlimited, undefined• Valid byte-oriented names:-IUC 33, Oct. 2009

• Although filenames are in bytes, users

8, 88.3,3, 16,6, 6
64,, 236,36, 255,55, 256,56, 5510,0, 5512,, 4032
03

• Pathname length bytes-6

A-Z,0-9, .
A-Z, 0-9, CTRL characters
Any byte except :Any byte except NUL
ASCIIDOS, Windows, OS/2 restrictions \ / : ? * “ > < | NUL
All but“ . / \ [ ] : + | < > = ; , * ? and space

Copyright 2009 Tex Texin. All rights reserved.

7IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

• Encoding is often ill-defined and notrecorded with the filenames- Can vary with user locale
locale, not system widewide.
- User locale can vary with time, or application.

Copyright 2009 Tex Texin. All rights reserved.

8IUC 33, Oct. 2009

Page 2

Honey, My Unicode Disk Storage Went into the Circular FileUnicode-oriented filenames• Valid names on Unicode-based systems- Any
A UCS-2
UCS 2- Any UTF-16- Any Unicode except NUL /- Additional exceptions for U+FFFF, U+FFFE- Unicode version 3.2- UnicodeU i d version
i 3.2,3 2 minusilongllist
li t off restricted
ti t dcharacters- All UCS-2 except * / \ ; : ?

9IUC 33, Oct. 2009

Legacy coexistence• Systems offer multiple namingconventionstiffor bback
k compatibility
tibilitt-BBDE~1.txt
PROGRA~1
PROGRA~2- After restore (destabilizing…)• Program Files• Program Data

1011

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

IUC 33, Oct. 2009

It is a huge automated heterogeneous world• Heterogeneous- Multiple, smart devices• Automation creates

demand for IDs thatreflect human entity
names- Alerts, transactions via
web, etc.
- Generated filenames toddescribe
ib ttransaction
tiPROGRA~2
PROGRA~3

No relation between long and short name
Copyright 2009 Tex Texin. All rights reserved.

Copyright 2009 Tex Texin. All rights reserved.

(phone, pda, laptop,camera, smart printers,etc.)- Networks, clouds- Archives- E.g. LFN (long file name) and 8.3• t€-ß.txt• Program Files• Program Data

• Case-sensitivity- (based(bd on locale,ll some are ASCII-only)
ASCII l )• Case-preservation• Normalization (none, NFC, NFD)
• Search and wildcard behavior- CIFS ? = 1 character,, except
p at the end of afilename where it is 0 or 1 characters.
• "??x" equals "abx" but not "abcx" or "ax".
• "x??" matches "xab", "xa" and "x", but not "xabc"• What about full-width * / \ ; : ? ?
Copyright 2009 Tex Texin. All rights reserved.

More variations

Huge: Terabytes, Petabytes, Exabytes…
Copyright 2009 Tex Texin. All rights reserved.

12IUC 33, Oct. 2009

Page 3

Honey, My Unicode Disk Storage Went into the Circular File

Problem cases• Valid characters for names- Differences
Diff
iin restrictedt i t d andd remappedd charsh• Case-insensitive User1 describes afilename to Case-sensitive User2• No system-wide character encoding- Locale1 User gives filename to Locale2 User
- E.g. Apache Web Server module: mod_fileiri• Maps utf-8 filename to locale encoding• Generated names need full character set- Customer-first-last-address-etc.doc
Copyright 2009 Tex Texin. All rights reserved.

13IUC 33, Oct. 2009

Problem cases• Unicode evolution (version 3, 4, 5, …)- When new characters usable by all file services?
• Correct spelling for bidi names• Search rules- Dir tèxìnß.txt• Match Case, accent, width, ß=SS

• Conflict prevention- Cat > tExìnSS.txt• Rendering names as characters- Need encoding information- Is it in fact a textual name?
Copyright 2009 Tex Texin. All rights reserved.

14

IUC 33, Oct. 2009

Problem cases• Program languages want to represent

filenames as strings- implying characters- Modern languages use Unicode strings• Python PEP 383 solution- Map bytes 0x80-0xFF to U+DC80..U+DCFF- Using surrogates is clearly a Unicode violation

Wake Up Call!
The industry must head off even more
proprietary solutions being developed
Copyright 2009 Tex Texin. All rights reserved.

15

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Copyright 2009 Tex Texin. All rights reserved.

16

IUC 33, Oct. 2009

Page 4

Honey, My Unicode Disk Storage Went into the Circular File

Similar to other problems
Let’s reuse solutions• Program Identifiers UAX #31- Not
N t as complete
l t as hhuman id
identifierstifi- Tailored map (Natural->Program) possible• Domain Names- Yes, Perhaps basis for standards building on
Unicode 3.2 and fixed lists of restricted andremappedd charactersht• URI/IRI- Has good flexibility in using #hh when bytes are
not UTF-8 characters
Copyright 2009 Tex Texin. All rights reserved.

17IUC 33, Oct. 2009

Proposal – Change in orientation
Higher level abstraction• Stop thinking in terms ofbytes• Allows Unicodealgorithms- But provide back compatibility,- e.g. regular expressions,migration• Define conventions for“noun phrase” space- Natural language, automatedname generation• Maximal utility but

reduced ambiguity(for search large filespaces)• Plan for evolution (fornew versions)• (Ignores multiple nameand encodingdetection)- Napkin passing, security- Normalization

Copyright 2009 Tex Texin. All rights reserved.

19

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Proposals• File Systems, Unicode, and Normalization• David
D id R
Robinson,bi
Ienup
ISung,
SNicolasNi l WilliWilliams• Sun Microsystems, Inc. (IUC29)• Provides good details for migration to Unicode

Copyright 2009 Tex Texin. All rights reserved.

18

IUC 33, Oct. 2009

W3C reference processing model• Logically,characters areUnicode- Specifications in termsof Unicode characters- Implementations do

NOT have to use
Unicode, only behave
as if they did

Copyright 2009 Tex Texin. All rights reserved.

• Benefits- Removes ambiguity,simplifies specifications- Allows flexibility forcommon localencodings- Backward compatible- Large character set forinternational- Removes orientation on
byte values
20

IUC 33, Oct. 2009Page 5

Honey, My Unicode Disk Storage Went into the Circular FileW3C reference processing model

In/outAny encoding on thewireCSS

Any encoding forinternal implementation

Copyright 2009 Tex Texin. All rights reserved.

XML21Interoperable
Name Convention

Any UTF-* on thewire/for interchange

HTMLAbstraction layer using
unicodeFile system reference processing model

IUC 33, Oct. 2009

Interoperable name abstraction• Recommend a Unicode (sub)set- PickPi k a versioni (5(5.1)
1)- Determine allowed, disallowed characters• Provide a property or a publicly available table• Tailoring UAX #31 program identifiers

CIFS

Abstraction layer using
unicodeNFS

Any encoding forinternal implementation

Copyright 2009 Tex Texin. All rights reserved.

HFS22Interoperable name abstraction

AllAllowstoolsl tomigrate names(examine #hh indifferent encodings)

My NaturalLanguage FileNameUnicode name+#hh#hh#hh- Recommend bidi spelling, etc.

• Disallowed characters can be accepted

but then names are not interoperable

IUC 33, Oct. 2009

UAX #31 Tailored
File Name(But 1-1 reversible)

Legacy name withinvalid chars orbytes- Systems can provide warnings• Escapes, ala #hh, for invalid names
Copyright 2009 Tex Texin. All rights reserved.

23

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Copyright 2009 Tex Texin. All rights reserved.

24

IUC 33, Oct. 2009

Page 6

Honey, My Unicode Disk Storage Went into the Circular File

Interoperable name abstraction• Support capability negotiation- Label or query version/supported characters- Use escapes for mismatch• Dynamic version upgrade where possible• Recommendations- Lengths for commonality- 8-bit
8 bit ACE/Punycode
ACE/Pd maps ffor shorth t names- Locale-invariant, precise match rules

Wild and crazy• Benefits drive new version support- Needed forf Hong Kong and newer Unicodecharacters• Define characters to allow a syntax

without legacy conflicts- Root,, path
separator,pp, typeyp separator,p, parent,p, self
• (equivalent to /, / or \, ., .., .)- Escape (U+) followed by code points.

• For insensitivity, conflict resolution- Big-endian, composed for interchange
Copyright 2009 Tex Texin. All rights reserved.

25

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

26

IUC 33, Oct. 2009

Summary names• Change how we think about names- BytesB tnever againi• Have a high level abstraction• Drive evolution to match Unicode• Drive legacy migration to meetabstraction• Standardize name conventions to reduce
conflicts
Copyright 2009 Tex Texin. All rights reserved.

27

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Wake Up Call!
The industry must head off even more
proprietary solutions being developed
Copyright 2009 Tex Texin. All rights reserved.

28

IUC 33, Oct. 2009

Page 7

Honey, My Unicode Disk Storage Went into the Circular File

Questions

Copyright 2009 Tex Texin. All rights reserved.

29

IUC 33, Oct. 2009

Copyright 2009 Tex Texin. All rights reserved.

Page 8