How to be a CSI (encoding Crime Scene Investigator)

Objectives for Crime SceneInvestigation• Have some fun• Prevent death by bullet points• Introduce strategies for analyzingcharacter corruption• Compare TV Show and Software
investigation of mutilated character
corpses• The latest version of the presentation is at

How to be a CSI (encodingCrime Scene Investigator)
Tex Texin
Internationalization Architect

www.i18nguy.com/How-To-Be-A-CSI.pdf

Yahoo Inc.

How to be a CSI (encoding Crime Scene Investigator)

Our CSI Heroes
TV CSI vs. Software CSI

copyright © 2006 Tex Texin All rights reserved.

2

Our Software CSI Heroes

Gil Grissom
Catherine
Willows
Nick Stokes
Sara Sidle
Greg Sanders
Warrick Brown
Dr. Al Robbins
Capt. Brass

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference3

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

41

How to be a CSI (encoding Crime Scene Investigator)

Crime Discovery On TV

Crime Discovery in Software
A woman calls 911 yelling: “There has been aMojibake! Someone commited Mojibake!”*• First steps• Close off crime scene and gather clues– Take lots of odd pictures– Collect potential forensic evidence– Identify the victim(s) (or what’s left of them)– Interview witnesses

Charactershave died.

• They lie! They presume! They are prejudiced!

Decomposition
has already setin.

– Put down either the jaded or newbie cop

Who did it?

*Mojibake = Japanesefor garbage characters.

• If it’s suicide then how do you explain…
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

5Crime Discovery in Software

How to be a CSI (encoding Crime Scene Investigator)• Often included in bug
reports– Wrong characters– White or black boxes– Better than nothing– You can detect patterns– But often not helpful• Collect forensic evidence– Screen captures, Keystrokes, Environment• Better to collect:• Witnesses/Users: “It happens when I…”– Expected bytes/chars– Actual bytes/chars– Fonts referenced, available– Expected/detected encoding– Keystrokes– “The entire application is busted”– They lie, They presume, They’re prejudiced• The newbie or jaded developer“That vendor always does that. Our code is fine.”
copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference

6

Collecting Images• Characters Corrupted

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

• A way to reproduce
7

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

82

How to be a CSI (encoding Crime Scene Investigator)

Example Case History

Analysis On TV• Autopsy: More data collection• Possibilities: What could have happened?
• Conjecture: How could it have happened?
• Reconstruction: Does it fit the data?
• Research: Does other data fit the theory?

• Developers refused to debug based onan urban legend– “Crime happens. Can’t catch these guys.”– “It’s Microsoft IE’s fault- Our software is fine”• Instead of:– Always ask the most narrowing question last

<meta http-equiv=Content-Type content="text/html; charset=utf-8">• Experiments• The code said:– Measurements and proof of concept– Decay rates under abnormal circumstances<meta http-equiv=Content-Type content="text/html; charset=utf8">(Note the missing hyphen in “utf8”)
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

9

How to be a CSI (encoding Crime Scene Investigator)

Analysis on TV
AnatomyanddecayscienceAutopsy

10

Analysis in Software

Typical computer queries• How many trucks on the road?
• How many on route 1?
• How many on route 1 at 3pm?
• How many stopped at Joe’s bar?

Trajectory
Reconstruction

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

Coredumps,architectureanddebugging

Typical computer queries• Names of encodings• Relations between encodings• Variations of encodings• Quality of encoding conversions

Data flow
analysis
And points
ofconversion

Experiments with controls
copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference11

How to be a CSI (encoding Crime Scene Investigator)

Known data injection
copyright © 2006 Tex Texin All rights reserved.

123

How to be a CSI (encoding Crime Scene Investigator)

Analysis In Software

Character Encoding Negotiation• Architecture: The Web is complex– Clients, Browsers, mix technologies (DOM,java, css, php, active-x, applets, etc.), *ml
standard+non-standard behaviors, OSdependencies, etc.
– Servers, protocols are also mixed– Language, locale, encoding, are ambiguous– Negotiation tactics are unreliable

Unix user

Windows user

GB23121252

DOM Encoding
HTML Encoding

CSS
Encoding• But must be understood & kept up-to-date

JavaScriptEncoding

html

html

HTML DCSSOJava
Script M– See Web Internationalization Tutorial, et al.
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

13Architecture

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

Architecture• Data flow and points of (mis)conversion• Other contributors to conversion errors– Interfaces are common failure points– Missing or incorrect encoding identifier– False detection• Between Legacy, new and 3rd party software• API, web services, protocols, devices• Data sources, sinks• e.g. ASCII vs UTF-7• Too little data to detect• Text not like corpus used for detection statistics– Types of misconversion• Missing conversion Source ➽ Source• Incorrect conversion Source ➽ X• backwards conversion Target ➽ Source• double conversion (Source ➽ Target) ➽ Target
How to be a CSI (encoding Crime Scene Investigator)

14

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference– Encoding name variations• “I didn’t know it was a crime. I just did what I wastold!”

15

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

164

How to be a CSI (encoding Crime Scene Investigator)

Encoding Name Variations

Encoding Name Variations• E.g. ISO 8859-1 vs Windows-1252• E.g. ISO 8859-1 vs Windows-1252

80-9F Control Codes in ISO 8859-1

1252-specific characters
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

17

Encoding Naming Variations

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

18

Encoding Name Variations• Several variations of big-5, shift-jis, etc.

• Poor naming conventions in the industry• Application dependent– E.g. Microsoft, Apple added chars to shift-jis• Then there is big5-hkscs– Hong Kong Supplementary Character Set• Application dependent names– Windows IE expects “KSC5601” or “Korean”,
not cp949, windows-949– Firefox would expect euc-kr• Sometimes the error is font not encoding
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference

19

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

205

How to be a CSI (encoding Crime Scene Investigator)

Encoding Conversions

Missing Character Conversions• Correct conversion is not always obvious:– 0x5C is it a file separator “\” or currency “¥”?
?

• (or Won, or other currency)• Half-width or Full-Width?

– Sometimes you need different converters?

Converts to “?” withouterror code.
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

21Encoding Conversions– Unicode is an evolving standard– As characters are added the opportunity for
improved conversions exist– “Which Unicode version is the converter for?”– If I convert data today how will it comparewith the same data converted 3 years ago?
– How long after death before rigor mortis sets
in and when do the maggots come?
copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference

copyright © 2006 Tex Texin All rights reserved.

22

Breaking Multibyte Characters• The family secret: Versioning

How to be a CSI (encoding Crime Scene Investigator)How to be a CSI (encoding Crime Scene Investigator)

23

By not treating all bytes as one character• Don’t insert bytes in the middle• Don’t delete 1 byte of a multi-byte char.
• Be careful with block boundaries• Don’t treat individual bytes as characters– E.g. uppercase a trailing byte
– E.g. Treat trailing byte 0x5C as “\” in filepathnames
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

246

How to be a CSI (encoding Crime Scene Investigator)

Encoding Pathology andForensics

Making Mojibake
Splitting multi-byte characters

日Byte type:
Bytes:本• Piecing together who did what to whom

語

L T L T L T
9 F 9 7 8 E
3 A 6 B C A– Knowing what was expected and what
resulted, and comparing with typical patterns
of failure, we can deduce what occurred

Byte types:
L = Lead Byte
T = Trail Byte

• Do all characters fail or certain ones?
• How do they fail? Wrong characters,
Question marks, Blackboxes, etc.

Inserting “a” (61) in second byte

Byte type:
Bytes:

殿坙 {L T
9 63 1a

L T S L T
F 9 7 8 E
A 6 B C A

How to be a CSI (encoding Crime Scene Investigator)

語

copyright © 2006 Tex Texin All rights reserved.

25

Encoding Forensics andPathology

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

26

Encoding Forensics andPathology• Question Marks• Only certain characters misconverted– Conversion to an encoding that doesn’t havethe characters replaced by “?”.

– Variant encoding, out of date conversion,
data flow, incomplete font
– E.g. Only euro, trademark, smart quotes, OEligature,...
– Used iso-8859-1 to utf-8 conversion insteadof windows-1252 to utf-8• All non-Ascii characters are mojibake– Missing, wrong, backwards, or 2x conversion• Expected “ç”. Actual: “Ã§”• Expected 0xE7 (ç). Actual: 0xC3 0xA7 (Ã§)• Note that U+00E7 in UTF-8, “ç” is 0xC3 0XA7
• Conclusion: UTF-8 bytes displayed as ISO8859-1
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference27

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

28

7

How to be a CSI (encoding Crime Scene Investigator)

EncodingError Patterns

Encoding Research• How many trucks stopped at Joes bar?

• Comparing conversion error patterns is like
identifying the gun that fired a bullet by theshared striation marks• It’s encoding DNA• Often there are obvious candidate patterns• Look for variants you might be incurring– www.iana.org/assignments/character-sets– Conversions supported• See iconv, ICU, or doc for your converter– “Round up all the usual suspects”– A stoolie (er… tool) for testing conversions– Check for font versions/updates• Given an input string, try the usual and expected error
conversions, See if any of the results match– Other tools: Encoding validator, byte to %hh
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

29

Data Injection

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30

Setting Traps for Criminals• Testing with live data feeds• Use validation routines at key points ofdata acceptance or conversion– like being on a stakeout- Hoping the criminal
will come by while you wait

– Especially useful for UTF-8, which has illegal
byte patterns.
– Reference:www.w3.org/International/questions/qaforms-utf-8.html• Create a pseudo-feed or simulate data
entry for “controlled experiments”.
– Known, repeatable values simplify
debugging.
• Less threatening for debugging foreign languages– Inject directly into subsystems to eliminateother components as “suspects”.
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference31

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

32

8

How to be a CSI (encoding Crime Scene Investigator)

Conclusions: How to be a CSI

Thank You!
¿Questions?

• Gather forensic evidence• Be objective, be thorough• Understand the architecture, the dataflow, the points of conversion• Identify the potential patterns of failure• “Concentrate on what cannot lie. Theevidence” -Gil Grissom
How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

30th Internationalization and Unicode Conference

Note: Images from the CSI TV show and those of CSI cast members are courtesy of CBS Broadcasting Inc. All Rights
Reserved, and/or the actual copyright owners.
The owner and copyright for these photos were generally not documented from the sites I downloaded them. If notified
I will update this file to include the correct copyright.

33

How to be a CSI (encoding Crime Scene Investigator)

copyright © 2006 Tex Texin All rights reserved.

349