jbee/fly — Issue #3: Layer 0: Characters
Author: jbee
Created: 2012-03-05 18:59:53+00:00
State: open

### Step 1 : Classify a character
A character will be classified using `CharacterClass`es. A `CharacterClass` is a virtual set of characters that have something in common like letters or digits. A character can be a member of different classes at the same time. For example a dot `.` is a full stop punctuation mark as well as a floating point separator in a floating point number.

`CharacterClass`es are:

* deterministic : there is no state changing whether a character is a member or not

### Step 2 : Is it a changeover or terminator ?
A character is a _changeover_ if it is the beginning (first character) of a new token. A token is a sequence of characters make a single meaning or information like a word in natural language.
A character is a _terminator_ if it is a _changeover_ *and* the end of a part. A part is a sequence of tokens that make a single sense or has a single (set) kind of tokens.