initial commit
re-order functions. return list of tokens instead of just true/false
simplify getting best match
made get-best-match more readable
when two rules yield match of equal length, use first rule.
do not add :IGNORE tokens to list
converted to tail recursion. renamed process function to get-tokens. removed sample rules.
Merge branch 'master' of https://github.com/jhewlett/token.clj