added whitespace tokenizer with more standard package names and documentation
Merge branch 'master' into exp

* master:
  update readme
fix package name, toying with one state machine implementation
moved functions and vars around
Merge branch 'master' into exp

* master:
  link to exp branch and added ignore
  remove old file, update exp branch
setting up the punkt and lexer options for tokenizer
more comments and attribution
more fake code for the nlp token lexer
adding logic functions to tmachine tokenizer
added unicode code point support; can build alphabets via unicode ranges
added cjk ideograms, next to start on yi syllables
moved unicode alphabet stuff to alphabet.go; two main functions are SimpleAlphabet and UnicodeAlphabet
rename Cyrillic struct to CyrillicSimple
function names consistent now
small update
correct value for basic latin unicode set
update comments; take out functions in tmachine
minor tweaks
in the middle of porting Rob Pike's lexical scanning to a natural language tokenizer, unicodes are const not var.
lexer unicode dev up and running