Fixed preprocessing
Fixed unknown in FSM, supporting multiple head symbols + Removed default symbol options
Fix some trace out messsage to include source in lookuptable fsm
Included save_src_str config into lookuptable-fsm
Added a tokenizer
Added test for the Penn Tokenizer
Merge branch 'master' of github.com:neubig/travatar
Made the tokenizer executable
Merge pull request #17 from philip30/master

Bunch of fixes in LookupTable FSM
Fixed memory access in FSM LookupTable
Merge pull request #18 from philip30/master

Fixed memory access in FSM LookupTable
Engineered the tokenizer
Updated training guide to use Ckylark
Added advance interpolation measurement
Memory updates to words
Halfway through refactoring lookup-table-fsm
Fixed bugs and memory leaks
Changed to dynamic calculation to save memory
Merge pull request #19 from philip30/master

Added Advance Interpolation Evaluation Measurement
Added information about the calculation score