Simplified tokenizer and a bit wider beams for decoder



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12549 94700074-3cef-4d97-a70e-9c8c206c02f5
Ensure utf-8 output on serialization



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12550 94700074-3cef-4d97-a70e-9c8c206c02f5
Don't complain on Java 8 javadoc



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12551 94700074-3cef-4d97-a70e-9c8c206c02f5
Better way to silence Java 8



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12552 94700074-3cef-4d97-a70e-9c8c206c02f5
More universal text aligner to handle arbitrary texts



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12553 94700074-3cef-4d97-a70e-9c8c206c02f5
Ignore ranges based on density



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12554 94700074-3cef-4d97-a70e-9c8c206c02f5
Fix minor issues.

git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12555 94700074-3cef-4d97-a70e-9c8c206c02f5
Properly display alignment results on iterations, properly align short chunks



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12556 94700074-3cef-4d97-a70e-9c8c206c02f5
Fixes tests



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12557 94700074-3cef-4d97-a70e-9c8c206c02f5
Output debug information through logger



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12558 94700074-3cef-4d97-a70e-9c8c206c02f5
Last stdout bit



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12559 94700074-3cef-4d97-a70e-9c8c206c02f5
get token order from linguist



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12560 94700074-3cef-4d97-a70e-9c8c206c02f5
No need for a separate package



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12561 94700074-3cef-4d97-a70e-9c8c206c02f5
Removes empty folder



git-svn-id: svn+ssh://svn.code.sf.net/p/cmusphinx/code/trunk/sphinx4@12562 94700074-3cef-4d97-a70e-9c8c206c02f5
Import tokenizer from FreeTTS.
Tokenization test.
Tokenization test.