change the arg of viterbi_search, which has a arg of function called score_function(), which uses score_it() in current test. But in future use, a method of any scoring model can be passed in, as long as this score function takes (bigram, incoming_char) as parameters.
fix a bug in best_index_seq generation, where the "j" was originally not updated!
implement lattice2segmentation model. It takes lexicon and a trained (maxent) model to score (bigram, incoming_character) as inputs to segment raw sentences.
It uses Viterbi algorithm (viterbi_search.py) to score each segmentation hypothesis by summing the log score of distinct word bigrams. The individule score is given by the Maxent model, the feature for which is generated by feature_gen.py using (bigram, incoming character).