Release 1.6.4b
Fixed compound pattern tokeniser to function correctly with overlapping patterns
talismane_core: created AnnotatedCorpusReader interface with built-in cross-validation
talismane_core: added token probabilities to built-in template "with_prob"
talismane_core/Token: added getProbability, to get probs from atomic decisions
talismane_core: created TokenEvaluationCorpusWriter, to write out the errors in an annotated corpus in a way that's easy to correct
talismane_core/TokenEvaluationFScoreCalculator: added possibility to generate CSV file of f-scores (to simplify combination/analysis of cross-validation results)
talismane_core/TokeniserOutcome: changed outcome names to JOIN, SEPARATE
talismane_core/PatternMatchWordFormFeature: fixed to leave out \b at start or end
talismane_core/CompoundPatternEventStream: pattern must be consistent, otherwise report it as the default outcome for the full pattern (to handle overlaps)
talismane_core/CompoundPatternTokeniser: major update to handle multiple patterns starting on the same token via simple statistical combinations
talismane_core/TokenPatternImpl: allow \b to match start and end of sentence, allow character ranges (e.g. [A-Z]), correct upper-case options when first pattern part inside exception, correct method for determining which indexes to test further, remove underscore from name
talismane_core/TokenPatternMatchSequenceImpl: handle null tokens for \b at start & end of sentence
talismane_core/TokeniserPatternManager: add method getDefaultOutcomes, to centralise functionality for getting the default outcomes of a TokenSequence
talismane_core/TokeniserPatternManagerImpl: implementation of above
talisamne_core/tests/TokenPatternImplTest: added new tests for pattern matching bugs
talismane_fr/lexicons: some minor additions to custom lexicon
talismane_fr/TalismaneFrench: allowing pos-tag map in ftb token reader, changed default tokeniser model
talismane_fr/token_filters: changed "number in letters" to "deux" instead of "999", to avoid tokenising "1982 deux" as a single token
talismane_ftb/FrenchTreebankTokenReader: fixed bugs on empty token not getting removed
talismane_trainer_fr/tokeniserPatterns_compound_fr: many updates to correct the tokeniser patterns
talismane_trainer_fr/token_filters: same "number in letters" fix as above, added fix for 1999-2000
talismane_trainer_fr/TokeniserMaxentRunner: added cross-validation functionality
talismane_utils/CSVFormatter: changed from static methods to class, to allow formatter to remember number of decimal places - updates on all client classes as well