v1.8.5.1b
Better resource caching for multiple instances of Talismane in the same process: caching default models and lexicons at static level in language specific implementations (TalismaneEnglish, TalismaneFrench)
Added possibility to preload all resources on TalismangeConfig
Added possibility to indicate all files relative to a base dir in the configuration file (to be compatible with web implementations)
SentenceImpl: fixed small bug in getOriginalIndex if the index tested is just beyond the last original index in the sentence: was returning -1, now is correctly returning last original index + 1
TokenImpl: fixed bug where attributes were not initialized
FSocreCalculator: fixed bug where totals were not evaluated if only getKappa was called
Release 1.8.5.2b
TalismaneConfigImpl: added options tokeniserType to enable simple deterministic tokenisation, and preloadLexicon, to indicate whether or not the lexicon has to be loaded ahead of time
SimpleTokeniser: complete rewrite to be based entirely on filters
Token: changed attributes to a Map<String,String>, for storing attributes against tokens
TokenImpl: reads attributes from atomic parts when token has only a single non-whitespace part
TokenSequenceImpl: read attributes from placeholders
DiacriticRemover: new TokenFilter for removing all diacritics
LowercaseFilter: new TokenFilter for placing all text in lower case
TokenFilterServiceImpl: added ability to specify group index in parameters
TokenRegexFilter: added attributes, ability to set group index after instantiation
talismane_fr: new classes specifically for handling the SPMRL French corpus
Minor fix for Java 7 Scanner bug
TalismaneConfigImpl: added support for multiple token filter and text filter paths
talismane_core: made all Patterns use the UNICODE_CHARACTER_CLASS
TokenisedAtomicTokenSequenceImpl: fixed bug on transfer of attributes when inferring the token sequence
TokenFilterServiceImpl: any unknown parameter is assumed to be an attribute to be assigned to the token
Added external word lists
talismane_core/TokenRegexFilterImpl: \p{WordList(Name,true|false,true|false)} will replace contents of word list with a pipe delimited list of words in the regex
If first parameter is true, diacritics are optional (e.g. é is replaced by [ée])
If the second parameter is true, uppercase characters are optional (e.g. E is replaced by [Ee])
The regex should surround this by parentheses for it to work, e.g. \b(\p{WordList(FrenchNames,true,false)})\b
talismane_machine_learning/ExternalWordList - when read from a text file, should start with:
Type: WordList
Name: name
followed by the list of words
Release 1.8.6b - pulling changes for previous dev releases