Release 1.8.4.1b
Sentence: Added possibility to store/retrieve start line number for each sentence (when reading from an annotated corpus with one token per line)
ParserRegexBasedCorpusReaderImpl: Added store/retrieve line number
LexiconFile: Added possibility to store multiple lexicon files in a single lexicon
LexiconSerializer: Read an entire directory in one fell swoop using DirectoryReader
default PosTagger/Tokeniser templates: changed to start indexing at 1 instead of 0
ParseFeatureTester: added line number to output
ParserRegexBasedCorpusReaderImpl: better performance when skipping sentences
TokenSearchFeature: added possibility to indicate skip criteria, which stop searching when a skip criterion is hit and start searching again when its matching resume criterion is hit
default token filters: fixed number filters to correctly handle +/- at start of number
AbstractFeatureParser: added a cache for all parsed features (parsedFeatures map), vastly improving performance when parsing complex features
DirectoryReader: fixed bug on attempting to close a reader twice
Release 1.8.4.2b
talismane_core/TalismaneConfig: new option - includeUnknownWordResults to create additional separate f-score files for words known and unknown in lexicon
talismane_core/PosTagOpenClassIndicator: added new class: PUNCTUATION, for special handling
talismane_core/PosTagRegexBasedCorpusReaderImpl: changed regex to reluctant from greedy to ease handling of optional placeholders
talismane_en: fixed punctuation tag class
talismane_fr: fixed punctuation tag class
talismane_lefff: fixed punctuation tag class
talismane_machine_learning: fixed IMPORTANT bug - was using descriptor instead of modifiedDescriptor in cache
talismane_other: addedPosTaggerStatistics for statistics on pos-tagger corpora
Release 1.8.4.3b
Added possibility to include multiple instances of Talismane in same thread (e.g. to handle multiple languages)
To this end, removed all ThreadLocal properties from TalismaneSession, and changed static methods to normal
TalismaneSession is now retrieved through TalismaneService
TalismaneServiceLocator is now instantiated with an arbitrary "sessionId" - e.g. "en" and "fr" to include one per language.
Fixed bug where the final unprocessed output is not written to the output file (e.g. closing XML tags)
Updated default French parser model to attach all punctuation systematicall to previous non-punct, and to coordinate as follows:
If conjuncts are separated by a comma, each conjunct is governed by the previous conjunct via the "coord" label
If conjuncts are separated by a conjunction, we use the "coord" from the prev conjunct to the conjunction, and "dep_coord" from the conjunction to the next conjunct
Added default French token_filters:
- to handle all numbers identically (except years)
- to handle all weekdays identically
- to handle all month names identically
- to recognise French phone numbers
This required us to changed PosTaggedTokenImpl.getLemmaForCoNLL to see if the original text has a lemma.
Release 1.8.5b
Identical to dev release 1.8.4.3b - merged with master
v1.8.5.1b
Better resource caching for multiple instances of Talismane in the same process: caching default models and lexicons at static level in language specific implementations (TalismaneEnglish, TalismaneFrench)
Added possibility to preload all resources on TalismangeConfig
Added possibility to indicate all files relative to a base dir in the configuration file (to be compatible with web implementations)
SentenceImpl: fixed small bug in getOriginalIndex if the index tested is just beyond the last original index in the sentence: was returning -1, now is correctly returning last original index + 1
TokenImpl: fixed bug where attributes were not initialized
FSocreCalculator: fixed bug where totals were not evaluated if only getKappa was called
Release 1.8.5.2b
TalismaneConfigImpl: added options tokeniserType to enable simple deterministic tokenisation, and preloadLexicon, to indicate whether or not the lexicon has to be loaded ahead of time
SimpleTokeniser: complete rewrite to be based entirely on filters
Token: changed attributes to a Map<String,String>, for storing attributes against tokens
TokenImpl: reads attributes from atomic parts when token has only a single non-whitespace part
TokenSequenceImpl: read attributes from placeholders
DiacriticRemover: new TokenFilter for removing all diacritics
LowercaseFilter: new TokenFilter for placing all text in lower case
TokenFilterServiceImpl: added ability to specify group index in parameters
TokenRegexFilter: added attributes, ability to set group index after instantiation
talismane_fr: new classes specifically for handling the SPMRL French corpus
Minor fix for Java 7 Scanner bug
TalismaneConfigImpl: added support for multiple token filter and text filter paths
talismane_core: made all Patterns use the UNICODE_CHARACTER_CLASS
TokenisedAtomicTokenSequenceImpl: fixed bug on transfer of attributes when inferring the token sequence
TokenFilterServiceImpl: any unknown parameter is assumed to be an attribute to be assigned to the token
Added external word lists
talismane_core/TokenRegexFilterImpl: \p{WordList(Name,true|false,true|false)} will replace contents of word list with a pipe delimited list of words in the regex
If first parameter is true, diacritics are optional (e.g. é is replaced by [ée])
If the second parameter is true, uppercase characters are optional (e.g. E is replaced by [Ee])
The regex should surround this by parentheses for it to work, e.g. \b(\p{WordList(FrenchNames,true,false)})\b
talismane_machine_learning/ExternalWordList - when read from a text file, should start with:
Type: WordList
Name: name
followed by the list of words
Release 1.8.6b - pulling changes for previous dev releases
Update Talismane
Merge branch 'original'