Release 1.8.4.1b
Sentence: Added possibility to store/retrieve start line number for each sentence (when reading from an annotated corpus with one token per line)
ParserRegexBasedCorpusReaderImpl: Added store/retrieve line number
LexiconFile: Added possibility to store multiple lexicon files in a single lexicon
LexiconSerializer: Read an entire directory in one fell swoop using DirectoryReader
default PosTagger/Tokeniser templates: changed to start indexing at 1 instead of 0
ParseFeatureTester: added line number to output
ParserRegexBasedCorpusReaderImpl: better performance when skipping sentences
TokenSearchFeature: added possibility to indicate skip criteria, which stop searching when a skip criterion is hit and start searching again when its matching resume criterion is hit
default token filters: fixed number filters to correctly handle +/- at start of number
AbstractFeatureParser: added a cache for all parsed features (parsedFeatures map), vastly improving performance when parsing complex features
DirectoryReader: fixed bug on attempting to close a reader twice
Release 1.8.4.2b
talismane_core/TalismaneConfig: new option - includeUnknownWordResults to create additional separate f-score files for words known and unknown in lexicon
talismane_core/PosTagOpenClassIndicator: added new class: PUNCTUATION, for special handling
talismane_core/PosTagRegexBasedCorpusReaderImpl: changed regex to reluctant from greedy to ease handling of optional placeholders
talismane_en: fixed punctuation tag class
talismane_fr: fixed punctuation tag class
talismane_lefff: fixed punctuation tag class
talismane_machine_learning: fixed IMPORTANT bug - was using descriptor instead of modifiedDescriptor in cache
talismane_other: addedPosTaggerStatistics for statistics on pos-tagger corpora
Release 1.8.4.3b
Added possibility to include multiple instances of Talismane in same thread (e.g. to handle multiple languages)
To this end, removed all ThreadLocal properties from TalismaneSession, and changed static methods to normal
TalismaneSession is now retrieved through TalismaneService
TalismaneServiceLocator is now instantiated with an arbitrary "sessionId" - e.g. "en" and "fr" to include one per language.
Fixed bug where the final unprocessed output is not written to the output file (e.g. closing XML tags)
Updated default French parser model to attach all punctuation systematicall to previous non-punct, and to coordinate as follows:
If conjuncts are separated by a comma, each conjunct is governed by the previous conjunct via the "coord" label
If conjuncts are separated by a conjunction, we use the "coord" from the prev conjunct to the conjunction, and "dep_coord" from the conjunction to the next conjunct
Added default French token_filters:
- to handle all numbers identically (except years)
- to handle all weekdays identically
- to handle all month names identically
- to recognise French phone numbers
This required us to changed PosTaggedTokenImpl.getLemmaForCoNLL to see if the original text has a lemma.
Release 1.8.5b
Identical to dev release 1.8.4.3b - merged with master