Release 1.3.1b
New possibility: parserRules (via the new parserRules argument).
Language-specific implementations now extend TalismaneConfig instead of AbstractTalismane: Refactoring of AbstractTalismane into a non-abstract TalismaneImpl class, and TalismaneConfig into an abstract class.
New parameter: suffix - automatic addition of a suffix to the basename.
New output type for parser evaluation: freemarker template output for the best guess
Fixed bug on RegexLexicalEntryReader to match other corpus readers in format.
Fixed bug on undescores in CoNLL format: Added token.getTextForCoNLL() and posTaggedToken.getLemmaForCoNLL() to simplify CoNLL output templates, and changed readers to use CoNLLFormatter.fromCoNLL method.
Fixed bug on ParserFeatureParser that didn't wrap generic features in ParseConfigurationFeatures
Fixed bug on PosTagSequenceImpl.toString() not getting reset
Fixed bug on missing words at start of sentence: AbstractTokenSequence.cloneTokenSequence now does deep clone of listWithWhiteSpace
Added Token.getLexicalEntry(PosTag) to enable tokeniser features to make use of lexicon data
Added tokeniser feature type TokenAddressFunction returning another token in the sequence
Added tokeniser features: BackwardSearchFeature, ForwardSearchFeature, CurrentPatternIndex, LemmaForPosTagFeature, TokenReferenceFeature (to be able to make use of TokenAddressFunctions)
Performance fix: add caching to TokeniserContext
Bug fix: when tranforming a descriptor to one descriptor per pattern, ensures consistency across line (instead of cross-product of patterns for each pattern-dependent function)
Added generic feature: ToStringNoNulls
Bug fix: using (?m)^ to indicate start of line or start of file in filters
Release 1.3.2b
talismane_fr: Added default parser rules to keep punctuation from cluttering the beam
talismane_trainer_fr: Updated French tokeniser features to make use of new available tokeniser features
talismane_fr: Created new French tokeniser model.
Release 1.3.3b
New command: compare, to compare two annotated corpora (one ref, one eval) and output standard evaluation output files - currently for parser only
Fixed bug: pos-tagger beam not working properly (results on all beams identical) - changed PosTaggerContextImpl to use cache correctly
New argument: logStats - to store the total time in a file
New argument: includeDistanceFScores - when evaluating the parser, include a file giving accuracy per arc attachment distance
New argument: evaluationFile - for "compare" command, indicating the file to be evaluated (inFile gives the ref corpus)
New argument: posTaggerFeatures - to override the built-in features of the pos-tagger model, useful for a quick test of new features in the log file
New class: ParseComparator, ParseComparatorImpl
Fixed ParseEvaluationFScoreCalculator to give identical results on compare/evaluate modes, by handling an unlabeled root attachment as "NoHead"
CSVFormatter.getCsvSeparator() - updated everywhere to allow for ; separators (in French locale)
New class: ParserFScoreCalculatorByDistance - to calculate accuracy at each attachment distance
New class: PosTaggerGuessTemplateWriter - to write the "best guess" of the pos-tagger to an output file using a freemarker template
Made TaggedToken extend TokenWrapper, so that  PosTaggedTokenFeatures can be mixed freely with TokenFeatures
Corrected "bug" in AndRangeFeature and OrRangeFeature, which were expecting absolute indexes, but getting relative indexes
talismane_fr: added dependency on talismane_ftb, and two new parameters: corpusReader=ftb and treebankDir=??? to read XML French Treebank files directly for pos-tagger evaluation
talismane_fr: updated pos-tagger model to use new set of features
FunctionDescriptorParserImpl: fixed to only escape the backslash when followed by a double-quote or backslash
Release 1.4.0
Added command=compare for pos-tagger output
Added LexiconChain, LexiconFile, LexiconSerializer and LexiconDeserializer for simple handling of multiple lexicons
Changed RegexMarkerFilter to apply to the entire regex by default (rather than to the last group - Not backwards compatible)
Added option evaluationPatternFile/evaluationPattern for command=compare, to allow different pattern in ref file and evaluation file
Added option labeledEvaluation to allow labaled/unlabeled evaluation of parse output
Bug: TalismaneConfig: Fixed several filter bugs
Bug: RollingSentenceProcessor: fixed bug when log-level = trace
Bug: ParseEvaluationFScoreCalculator: updated to compare indices instead of equality on pos-tagged tokens, for "noHead" and "wrongHead"
Bug: ParserEvaluatorImpl: remove root before evaluating token sequence
Bug: PosTagEvaluationFScoreCalculator: correctly compare empty/null tokens
Bug: AbstractTokenSequence: wasn't removing emtpy token from listWithWhiteSpace
Added PosTagSequenceFilters and posTaggerPostProcessingFilters for automatic removal of empty tokens
Moved sentenceCount to corpus readers and anlayze command, thus generalizing its applicability
ParserRegexBasedCorpusReader: implement PosTaggerAnnotatedCorpusReader and TokeniserAnnotatedCorpusReader to simplify reading of parsed corpora at all levels
ParserRegexBasedCorpusReaderImpl: added token filters, over-ridable readWord method for strange token encoding conventions
New tokeniser feature: LexiconPosTagForStringFeature (get lexicon pos-tags for any string)
Created separate lexicons for original Lefff, Talismane additions, and additions specific to the P+D tag
TalismaneFrench: allow option on whether to use compound postags (P+D, P+PRO) or separate tags with empty tokens
TalismaneFrench: trained new tokeniser, pos-tagger and parser models (latter two for P+D tagset)
talismane_ftb: allow conversion of empty tokens to P+D compound tags
talismane_ftbDep/FtbDepReader: possibility of keeping P+D compound tags
talismane_lefff: change LefffEntry to LexicalEntry wherever possible
Bug: talismane_machine_learning/FeatureResultImpl: only encode feature result when writing training name, otherwise return original result
talismane_trainer_fr: new posTagger features
talismane_trainer_fr: new "number in all letters" token filter
talismane_trainer_fr: when training, option to use multiple lexicons and compound postags (P+D)
Release 1.4.1b
New parameters: tokeniserBeamWidth, posTaggerBeamWidth, parserBeamWidth, propagateTokeniserBeam, to separate tokeniser beam/propagation from the rest
talismane_core/TalismaneImpl: changed tokeniser to use a SentenceDetectorAnnotatedCorpusReader
talismane_core/SentencePerLineCorpusReader: new class reading one sentence per line
talismane_core/TalismaneConfig: updated to read defaults from Scanners instead of InputStreams, so that character encoding can be set by sub-classes
talismane_core/TokeniserEvaluator: updated to use new TokenEvaluationObserver
talismane_fr/lexicons: updated the lefff additions lexicons
talismane_fr/TalismaneFrench: updated to return scanners with corrected charset
talismane_machine_learning/LinearSVMDecisionMaker: fixed bug when none of the features describing a given context are represented in the model
talismane_trainer_fr: new pattern list for French
talismane_trainer_fr/TokeniserMaxentRunner: updated to use lexicon chains
talismane_ftb/FrenchTreebankTokenReader: fixed bug on null empty tokens not getting removed
talismane_ftb/FtbPosTagMapperImpl: fixed bug when missing a postag mapping in the mapper
Release 1.4.2b
Bugfix: filters which extend beyond the block size are not allowed (could cause the start of the filter to be applied but not the end, and vice versa)
New parameter: blockSize (to increase the block size if really big filters are really required)
Bugfix (major): talismane_core/DuplicateWhiteSpaceFilter - duplicate white space was getting removed, instead of being replaced by a single white space
talismane_core/TextMarker: added getSource and getMatchTest for better logging
Bugfix: talismane_core/PatternTokeniserImpl: escaping separators in default patterns
talismane_fr: added underscore to default "IS_NOT_SEPARATOR" list
Release 1.4.3b
talismane_core: Changed default CoNLL format to respect morphology column guidelines
talismane_core: Allow reading of annotated parsed corpora without attempting to predict transitions
talismane_core: Added TokenSequence.isWithRoot, to permit correct processing of index-based methods (e.g. LowercaseFirstWordFrenchFilter) with and without an initial root
talismane_core: Created AllUppercaseFrnehFilter to convert any all-uppercase word to lowercase when in lexicon
talismane_fr: Several new entries in lefff-2.1-additions.txt for missing French closed-class words
talismane_fr: Fixed lefffCC.zip to include thousands of missing lefff entries (accidentally removed in last release)
talismane_fr: : trained new models for tokeniser, pos-tagger and parser for updated lexicon entries and tokeniser patterns
talismane_fr: Fixed URL filter (token_filters.txt) to be slightly more permissive
talismane_fr: Several new compound words permitted in tokeniserPatterns_fr.txt
talismane_trainer_fr: added Main class to enable direct calling of trainers using the java -jar command-line option
talismane_utils: fixed Java 7 bug in StringUtils.java
Created talismane_standoff package to convert to-and-from CoNLL format to Brat standoff format
Release 1.4.4b
New option minFreeMemory, giving the minimum amount of free memory, in kilobytes, to begin the next parsing step - for avoiding an OutOfMemory exception (maxParseAnalysisTime was supposed to be enough, but there was another reported case of an OutOfMemoryException)
StandoffReader corrected to read randomly ordered standoff format (as output by Brat)
Release 1.4.5b
talismane_core: Fixed OutOfMemoryError in AllUppercaseFrenchFilter (for uppercase words like AAAEEEAAAEEEAAAEEE)
talismane_fr: Better handling for punctuation in posTaggerConstraints_fr - able to handle tokens like ".........." and "!?!?"
Release 1.4.6b
talismane_machine_learning: Major new functionality: StringCollectionFeature which allows multiple feature results at runtime (one per collection result)
talismane_core: New option: logConfigFile to override the default log4j.properties
talismane_core: fixed bug in LexiconChain and LexiconFile which gave random order for postags of a given token
talismane_core: all features now get a RuntimeEnvironment which can store variables set at initialisation
talismane_machine_learing: AbstractCacheableFeature and HasFeatureCache now include the runtime environment
talismane_core: massive improvement in tokeniser speed using new StringCollectionFeature functionality: for each token, we only check the patterns it actually matches (rather than all patterns)
talismane_core: new tokeniser features: PosTagSetFeature, TokeniserPatternsFeature, TokeniserPatternsAndIndexesFeature, TokenIndexFeature
talismane_fr: removed unused ADJWH tag
talismane_fr: trained new tokeniser & pos-tagger models based on bug-fixes and new functionality
talismane_machine_learning: AbstractFeatureParser: added abstract injectDependencies method to allow injection feature dependencies after construction
talismane_machine_learning: OpenNLPDecisionMaker, OpenNLPEventStream, MaxentDetailedAnalysisWriter, LinearSVMDecisionMaker, LinearSVMModelTrainer: added StringCollectionFeature support in training and decision making
Release 1.5.0b
Major behind-the-scenes update to feature management, clearing the way to Colletion features and Variables stored at feature initialisation
Note: feature descriptors are not backwards compatible with Release 1.4.x
In particular, TokenAddressFeatures has been brought in as an optional parameter for all tokeniser features, and PosTaggedTokenAddressFunction as a required parameter for all pos-tagged token features
talismane_trainer_fr: The feature files posTagger_fr.txt and tokeniser_fr.txt have been rewritten for this update
talismane_fr: The models have been retrained to use the new feature descriptors
Release 1.5.1b
Added options performanceConfigFile, crossValidationSize, includeIndex, excludeIndex
Removed options logPerformance, posTaggerFeatures
Rewrote PerformanceMonitor to be based on config file (similar to Log4J)
Added ExternalResource (to machine learning models + features), and created TextFileResource
Added generic ExternalResourceFeature which can get information from an ExternalResource, given a set of keys to query it
Added feature groups (Feature, AbstractFeature, FunctionDescriptorParserImpl, AbstractFeatureParser), which allow us to group many features together and automatically add new features based on the entire group (e.g. to create feature combinations)
Fixed bug on negative parser rules (Transition needs to implement Comparable)
Added cross validation to ParserRegexBasedCorpusReader
Added ExternalResources directly into models, with ability to retrieve them
in lefff-2.1-additions.txt, gave all numbers the same lemma: #NUM#
Fixed bug on lefff entry order (étaient=etayer, faut=faillir)
Fixed bug where AbstractCacheableFeature was re-checking features many times if they returned null
FScoreCalculator: added functionality to automatically combine cross-validation results
Release 1.5.2b
talismane_fr: Fixed tokeniser features to use new StringCollection rules (String collection result only automatically concatenated if top-level feature is not a string feature)
talismane_fr: retrained all models to use fixed LefffCC lexicon, and to train based on the full corpus (not training portion only)
Release 1.5.3b
Include non-breaking space with space in tokeniser.
Added and updated documentation