first lines of code. It includes the java-based lexer (./ruled-lexer), a lexing interface to plug different lexers (./nlp-utils/ling), a generic break detector using the provided lexer (./nlp-utils/break-detection) and a specialization for Zedai (./zedai-break-detection)
add the html-break-detection module + allow one to identify word and sentences with an XML attribute in addition to the XML element it belongs to (e.g. span elements with role attribute) + improve the merge step to make a clear difference between mergeable and non-mergeable elements + fix the merge step so it can perform cascadable merges + fix a bug in the merge that caused some parts of the text nodes to be removed
prevent the lexing from adding tmp namespace declarations everywhere + properly remove the 'mergeable' attribute + handle processing instructions in the content document
add dtbook-break-detection module + add functions to undo the lexing (unwrap words) + change the way the sentences are tagged: they now carry an id and a new document is produced that contains all the ids
remove unnecessary logging step
make sure nothing is discarded when the words/sentences are created
update guava
* Add a light lexer with no word segmentation and a simplified sentence segmentation.
* Use code-points regexs to make the ruled-lexer work with java 6
* Change LexService interface to allow the break detector to choose between multiple lexers
* Remove the maven dependency between the break-detector to the ruled-lexer
* Add some languages
* Brand new XML Rebuilder (from the Lexer's results) that takes care of creating the words and sentences in their final location instead of doing post-swap and post-merge in XSLT. Get rid of the mergeable attribute and the list of elements equivalent to periods.
* JUnit tests for the XML Rebuilder.
* Deal with elements with unique ID.
* Add a new phase in the main xpl to split the the content around the skippable elements.
* Deal with possible null segments in the LightLexer and in the RuledLexer (not tested with big files).
* The lexers do not take as input a list of segments anymore. They take a single regular string instead ;
* Add a registry of lexers. The break step now retrieves the languages and maps them to lexers ;
* Deal with pre-existing sentences in XSLT (to be tested thoroughly) ;
* Bind the rule-based-lexer with a new SentenceDetector specialized for European languages ;
* Fix a bug in the xml rebuilding (see test #hard8) ;
* Simplify a bit the XmlBreakRebuilder.
* Use util.Locale instead of our own list of languages
* Fix a big in the timeout when registrying new lexer
* Add a list of 'ignored-elements' to prevent the lexer from adding word/sentences where it is forbidden by the standard
* Add OmnilangLexer, a generic lexer that uses the Java's BreakIterator.
* Transform the former comma-equivalent and space-equivalent into ensure-word-before/after and ensure-sentence-before/after to be more flexible.
* Add a new option to allow only a subset of the standard elements to be parent of subsentences
* Deal with the cases of "!)", "?)" etc. in the lexers
* Set the split-around-skippable to TRUE by default for the dtbook
rename xproc libraries
* Option to forbid the algorithm to split up any node.
* Trimming in the StringComposer, including the lexing marks. Trimming no longer necessary in the Lexers.
* New tests regarding the trimming.
* Match rules for abbreviations
* French lists of abbreviations in the resources
* Fixed a bug in the prefix matcher for unicode strings
* End-of-sentence detection when there are quotes around
* More tests (Categorizer, PrefixMatcher and RuleBasedLexer)
* Properly read the resources in the JUnit tests
* Simpler way to deal with forbidden/allowed positions of the new sentences and new words;
* Complete customization for DTBook;
* Removed the possibility for the abbreviations to not ending with a dot;
* Possibility to recycle some elements when creating sentences or splitting around the skippable elements;
* Minor improvements for the French categorizer.
* The main sentence detector locates more accurately the delimiters (e.g. the sentence enclosed in quotes), for it uses a list of patterns to match the end of sentences on the top of what the lexer returns;
* The 'tmp' namespace is discarded from the output;
* The latin words are no longer categorized, otherwise we would have to add a rule to ensure they are followed by a white space or a punctution mark;
* Some rules have been changed to make sure the words categorized as numbers or dates are followed by a proper character;
* Used xsl:key when possible.
* Silent text fragments (punctuations, spaces etc.) are no longer encapsulated into individual spans when they come after a skippable elements;
* New end-of-sentence pattern;
* The cursor to find the end of sentences is moved forward when negative patterns are found.
* New category of lexeme ;
* Escaped some dots in the end-of-sentence regex patterns ;
* The word-layer is no longer discarded by when the words' parents are tmp:sentences.
* Refactored the XProc part in order to easily plug tests ;
* Fixed a bug in the detection of duplicated nodes ;
* Misc improvements of the create-valid and split-around-skippable scripts ;
* Some new JUnit tests of the Java-based break detection.