rhdunn/cainteoir-engine — Issue #21: create a FSA/RegEx compiler framework and use it in the document parsers
Author: rhdunn
Created: 2012-02-24 14:10:20+00:00
State: open

Currently, the document parsers (specifically the xml and rtf parsers/tokenizers) are hand-written. These work, but we can do better.

I want to use regular expressions for handling the dictionary word and rule matching engine.

Also, there are advanced things like identifying Chapters/Parts/Sections/Paragraphs/Authors in plain text documents and handling the different pipelines (e.g. iterating over words or utf-8 characters).

The basic pipeline is:

RE => IR => optimize => FSA => IR => optimize => ( machine code | interpreter )

RE : Regular Expression
IR : Intermediate Representation / Intermediate Language
FSA : Finite State Automata

The advantages of this are:
1/ there is no hand-written code (specifically lexer machinery) dealing directly with strings/string matching [*];
2/ all string processing code goes through the same pipeline (code reuse);
3/ extra verification/coverage of the RE/FSA pipleline;
4/ easier to maintain;
5/ take advantage of the machine code machinery -- fast;
6/ optimizations made to the RE/FSA pipeline improve performance of the document parsers.

[*] There may be hand-constructed FSA or RE/IR objects.