owlcs/owlapi — pull request #216 by sesuncedu (2014-06-26 00:47:30+00:00)
Title: Custom tokenizer for OWL FSS

This is a version of the OWL FSS parser that uses a hand written tokenizer.

The tokenizer overgenerates slightly- it does not check for invalid code-points in putative IRIs. Since the existing codebase does not do much IRI validation anyway , this is not a major loss, especially since the expanded strings are immediately converted into IRIs objects, which do some code-point validation already.

Some benchmarking (using Gene Ontology)

Options | Runtime
---------- | ------------
LOOKAHEAD=2,JAVA_UNICODE_ESCAPE,KEEP_LINE_COLUMN | 4.61s
LOOKAHEAD=1,JAVA_UNICODE_ESCAPE,KEEP_LINE_COLUMN | 3.9869s
LOOKAHEAD=1,UNICODE_INPUT,KEEP_LINE_COLUMN | 3.65s
LOOKAHEAD=1,UNICODE_INPUT | 3.18s
LOOKAHEAD=1,UNICODE_INPUT,USER_TOKEN_MANAGER | 1.98s
LOOKAHEAD=1,UNICODE_INPUT,USER_TOKEN_MANAGER,KEEP_LINE_COLUMN | 2.00s

The first round of tuning went from 4.61s to 3.65s ; the custom tokenizer brings it down to 2.00s

There's room for improvement (55MB in 2 seconds is still not great) , but it's ok for now.