Hackerpilot/Dscanner — pull request #31 by blackwhale (2013-02-22 21:49:15+00:00)
Title: First steps towards leaner common path in Lexer.advance

Okay all things being equal Lexer.advance is the hotest spot (~30% not inclusive when fully inlined).

This shrugs off few extra steps that repeat work already done elsewhere. Some measly 2-3% of speed gained.

In the same vane advance should only check for whitespace and comments exactly once to completely avoid filtering out these tokens later.

Another trick is to integrate the *first* check for whitespace into the giant switch. Same goes for 2 EOF chars, they'd better be in this switch too, this way we save on 2 cmp/jmp ops on each loop iteration. The end of input as in "end of the array" still has to be checked though ;)

The additional benefit of integerating first byte "detection" into one huge switch would be having UTF-8check (& 0x80) done exactly once here (not counting the ones later in some lexXYZ).

I'm not sure I'll be able to get more twaeks done during this weekend so I just listed things I think worth looking into.