Hackerpilot/Dscanner — pull request #27 by blackwhale (2013-02-07 20:44:37+00:00)
Title: Avoid allocation happiness in StringCache + some minor tweaks

Summary of changes:
Avoid allocation happiness in StringCache.
Optimized isWhite and drop extra test in lexWhiteSpace
Added nice s-box driven hash function, noticeble improvement (still StringCache alone) especially with bigger pow-2 hash table.

Also I switched to -O2 switch of ldc2, as it makes no difference if it's bigger for me.

In this pull I tried to keep changes cosmetic and isolated, as not to break anything. In the end for me the difference is:
```
dmitry@dmitry-VirtualBox ~/Dscanner $ avgtime -q -r 200 ./dscanner --tokenCount phobos/std/datetime.d
------------------------
Total time (ms): 6819.77
Repetitions : 200
Sample mode : 33 (98 ocurrences)
Median time : 33.887
Avg time : 34.0988
Std dev. : 2.2451
Minimum : 32.261
Maximum : 62.195
95% conf.int. : [29.6985, 38.4991] e = 4.40031
99% conf.int. : [28.3158, 39.8818] e = 5.78299
EstimatedAvg95%: [33.7877, 34.41] e = 0.311149
EstimatedAvg99%: [33.6899, 34.5077] e = 0.408919

dmitry@dmitry-VirtualBox ~/Dscanner $ avgtime -q -r 200 ./old-dscanner --tokenCount phobos/std/datetime.d

------------------------
Total time (ms): 7411.82
Repetitions : 200
Sample mode : 36 (104 ocurrences)
Median time : 36.8205
Avg time : 37.0591
Std dev. : 1.45338
Minimum : 35.348
Maximum : 47.514
95% conf.int. : [34.2105, 39.9077] e = 2.84857
99% conf.int. : [33.3154, 40.8028] e = 3.74366
EstimatedAvg95%: [36.8577, 37.2605] e = 0.201424
EstimatedAvg99%: [36.7944, 37.3238] e = 0.264716

```
I call that almost 10% improvement.

Yet there is one big design bottleneck that needs to be addressed - buffering chars with keepChar and only _later_ deciding if you want to carve up a token out of it with setToken(!). It should be the other way around - e.g. inside of lexComment there got to be a single if that picks the function to use (delegate).

Even better add a compile time parameter to lexComment and such that means keep date for this kind of token. Then everywhere e.g. lexComment is called do something like this:

if(config.iterStyle & IterationStyle.includeComments)
lexComment!(Lexem.keep)();
else
lexComment!(Lexem.drop)();

I'd rather let you re-organize the code yourself to keep a single coherent view of it.
Also there must be a way to check if (my) optimizations produce correct lexer, for now I'm checking the number of tokens but that's hardly good indicator. I need to make a precies dump of some sorts - which options is that?

The other trick I'm about to do later (once I again have some time) is to use one single buffer both for contents of string buffer *and* the current accumulator (the current buffer is a accumulator for future token strictly speaking). This is going to be envasive but it removes the need to even copy string (I removed .idup but it's not enough) to put it into string cache. If only setTokenValue were used more consistently everywhere...

Other then this there are realms of seemingly small-scale double-work (and maybe even tripple) in the cirtical paths of lexer but let's deal with the major things first.

Hope this helps.