RichardHightower/parsers-in-java — pull request #4 by slandelle (2013-12-02 22:16:05+00:00)
Title: Clean up + add my own benchmark test

My point is that my input is a byte[], not a char[]. I want to select the best tool that will handle it.

I ran `RunBytesAll` with the JVM options: `-server -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms512M -Xmx512M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly`

I took Boon lazy out of the test for now as the test would have to include fetching the values and protecting against memory leaks (from a user perspective).

Then, I'm wondering if the benchmark "framework" isn't a bit flawed in the sense that file get processed one after the other, meaning that the code gets JITed *a lot*.

I personally use scalameter (in Scala):

* it randomly picks up inputs from a list and aggregate them, in order to avoid JIT over-optimization.
* it can run multiple concurrent threads

Micro-benchmarking is such a difficult art...

I got the following results:
```
Name Time File Iterations
jackson-object 1 169 actionLabel.json 100 000
jackson-ast 693 actionLabel.json 100 000
json-smart 815 actionLabel.json 100 000
gson 984 actionLabel.json 100 000
boon 1 667 actionLabel.json 100 000
boon full 681 actionLabel.json 100 000
Winner: boon 1
___________________________________________________________________________________
Name Time File Iterations
jackson-object 1 110 citm_catalog.json 100
jackson-ast 781 citm_catalog.json 100
json-smart 883 citm_catalog.json 100
gson 735 citm_catalog.json 100
boon 1 1 227 citm_catalog.json 100
boon full 1 219 citm_catalog.json 100
Winner: gson
___________________________________________________________________________________
Name Time File Iterations
jackson-object 960 medium.json 100 000
jackson-ast 861 medium.json 100 000
json-smart 1 068 medium.json 100 000
gson 1 197 medium.json 100 000
boon 1 986 medium.json 100 000
boon full 1 021 medium.json 100 000
Winner: jackson-ast
___________________________________________________________________________________
Name Time File Iterations
jackson-object 218 menu.json 100 000
jackson-ast 179 menu.json 100 000
json-smart 214 menu.json 100 000
gson 312 menu.json 100 000
boon 1 189 menu.json 100 000
boon full 189 menu.json 100 000
Winner: jackson-ast
___________________________________________________________________________________
Name Time File Iterations
jackson-object 345 sgml.json 100 000
jackson-ast 297 sgml.json 100 000
json-smart 382 sgml.json 100 000
gson 465 sgml.json 100 000
boon 1 333 sgml.json 100 000
boon full 338 sgml.json 100 000
Winner: jackson-ast
___________________________________________________________________________________
Name Time File Iterations
jackson-object 61 small.json 100 000
jackson-ast 45 small.json 100 000
json-smart 60 small.json 100 000
gson 166 small.json 100 000
boon 1 36 small.json 100 000
boon full 45 small.json 100 000
Winner: boon 1
___________________________________________________________________________________
Name Time File Iterations
jackson-object 1 706 webxml.json 100 000
jackson-ast 1 561 webxml.json 100 000
json-smart 2 049 webxml.json 100 000
gson 2 327 webxml.json 100 000
boon 1 1 923 webxml.json 100 000
boon full 2 029 webxml.json 100 000
Winner: jackson-ast
___________________________________________________________________________________
Name Time File Iterations
jackson-object 378 widget.json 100 000
jackson-ast 329 widget.json 100 000
json-smart 471 widget.json 100 000
gson 515 widget.json 100 000
boon 1 375 widget.json 100 000
boon full 384 widget.json 100 000
Winner: jackson-ast
___________________________________________________________________________________
```