Benchmark for 3.22

Between Koha 3.16 and 3.18, users faced some performance issues.
To understand, the code has been benchmarked, see bug 13690, especially the comment 2.
The 3.22 release is getting close and it would be great to know what users can expect in term of performance.
To get these values, the selenium tests on bug 13691 have been used.
= Raw data =
== v3.16.00 ==
CP main = 3.37 + 3.28 + 3.40 + 3.40 + 3.37 = 3.36
CP add patron category = 4.12 + 4.23 + 4.03 + 3.75 + 4.09 = 4.04
CP add patron = 4.15 + 4.19 + 4.22 + 4.46 + 4.17 = 4.24
CP add biblio = 1.00 + 1.30 + 0.84 + 0.79 + 0.87 = 0.96
CP add items = 13.89 + 12.80 + 12.68 + 12.69 + 12.47 = 12.91
CP checkout = 11.92 + 11.39 + 11.45 + 11.09 + 11.34 = 11.44
CP checkin = 9.74 + 9.65 + 10.01 + 8.97 + 9.04 = 9.48
== v3.18.00 ==
CP main = 4.64 + 4.65 + 4.69 + 4.79 + 4.61 = 4.68
CP add patron category = 6.06 + 5.96 + 6.75 + 6.26 + 5.99 = 6.20
CP add patron = 6.76 + 6.45 + 6.59 + 6.65 + 6.48 = 6.59
CP add biblio = 1.06 + 1.06 + 1.13 + 1.09 + 1.11 = 1.09
CP add items = 18.93 + 17.03 + 17.50 + 17.26 + 17.41 = 17.63
CP checkout = 16.48 + 16.80 + 15.52 + 15.20 + 15.42 = 15.88
CP checkin = 16.53 + 13.92 + 13.61 + 13.85 + 14.10 = 14.40
== v3.20.00 ==
CP main = 4.63 + 4.91 + 4.83 + 4.89 + 5.05 = 4.86
CP add patron category = 4.49 + 4.56 + 4.58 + 4.45 + 4.39 = 4.49
CP add patron = 6.03 + 5.88 + 5.90 + 5.94 + 5.71 = 5.89
CP add biblio = 2.17 + 1.03 + 0.86 + 1.24 + 0.98 = 1.26
CP add items = 13.86 + 13.55 + 13.37 + 13.56 + 13.38 = 13.54
CP checkout = 16.03 + 15.86 + 16.06 + 15.91 + 16.08 = 15.99
CP checkin = 10.22 + 9.66 + 10.18 + 10.03 + 10.07 = 10.03
== master (3.22 f25fe6dd Fri Now 6 11:21:56 2015) ==
CP main = 4.98 + 5.13 + 5.16 + 5.46 + 5.14 = 5.17
CP add patron category = 7.32 + 7.22 + 7.58 + 7.17 + 7.30 = 7.32
CP add patron = 7.39 + 7.13 + 7.36 + 7.38 + 7.11 = 7.27
CP add biblio = 1.41 + 1.14 + 0.91 + 0.94 + 1.11 = 1.10
CP add items = 19.63 + 20.15 + 20.61 + 19.94 + 20.30 = 20.13
CP checkout = 17.06 + 17.69 + 17.77 + 17.02 + 17.68 = 17.44
CP checkin = 16.13 + 15.90 + 14.95 + 15.29 + 15.74 = 15.60
== v3.16.00 under Plack ==
CP main = 2.63 + 2.21 + 2.67 + 2.55 + 2.18 = 2.45
CP add patron category = 2.01 + 1.91 + 1.98 + 1.82 + 2.05 = 1.95
CP add patron = 2.19 + 2.33 + 2.21 + 2.40 + 2.22 = 2.27
CP add biblio = 1.03 + 0.99 + 0.79 + 0.89 + 1.03 = 0.95
CP add items = 9.52 + 9.91 + 9.02 + 9.26 + 9.08 = 9.36
CP checkout = 6.25 + 6.12 + 6.44 + 6.13 + 6.08 = 6.20
CP checkin = 5.96 + 5.81 + 5.83 + 5.03 + 5.24 = 5.57
== v3.18.00 under Plack ==
CP main = 2.66 + 2.40 + 2.45 + 2.61 + 2.66 = 2.56
CP add patron category = 1.86 + 2.03 + 2.05 + 2.08 + 2.11 = 2.03
CP add patron = 2.85 + 2.39 + 2.43 + 2.50 + 2.49 = 2.53
CP add biblio = 0.96 + 1.05 + 0.78 + 0.92 + 0.83 = 0.91
CP add items = 10.58 + 9.93 + 10.56 + 9.70 + 9.87 = 10.13
CP checkout = 7.08 + 7.22 + 6.66 + 6.57 + 6.81 = 6.87
CP checkin = 6.16 + 6.25 + 5.81 + 6.02 + 6.18 = 6.08
== v3.20.00 under Plack ==
CP main = 3.33 + 2.66 + 2.52 + 2.61 + 2.43 = 2.71
CP add patron category = 2.87 + 2.74 + 2.49 + 2.76 + 2.54 = 2.68
CP add patron = 2.44 + 2.30 + 2.24 + 2.46 + 2.32 = 2.35
CP add biblio = 0.93 + 0.90 + 0.93 + 0.90 + 0.77 = 0.89
CP add items = 9.81 + 9.29 + 9.96 + 9.62 + 10.71 = 9.88
CP checkout = 7.21 + 6.84 + 6.93 + 6.67 + 6.77 = 6.88
CP checkin = 5.57 + 6.14 + 6.48 + 5.61 + 5.93 = 5.95
== master (3.22) under Plack ==
CP main = 3.99 + 2.84 + 3.21 + 3.14 + 3.41 = 3.32
CP add patron category = 3.06 + 3.02 + 3.74 + 3.55 + 3.32 = 3.34
CP add patron = 3.13 + 2.91 + 3.32 + 3.26 + 2.92 = 3.11
CP add biblio = 1.01 + 1.14 + 1.10 + 0.94 + 0.78 = 0.99
CP add items = 14.14 + 14.61 + 12.08 + 12.16 + 12.39 = 13.08
CP checkout = 10.00 + 8.99 + 8.98 + 8.72 + 9.71 = 9.28
CP checkin = 7.70 + 7.36 + 7.88 + 7.63 + 7.19 = 7.55
= In a table =
 benchmark (number of seconds per step, less is better) 
x 	3.16 	3.18 	3.20 	Master (3.22)	3.16 Plack	3.18 Plack	3.20 Plack	Master (3.22) Plack
main 	3.36 	4.68 	4.86 	5.17 	2.45 	2.56 	2.71 	3.32
add patron category	4.04 	6.2 	4.49 	7.32 	1.95 	2.03 	2.68 	3.34
add patron 	4.24 	6.59 	5.89 	7.27 	2.27 	2.53 	2.35 	3.11
add items 	12.91	17.63	13.54	20.13 	9.36 	10.13 	9.88 	13.08
checkout 	11.44	15.88	15.99	17.44 	6.2 	6.87 	6.88 	9.28
checkin 	9.48 	14.4 	14.4 	15.6 	5.57 	6.08 	5.95 	7.55
= And the handsome graph =
= Quick conclusion/interpretation =
* We are almost reaching the same performance with Plack than the 3.16, without Plack, which is a bit scary.
* The gain introduced in 3.20 because of is considerably lost in 3.22
* There is worrying gap between 3.20 and master for the "add items" step (to investigate...)
* The difference in the "add patron category" step is certainly caused by bug 14836 - Move the patron categories related code to Koha::PatronCategories (to confirm)
* ... ?
=Koha 3.8 on Raspberry Pi 1 vs. Koha 3.22 on Raspberry Pi 2=
This was done using benchmark_staff.pl. The 3.8 figures are taken from the tests I did mid-2012. The 3.22 figures were done with a self-built package of 3.22 the day after release.
Hardware:
*RPi1 had 1 CPU and 256MB RAM
*RPi2 has 4 CPU and 1024MB RAM
Results:
*Green means 3.22 with Plack is faster than 3.8 with Plack.
*Yellow means 3.22 with Plack is slower than 3.8 with Plack, but still much faster than without Plack. Compared to 3.8 with Plack we lose between 1/3 and 1/2 in performance.
*Red means 3.22 with Plack is much slower than 3.8 with Plack, and not much faster than 3.8 or 3.22 without Plack. Compared to 3.8 with Plack we lose almost everything.
Considering that the hardware used for the test of 3.22 is much better, this is even weirder. The 4 CPU cores of the RPi2 only get used in the last test, and there we have only about 2.5x performance gain with Plack left.
 	NoPlack 	NoPlack 	Plack 	Plack 	
 	RPi1 3.8	RPi2 3.22	RPi1 3.8	RPi2 3.22	
Step 1: staff client main page 	0.049 	0.076 	0.462 	0.582 	pages/sec
Step 2: catalog detail page 	0.045 	0.059 	0.523 	0.235 	biblios/sec
Step 3: catalogue search 	0.047 	0.055 	0.539 	0.086 	biblios/sec
Step 4: patron detail page 	0.049 	0.066 	0.518 	0.362 	borrowers/sec
Step 5: patron search page 	0.047 	0.068 	0.531 	0.386 	borrowers/sec
Step 6a circulation (checkouts)	0.049 	0.060 	0.478 	0.276 	checkouts/sec
Step 6b circulation (checkins) 	0.046 	0.069 	0.504 	0.390 	checkins/sec
all transactions at once 	0.014 	0,183 	0.522 	0.469 	operations/sec
Category:Development