akka/akka — pull request #1381 by patriknw (2013-04-26 14:39:40+00:00)
Title: Prepare Cluster StressSpec for heavy testing, see #2787

This makes it possible for us to start the real testing of large and long running clusters.

* Enable usage of MultiJvm nrOfNodes in cluster StressSpec, see #2787
* Adjustments to StressSpec for testing large clusters
* Performance improvement of mute deadLetters
* Quick fix for unreachable exiting, see #3266

This review should also include to verify the configuration of the jenkins job that I have added. It is using a0, a2, a3, a4, a5, a6. It is confugred with following jvm arguments

-XX:+UseCompressedOops -XX:MaxPermSize=384M -Xms256m -Xmx1g -Xss2m -Djava.security.egd=file:/dev/./urandom -Dakka.test.multi-node.java=/usr/lib/jvm/java-7-openjdk-amd64/bin/java -XX:ReservedCodeCacheSize=96M -Dmultinode.XX:+UseParallelGC -Dmultinode.XX:+UseNUMA -Dmultinode.XX:+UseCondCardMark -Dmultinode.XX:-UseBiasedLocking -Dmultinode.XX:+UseCompressedOops -Dmultinode.Xms128M -Dmultinode.Xmx256M

-Dakka.remote.netty.tcp.maximum-frame-size=256000b -DMultiJvm.akka.cluster.Stress.nrOfNodes=100 -Dmultinode.Dakka.test.cluster-stress-spec.nr-of-nodes-factor=8 -Dmultinode.Dakka.test.cluster-stress-spec.duration-factor=1 -Dmultinode.Dakka.test.cluster-stress-spec.idle-gossip-duration=120s -Dmultinode.Dakka.test.cluster-stress-spec.exercise-actors=on -Dmultinode.Dakka.test.cluster-stress-spec.infolog=on -Dakka.test.timefactor=1 -Dmultinode.Dakka.test.cluster-stress-spec.expected-test-duration=3600s -Dsbt.ivy.home=/localhome/jenkinsakka/.ivy2 -Dsbt.override.build.repos=true -Dakka.publish.credentials=/home/jenkinsakka/.sbt/moxie-nexus.credentials -Dakka.test.multi-node=true -Dakka.test.multi-node.targetDirName=/localhome/jenkinsakka/akka-cluster-stress1

The summary results for a cluster of 100 nodes running on 6 machines (144 cores in total):

[JVM-1] [INFO] [04/26/2013 16:20:16.265] [StressSpec-akka.actor.default-dispatcher-10] [akka://StressSpec/user/resultHistory] Cluster result history
[JVM-1] [Title]	[Duration (ms)]	[ClusterStats(gossip, merge, same, newer, older)]
[JVM-1] join seed nodes	15301	ClusterStats(376,4,254,50,68)
[JVM-1] join one to 19 nodes cluster	8819	ClusterStats(245,0,197,11,37)
[JVM-1] join one to 20 nodes cluster	7218	ClusterStats(237,0,181,17,39)
[JVM-1] join one to 21 nodes cluster	8866	ClusterStats(302,0,245,16,41)
[JVM-1] join one to 22 nodes cluster	8821	ClusterStats(317,0,254,20,43)
[JVM-1] join one to 23 nodes cluster	11220	ClusterStats(366,0,303,18,45)
[JVM-1] join one to 24 nodes cluster	8820	ClusterStats(330,0,269,14,47)
[JVM-1] join one to 25 nodes cluster	7217	ClusterStats(319,0,252,18,49)
[JVM-1] join one to 26 nodes cluster	11218	ClusterStats(419,0,351,17,51)
[JVM-1] join one to 27 nodes cluster	8818	ClusterStats(352,0,277,22,53)
[JVM-1] join one to 28 nodes cluster	8018	ClusterStats(361,0,282,24,55)
[JVM-1] join one to 29 nodes cluster	10463	ClusterStats(452,0,368,27,57)
[JVM-1] join one to 30 nodes cluster	8819	ClusterStats(445,0,362,24,59)
[JVM-1] join one to 31 nodes cluster	8815	ClusterStats(440,0,354,25,61)
[JVM-1] join one to 32 nodes cluster	9655	ClusterStats(499,0,406,30,63)
[JVM-1] join one to 33 nodes cluster	9621	ClusterStats(485,0,397,23,65)
[JVM-1] join one to 34 nodes cluster	10418	ClusterStats(541,0,446,28,67)
[JVM-1] join 16 to one node, in 35 nodes cluster	11222	ClusterStats(789,0,625,65,99)
[JVM-1] join 33 to seed nodes, in 51 nodes cluster	12023	ClusterStats(1564,18,1133,157,256)
[JVM-1] join one to 84 nodes cluster	12027	ClusterStats(1601,0,1368,66,167)
[JVM-1] join one to 85 nodes cluster	12023	ClusterStats(1618,0,1386,63,169)
[JVM-1] join one to 86 nodes cluster	12023	ClusterStats(1604,0,1366,67,171)
[JVM-1] join one to 87 nodes cluster	13629	ClusterStats(1782,0,1539,70,173)
[JVM-1] join one to 88 nodes cluster	12832	ClusterStats(1691,0,1439,77,175)
[JVM-1] join one to 89 nodes cluster	13625	ClusterStats(1795,0,1553,65,177)
[JVM-1] join one to 90 nodes cluster	12823	ClusterStats(1727,0,1462,86,179)
[JVM-1] join one to 91 nodes cluster	12822	ClusterStats(1725,0,1462,82,181)
[JVM-1] join one to 92 nodes cluster	11221	ClusterStats(1653,0,1382,88,183)
[JVM-1] join one to 93 nodes cluster	14430	ClusterStats(2005,0,1747,73,185)
[JVM-1] join one to 94 nodes cluster	13624	ClusterStats(1878,0,1612,79,187)
[JVM-1] join one to 95 nodes cluster	13629	ClusterStats(1864,0,1600,75,189)
[JVM-1] join one to 96 nodes cluster	12824	ClusterStats(1839,0,1561,87,191)
[JVM-1] join one to 97 nodes cluster	13627	ClusterStats(2010,0,1748,69,193)
[JVM-1] join one to 98 nodes cluster	12026	ClusterStats(1843,0,1565,83,195)
[JVM-1] join one to 99 nodes cluster	13682	ClusterStats(2018,0,1752,69,197)
[JVM-1] exercise join/remove round 1	13651	ClusterStats(2295,12,1879,151,253)
[JVM-1] idle gossip	120014	ClusterStats(12539,0,12381,61,97)
[JVM-1] remove one from 100 nodes cluster	21633	ClusterStats(3162,0,2725,142,295)
[JVM-1] remove one from 99 nodes cluster	22432	ClusterStats(3422,0,2985,145,292)
[JVM-1] remove one from 98 nodes cluster	19231	ClusterStats(3044,0,2609,146,289)
[JVM-1] remove one from 97 nodes cluster	22433	ClusterStats(3255,0,2822,147,286)
[JVM-1] remove one from 96 nodes cluster	20825	ClusterStats(2995,0,2583,130,282)
[JVM-1] remove one from 95 nodes cluster	23234	ClusterStats(3282,0,2882,120,280)
[JVM-1] remove one from 94 nodes cluster	22437	ClusterStats(3277,0,2878,122,277)
[JVM-1] remove one from 93 nodes cluster	20833	ClusterStats(2984,0,2588,123,273)
[JVM-1] remove one from 92 nodes cluster	18428	ClusterStats(2681,0,2284,126,271)
[JVM-1] remove one from 91 nodes cluster	22433	ClusterStats(3062,0,2659,135,268)
[JVM-1] remove one from 90 nodes cluster	21629	ClusterStats(2950,0,2560,125,265)
[JVM-1] remove one from 89 nodes cluster	17628	ClusterStats(2574,0,2175,138,261)
[JVM-1] remove one from 88 nodes cluster	20030	ClusterStats(2758,0,2363,136,259)
[JVM-1] remove one from 87 nodes cluster	19231	ClusterStats(2655,0,2271,128,256)
[JVM-1] remove one from 86 nodes cluster	20833	ClusterStats(2772,0,2396,123,253)
[JVM-1] remove one from 85 nodes cluster	20834	ClusterStats(2796,0,2428,118,250)
[JVM-1] shutdown one from 84 nodes cluster	8016	ClusterStats(826,9,716,16,85)
[JVM-1] shutdown one from 83 nodes cluster	11217	ClusterStats(947,6,714,67,160)
[JVM-1] shutdown one from 82 nodes cluster	8016	ClusterStats(989,10,819,44,116)
[JVM-1] shutdown one from 81 nodes cluster	8014	ClusterStats(823,11,687,27,98)
[JVM-1] shutdown one from 80 nodes cluster	8813	ClusterStats(842,7,717,29,89)
[JVM-1] shutdown one from 79 nodes cluster	10420	ClusterStats(736,5,610,21,100)
[JVM-1] shutdown one from 78 nodes cluster	7214	ClusterStats(823,5,590,76,152)
[JVM-1] shutdown one from 77 nodes cluster	8014	ClusterStats(820,8,674,36,102)
[JVM-1] shutdown one from 76 nodes cluster	8015	ClusterStats(858,9,710,35,104)
[JVM-1] shutdown one from 75 nodes cluster	7214	ClusterStats(820,8,649,41,122)
[JVM-1] shutdown one from 74 nodes cluster	8818	ClusterStats(884,8,730,36,110)
[JVM-1] shutdown one from 73 nodes cluster	8363	ClusterStats(789,9,674,28,78)
[JVM-1] shutdown one from 72 nodes cluster	7214	ClusterStats(819,13,620,52,134)
[JVM-1] shutdown one from 71 nodes cluster	7216	ClusterStats(769,5,628,32,104)
[JVM-1] shutdown one from 70 nodes cluster	10418	ClusterStats(674,7,556,16,95)
[JVM-1] shutdown one from 69 nodes cluster	8014	ClusterStats(775,8,641,27,99)
[JVM-1] leave 16 in 68 nodes cluster	20828	ClusterStats(1837,111,1309,105,312)
[JVM-1] shutdown 16 in 52 nodes cluster	12822	ClusterStats(721,144,359,38,180)
[JVM-1] shutdown one from 36 nodes cluster	8814	ClusterStats(352,12,267,15,58)
[JVM-1] shutdown one from 35 nodes cluster	8012	ClusterStats(328,5,234,24,65)
[JVM-1] shutdown one from 34 nodes cluster	9616	ClusterStats(308,0,228,20,60)
[JVM-1] shutdown one from 33 nodes cluster	7213	ClusterStats(319,7,244,19,49)
[JVM-1] shutdown one from 32 nodes cluster	7213	ClusterStats(290,4,217,16,53)
[JVM-1] shutdown one from 31 nodes cluster	8816	ClusterStats(278,3,211,16,48)
[JVM-1] shutdown one from 30 nodes cluster	7213	ClusterStats(302,6,236,15,45)
[JVM-1] shutdown one from 29 nodes cluster	8013	ClusterStats(299,4,243,12,40)
[JVM-1] remove one from 28 nodes cluster	14426	ClusterStats(654,2,521,42,89)
[JVM-1] remove one from 27 nodes cluster	14430	ClusterStats(607,0,490,41,76)
[JVM-1] remove one from 26 nodes cluster	13621	ClusterStats(567,0,457,37,73)
[JVM-1] remove one from 25 nodes cluster	14421	ClusterStats(562,0,460,32,70)
[JVM-1] remove one from 24 nodes cluster	14418	ClusterStats(509,0,412,30,67)
[JVM-1] remove one from 23 nodes cluster	14423	ClusterStats(486,0,390,32,64)
[JVM-1] remove one from 22 nodes cluster	14420	ClusterStats(461,0,371,29,61)

The full log can be seen here: https://jenkins.akka.io:8498/job/akka-cluster-stress1/34/consoleText