mbsimonovic/jepsen-couchbase — pull request #2 by daschl (2014-08-14 09:09:04+00:00)
Title: Move gets call into try/catch so that its timeout exception is also caught

without this change, the exception that is raised get swallowed in nowhere (I think in the same nirvana stdout and err are lost due to a logger config setting?).

When this change is applied, the code keeps on going and the other ops will succeed once the partition is gone:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:61775,suspend=y,server=n -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/michael/tmp/jepsen-couchbase/target/classes:/Users/michael/.m2/repository/com/couchbase/client/couchbase-client/1.4.3/couchbase-client-1.4.3.jar:/Users/michael/.m2/repository/io/netty/netty/3.5.5.Final/netty-3.5.5.Final.jar:/Users/michael/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/michael/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/michael/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/michael/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore-nio/4.3/httpcore-nio-4.3.jar:/Users/michael/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/michael/.m2/repository/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/michael/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/michael/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/michael/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Applications/IntelliJ IDEA 13.app/lib/idea_rt.jar" com.milansimonovic.jepsen.CouchTimeoutTest
Connected to the target VM, address: '127.0.0.1:61775', transport: 'socket'
2014-08-14 11:02:47,408 INFO [com.milansimonovic.jepsen.RemoteConnection] - 192.168.56.101
2014-08-14 11:02:47,771 INFO [com.milansimonovic.jepsen.CouchbaseClientFactory] - CouchbaseClientFactory{192.168.56.101, default, }
2014-08-14 11:02:47,908 INFO [com.couchbase.client.vbucket.provider.BucketConfigurationProvider] - Could bootstrap through carrier publication.
2014-08-14 11:02:47,920 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 11:02:47,923 INFO [com.couchbase.client.CouchbaseClient] - CouchbaseConnectionFactory{bucket='default', nodes=[http://192.168.56.101:8091/pools], order=RANDOM, opTimeout=10000, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=1000, obsTimeout=10000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-08-14 11:02:47,956 INFO [com.couchbase.client.CouchbaseClient] - viewmode property isn't defined. Setting viewmode to production mode
2014-08-14 11:02:47,956 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:02:48,970 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:48,970 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
counting down 943596453
counting down 99276090
2014-08-14 11:02:49,042 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 1 in 61
counting down 1076537872
2014-08-14 11:02:49,048 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 2 in 6
2014-08-14 11:02:49,054 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 3 in 6
counting down 1413217636
2014-08-14 11:02:49,060 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 4 in 6
counting down 491904845
2014-08-14 11:02:49,066 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 5 in 6
counting down 1312922328
2014-08-14 11:02:49,072 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 6 in 6
counting down 1529791274
2014-08-14 11:02:49,078 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 7 in 6
counting down 1349895670
2014-08-14 11:02:49,083 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 8 in 5
counting down 1129169666
2014-08-14 11:02:49,089 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 9 in 6
counting down 26740457
counting down 746758755
2014-08-14 11:02:49,094 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 10 in 5
counting down 1386645616
2014-08-14 11:02:49,100 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 11 in 5
2014-08-14 11:02:49,106 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 12 in 6
counting down 2145256693
2014-08-14 11:02:49,111 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 13 in 5
counting down 1051781801
2014-08-14 11:02:49,117 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 14 in 6
counting down 119068326
2014-08-14 11:02:49,123 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 15 in 6
counting down 1637559101
2014-08-14 11:02:49,128 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 16 in 5
counting down 1861050967
2014-08-14 11:02:49,134 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 17 in 6
counting down 1462839425
2014-08-14 11:02:49,140 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 18 in 6
counting down 383609127
2014-08-14 11:02:49,147 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 19 in 6
counting down 774334978
2014-08-14 11:02:49,156 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 20 in 8
counting down 735922763
2014-08-14 11:02:49,162 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 21 in 6
counting down 1970674321
2014-08-14 11:02:49,170 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 22 in 7
counting down 136298825
2014-08-14 11:02:49,177 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 23 in 7
counting down 494783794
2014-08-14 11:02:49,185 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 24 in 8
counting down 1546867652
2014-08-14 11:02:49,192 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 25 in 7
counting down 433354817
2014-08-14 11:02:49,199 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 26 in 7
counting down 1865938875
2014-08-14 11:02:49,207 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 27 in 8
counting down 483439410
2014-08-14 11:02:49,214 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 28 in 7
counting down 1513848351
2014-08-14 11:02:49,221 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 29 in 7
counting down 1612120937
2014-08-14 11:02:49,228 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 30 in 7
counting down 810400511
2014-08-14 11:02:49,235 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 31 in 7
counting down 689821415
2014-08-14 11:02:49,242 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 32 in 7
counting down 685219421
2014-08-14 11:02:49,248 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 33 in 6
counting down 1887689829
counting down 759433928
2014-08-14 11:02:49,254 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 34 in 6
2014-08-14 11:02:49,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 35 in 5
counting down 1781091177
2014-08-14 11:02:49,266 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 36 in 7
counting down 2123891975
2014-08-14 11:02:49,271 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 37 in 5
counting down 278306373
2014-08-14 11:02:49,278 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 38 in 6
counting down 495568085
2014-08-14 11:02:49,284 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 39 in 6
counting down 1029420558
2014-08-14 11:02:49,290 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 40 in 5
counting down 55913631
2014-08-14 11:02:49,296 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 41 in 6
counting down 664141318
2014-08-14 11:02:49,302 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 42 in 6
counting down 1953048909
2014-08-14 11:02:49,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 43 in 5
counting down 350362120
counting down 1048178143
2014-08-14 11:02:49,313 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 44 in 6
2014-08-14 11:02:49,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 45 in 5
counting down 907297240
2014-08-14 11:02:49,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 46 in 5
counting down 193602838
2014-08-14 11:02:49,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 47 in 6
counting down 255614005
2014-08-14 11:02:49,335 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 48 in 6
counting down 2128240408
2014-08-14 11:02:49,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 49 in 5
counting down 1783946138
2014-08-14 11:02:49,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 50 in 6
2014-08-14 11:02:49,347 INFO [com.milansimonovic.jepsen.Cluster] - creating 1 way partition
2014-08-14 11:02:50,348 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:50,348 INFO [com.milansimonovic.jepsen.Cluster] - creating partition DONE
2014-08-14 11:02:50,349 INFO [com.milansimonovic.jepsen.Cluster] - scheduled heal in 15 seconds
counting down 1330247343
2014-08-14 11:03:03,213 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - timed out adding 51
2014-08-14 11:03:03,214 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - error adding 51
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1147)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1211)
at com.milansimonovic.jepsen.CouchTimeoutTest.addElementToArray(CouchTimeoutTest.java:65)
at com.milansimonovic.jepsen.CouchTimeoutTest.appendToDocument(CouchTimeoutTest.java:102)
at com.milansimonovic.jepsen.CouchTimeoutTest.run(CouchTimeoutTest.java:49)
at com.milansimonovic.jepsen.CouchTimeoutTest.main(CouchTimeoutTest.java:39)
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 192.168.56.101/192.168.56.101:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1137)
... 5 more
2014-08-14 11:03:05,349 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:03:06,351 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - closing connections
counting down 1330247343
counting down 808247635
counting down 1562173079
2014-08-14 11:03:08,230 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 52 in 5014
counting down 1897031859
2014-08-14 11:03:08,236 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 53 in 6
2014-08-14 11:03:08,241 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 54 in 5
counting down 1029104896
2014-08-14 11:03:08,247 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 55 in 6
counting down 1789373824
2014-08-14 11:03:08,252 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 56 in 5
counting down 1374957093
2014-08-14 11:03:08,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 57 in 7
counting down 859935576
2014-08-14 11:03:08,265 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 58 in 5
counting down 959778517
2014-08-14 11:03:08,270 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 59 in 5
counting down 1856597982
2014-08-14 11:03:08,276 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 60 in 6
counting down 509403432
2014-08-14 11:03:08,283 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 61 in 7
counting down 565734120
2014-08-14 11:03:08,289 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 62 in 6
counting down 812288940
2014-08-14 11:03:08,295 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 63 in 6
counting down 362337760
2014-08-14 11:03:08,301 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 64 in 6
counting down 1815360474
counting down 238680990
2014-08-14 11:03:08,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 65 in 6
2014-08-14 11:03:08,312 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 66 in 5
counting down 1793734174
2014-08-14 11:03:08,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 67 in 6
counting down 965663094
2014-08-14 11:03:08,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 68 in 5
counting down 1961465726
2014-08-14 11:03:08,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 69 in 6
counting down 894265206
counting down 237825287
2014-08-14 11:03:08,334 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 70 in 5
2014-08-14 11:03:08,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 71 in 6
counting down 1093062877
2014-08-14 11:03:08,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 72 in 6
counting down 2043477472
2014-08-14 11:03:08,352 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 73 in 6
counting down 660886392
2014-08-14 11:03:08,358 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 74 in 6
counting down 1321850438
2014-08-14 11:03:08,363 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 75 in 5
counting down 966674298
2014-08-14 11:03:08,368 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 76 in 5
counting down 665030228
counting down 584709707
2014-08-14 11:03:08,373 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 77 in 5
2014-08-14 11:03:08,379 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 78 in 6
counting down 581070768
2014-08-14 11:03:08,384 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 79 in 5
counting down 1766611209
2014-08-14 11:03:08,390 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 80 in 6
counting down 1378265946
2014-08-14 11:03:08,395 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 81 in 5
counting down 1067480315
2014-08-14 11:03:08,400 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 82 in 5
counting down 240010064
2014-08-14 11:03:08,406 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 83 in 6
counting down 96589697
counting down 1650118680
2014-08-14 11:03:08,411 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 84 in 5
2014-08-14 11:03:08,417 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 85 in 6
counting down 1583441668
2014-08-14 11:03:08,422 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 86 in 5
counting down 575269723
2014-08-14 11:03:08,427 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 87 in 5
counting down 692018508
2014-08-14 11:03:08,434 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 88 in 6
counting down 500269667
2014-08-14 11:03:08,440 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 89 in 6
counting down 1214091162
2014-08-14 11:03:08,446 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 90 in 6
counting down 631482105
2014-08-14 11:03:08,452 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 91 in 6
counting down 1982159420
2014-08-14 11:03:08,458 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 92 in 6
counting down 1744182273
counting down 1131642426
2014-08-14 11:03:08,463 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 93 in 5
2014-08-14 11:03:08,469 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 94 in 6
counting down 2069859523
2014-08-14 11:03:08,475 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 95 in 6
counting down 662166830
2014-08-14 11:03:08,481 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 96 in 6
counting down 1882643413
2014-08-14 11:03:08,487 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 97 in 6
counting down 678230343
2014-08-14 11:03:08,493 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 98 in 6
counting down 793919773
2014-08-14 11:03:08,498 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 99 in 5
counting down 911566708
2014-08-14 11:03:08,505 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 100 in 7
counting down 1887173144
2014-08-14 11:03:08,510 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 101 in 5
counting down 1058177937
2014-08-14 11:03:08,516 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 102 in 6
counting down 41252807
2014-08-14 11:03:08,523 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 103 in 6
counting down 528932410
2014-08-14 11:03:08,531 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 104 in 8
counting down 805461224
2014-08-14 11:03:08,538 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 105 in 7
counting down 463949682
2014-08-14 11:03:08,546 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 106 in 8
counting down 851822734
2014-08-14 11:03:08,554 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 107 in 8
counting down 454797282
2014-08-14 11:03:08,561 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 108 in 7
counting down 962354549
2014-08-14 11:03:08,568 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 109 in 7
counting down 413223411
2014-08-14 11:03:08,574 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 110 in 6
counting down 79455185
2014-08-14 11:03:08,580 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 111 in 6
counting down 1113518099
2014-08-14 11:03:08,586 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 112 in 6
counting down 431139544
2014-08-14 11:03:08,592 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 113 in 6
counting down 198084300
2014-08-14 11:03:08,599 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 114 in 7
counting down 1821101711
2014-08-14 11:03:08,605 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 115 in 6
counting down 1499489637
2014-08-14 11:03:08,611 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 116 in 6
counting down 1435037947
2014-08-14 11:03:08,618 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 117 in 7
counting down 705688672
2014-08-14 11:03:08,624 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 118 in 6
counting down 407021592
2014-08-14 11:03:08,630 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 119 in 6
counting down 1121168696
2014-08-14 11:03:08,637 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 120 in 7
counting down 1347326641
counting down 87539756
2014-08-14 11:03:08,642 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 121 in 5
2014-08-14 11:03:08,647 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 122 in 5
counting down 137935610
2014-08-14 11:03:08,653 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 123 in 6
counting down 1474133160
2014-08-14 11:03:08,659 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 124 in 6
counting down 1931651064
2014-08-14 11:03:08,665 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 125 in 6
counting down 1211051792
2014-08-14 11:03:08,688 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 126 in 23
counting down 93172613
2014-08-14 11:03:08,693 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 127 in 5
counting down 965322656
2014-08-14 11:03:08,699 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 128 in 6
counting down 1601358350
2014-08-14 11:03:08,705 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 129 in 6
counting down 1855207775
2014-08-14 11:03:08,710 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 130 in 5
counting down 107851411
2014-08-14 11:03:08,716 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 131 in 6
counting down 1915600217
2014-08-14 11:03:08,721 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 132 in 5
counting down 1002847144
2014-08-14 11:03:08,727 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 133 in 6
counting down 1960289102
2014-08-14 11:03:08,732 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 134 in 5
counting down 530940529
2014-08-14 11:03:08,738 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 135 in 6
counting down 16593833
2014-08-14 11:03:08,744 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 136 in 6
counting down 183255909
2014-08-14 11:03:08,749 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 137 in 5
counting down 1139297445
counting down 1886255267
2014-08-14 11:03:08,755 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 138 in 5
2014-08-14 11:03:08,760 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 139 in 5
counting down 426044796
2014-08-14 11:03:08,765 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 140 in 5
counting down 1903915614
2014-08-14 11:03:08,770 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 141 in 5
counting down 1895269808
2014-08-14 11:03:08,776 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 142 in 6
counting down 1866519533
counting down 1366412404
2014-08-14 11:03:08,781 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 143 in 5
2014-08-14 11:03:08,786 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 144 in 5
counting down 2075311494
2014-08-14 11:03:08,792 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 145 in 6
counting down 925138245
2014-08-14 11:03:08,797 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 146 in 5
counting down 1578242039
2014-08-14 11:03:08,802 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 147 in 5
counting down 23034413
2014-08-14 11:03:08,808 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 148 in 6
counting down 1869588749
counting down 2067087344
2014-08-14 11:03:08,814 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 149 in 6
2014-08-14 11:03:08,820 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 150 in 6
counting down 899087062
2014-08-14 11:03:08,826 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 151 in 6
counting down 1340235867
2014-08-14 11:03:08,832 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 152 in 6
counting down 1574412139
2014-08-14 11:03:08,838 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 153 in 6
counting down 256897438
2014-08-14 11:03:08,844 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 154 in 6
counting down 227404523
2014-08-14 11:03:08,849 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 155 in 5
counting down 1370600377
2014-08-14 11:03:08,855 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 156 in 5
counting down 821838371
2014-08-14 11:03:08,861 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 157 in 6
counting down 1087346339
counting down 578287331
2014-08-14 11:03:08,866 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 158 in 5
2014-08-14 11:03:08,872 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 159 in 6
counting down 2114430319
2014-08-14 11:03:08,877 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 160 in 5
counting down 750263766
2014-08-14 11:03:08,883 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 161 in 6
counting down 285142124
2014-08-14 11:03:08,889 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 162 in 6
counting down 228835454
2014-08-14 11:03:08,895 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 163 in 6
counting down 1322000504
2014-08-14 11:03:08,900 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 164 in 5
counting down 578208474
2014-08-14 11:03:08,906 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 165 in 6
counting down 1158469492
counting down 1446993819
2014-08-14 11:03:08,911 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 166 in 5
2014-08-14 11:03:08,917 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 167 in 6
counting down 2036300484
2014-08-14 11:03:08,923 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 168 in 5
counting down 1839929127
2014-08-14 11:03:08,928 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 169 in 5
counting down 1321196177
2014-08-14 11:03:08,934 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 170 in 6
counting down 1657833306
2014-08-14 11:03:08,941 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 171 in 7
counting down 1616599198
2014-08-14 11:03:08,948 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 172 in 7
counting down 1434216861
2014-08-14 11:03:08,955 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 173 in 7
counting down 1697078001
2014-08-14 11:03:08,962 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 174 in 7
counting down 53209944
2014-08-14 11:03:08,969 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 175 in 7
counting down 957271051
2014-08-14 11:03:08,983 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 176 in 14
counting down 260602422
2014-08-14 11:03:08,991 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 177 in 8
counting down 1344752100
2014-08-14 11:03:08,996 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 178 in 5
counting down 210373011
2014-08-14 11:03:09,003 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 179 in 7
counting down 1135342237
counting down 255748372
2014-08-14 11:03:09,008 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 180 in 5
2014-08-14 11:03:09,013 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 181 in 5
counting down 1706850686
2014-08-14 11:03:09,018 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 182 in 5
counting down 958800587
2014-08-14 11:03:09,023 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 183 in 5
counting down 621084776
counting down 1494203157
2014-08-14 11:03:09,028 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 184 in 5
2014-08-14 11:03:09,034 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 185 in 6
counting down 1610501827
2014-08-14 11:03:09,039 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 186 in 5
counting down 1472601997
2014-08-14 11:03:09,046 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 187 in 6
counting down 1523401565
2014-08-14 11:03:09,052 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 188 in 6
counting down 2118032386
2014-08-14 11:03:09,058 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 189 in 6
counting down 1402690896
2014-08-14 11:03:09,063 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 190 in 5
counting down 877005529
counting down 69300391
2014-08-14 11:03:09,068 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 191 in 5
counting down 250950317
2014-08-14 11:03:09,074 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 192 in 6
2014-08-14 11:03:09,080 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 193 in 6
counting down 126717950
2014-08-14 11:03:09,086 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 194 in 6
counting down 60378388
2014-08-14 11:03:09,092 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 195 in 6
counting down 213037448
2014-08-14 11:03:09,098 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 196 in 6
counting down 737512169
2014-08-14 11:03:09,103 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 197 in 5
counting down 1089208854
2014-08-14 11:03:09,109 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 198 in 6
counting down 156501746
2014-08-14 11:03:09,115 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 199 in 6
counting down 284416161
2014-08-14 11:03:09,121 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 200 in 6
2014-08-14 11:03:09,123 INFO [com.couchbase.client.CouchbaseConnection] - Shut down Couchbase client
2014-08-14 11:03:09,125 INFO [com.couchbase.client.ViewConnection] - I/O reactor terminated
2014-08-14 11:03:09,125 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - DONE!
Disconnected from the target VM, address: '127.0.0.1:61775', transport: 'socket'

Process finished with exit code 0