Project: mesos
Issue MESOS-1425: LogZooKeeperTest.WriteRead test is flaky
{code} [ RUN ] LogZooKeeperTest.WriteRead I0527 23:23:48.286031 1352 zookeeper_test_server.cpp:158] Started ZooKeeperTestServer on port 39446 I0527 23:23:48.293916 1352 log_tests.cpp:1945] Using temporary directory '/tmp/LogZooKeeperTest_WriteRead_Vyty8g' I0527 23:23:48.296430 1352 leveldb.cpp:176] Opened db in 2.459713ms I0527 23:23:48.296740 1352 leveldb.cpp:183] Compacted db in 286843ns I0527 23:23:48.296761 1352 leveldb.cpp:198] Created db iterator in 3083ns I0527 23:23:48.296772 1352 leveldb.cpp:204] Seeked to beginning of db in 4541ns I0527 23:23:48.296777 1352 leveldb.cpp:273] Iterated through 0 keys in the db in 87ns I0527 23:23:48.296788 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0527 23:23:48.297499 1383 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 505340ns I0527 23:23:48.297513 1383 replica.cpp:320] Persisted replica status to VOTING I0527 23:23:48.299492 1352 leveldb.cpp:176] Opened db in 1.73582ms I0527 23:23:48.299773 1352 leveldb.cpp:183] Compacted db in 263937ns I0527 23:23:48.299793 1352 leveldb.cpp:198] Created db iterator in 7494ns I0527 23:23:48.299806 1352 leveldb.cpp:204] Seeked to beginning of db in 235ns I0527 23:23:48.299813 1352 leveldb.cpp:273] Iterated through 0 keys in the db in 93ns I0527 23:23:48.299821 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0527 23:23:48.300503 1380 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 492309ns I0527 23:23:48.300516 1380 replica.cpp:320] Persisted replica status to VOTING I0527 23:23:48.302500 1352 leveldb.cpp:176] Opened db in 1.793829ms I0527 23:23:48.303642 1352 leveldb.cpp:183] Compacted db in 1.123929ms I0527 23:23:48.303669 1352 leveldb.cpp:198] Created db iterator in 5865ns I0527 23:23:48.303689 1352 leveldb.cpp:204] Seeked to beginning of db in 8811ns I0527 23:23:48.303705 1352 leveldb.cpp:273] Iterated through 1 keys in the db in 9545ns I0527 23:23:48.303715 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,303:1352(0x2b1173e2b700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@733: Client environment:user.name=(null) I0527 23:23:48.303988 1380 log.cpp:238] Attempting to join replica to ZooKeeper group 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins I0527 23:23:48.304198 1385 recover.cpp:425] Starting replica recovery 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd= context=0x2b118002f4e0 flags=0 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g I0527 23:23:48.304352 1385 recover.cpp:451] Replica is in VOTING status 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd= context=0x2b1198015ca0 flags=0 I0527 23:23:48.304417 1385 recover.cpp:440] Recover process terminated 2014-05-27 23:23:48,304:1352(0x2b12897b8700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] 2014-05-27 23:23:48,304:1352(0x2b12891b5700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] I0527 23:23:48.311262 1352 leveldb.cpp:176] Opened db in 7.261703ms 2014-05-27 23:23:48,311:1352(0x2b12897b8700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0000, negotiated timeout=6000 I0527 23:23:48.312379 1381 group.cpp:310] Group process ((614)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.312407 1381 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0527 23:23:48.312417 1381 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.312422 1352 leveldb.cpp:183] Compacted db in 1.119843ms I0527 23:23:48.312505 1352 leveldb.cpp:198] Created db iterator in 3901ns I0527 23:23:48.312526 1352 leveldb.cpp:204] Seeked to beginning of db in 7398ns I0527 23:23:48.312541 1352 leveldb.cpp:273] Iterated through 1 keys in the db in 6345ns I0527 23:23:48.312553 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,312:1352(0x2b12891b5700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0001, negotiated timeout=6000 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd= context=0x2b119001fd20 flags=0 I0527 23:23:48.313247 1380 group.cpp:310] Group process ((616)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.313266 1380 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0527 23:23:48.313273 1380 group.cpp:382] Trying to create path '/log' in ZooKeeper 2014-05-27 23:23:48,313:1352(0x2b12889b0700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 I0527 23:23:48.313436 1387 log.cpp:238] Attempting to join replica to ZooKeeper group 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd= context=0x2b1190011ea0 flags=0 I0527 23:23:48.313601 1387 recover.cpp:425] Starting replica recovery I0527 23:23:48.313721 1382 recover.cpp:451] Replica is in VOTING status I0527 23:23:48.313794 1382 recover.cpp:440] Recover process terminated 2014-05-27 23:23:48,313:1352(0x2b1288bb1700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] I0527 23:23:48.313973 1383 log.cpp:656] Attempting to start the writer 2014-05-27 23:23:48,315:1352(0x2b12889b0700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0002, negotiated timeout=6000 I0527 23:23:48.315682 1387 group.cpp:310] Group process ((619)@67.195.138.8:35151) connected to ZooKeeper 2014-05-27 23:23:48,315:1352(0x2b1288bb1700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0003, negotiated timeout=6000 I0527 23:23:48.315709 1387 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0527 23:23:48.315738 1387 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.315964 1386 group.cpp:310] Group process ((621)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.315981 1386 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0527 23:23:48.315989 1386 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.317881 1385 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.317937 1381 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.318205 1382 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.318317 1383 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.319154 1382 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } I0527 23:23:48.319541 1386 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } I0527 23:23:48.319851 1381 replica.cpp:474] Replica received implicit promise request with proposal 1 I0527 23:23:48.319905 1387 replica.cpp:474] Replica received implicit promise request with proposal 1 I0527 23:23:48.319907 1384 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.320091 1385 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.320384 1383 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.320441 1381 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 568396ns I0527 23:23:48.320456 1384 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.320461 1381 replica.cpp:342] Persisted promised to 1 I0527 23:23:48.320446 1387 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 516015ns I0527 23:23:48.320497 1387 replica.cpp:342] Persisted promised to 1 I0527 23:23:48.320814 1383 coordinator.cpp:230] Coordinator attemping to fill missing position I0527 23:23:48.321050 1384 group.cpp:655] Trying to get '/log/0000000001' in ZooKeeper I0527 23:23:48.321063 1385 group.cpp:655] Trying to get '/log/0000000001' in ZooKeeper I0527 23:23:48.321341 1387 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0527 23:23:48.321375 1381 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0527 23:23:48.321506 1387 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 89us I0527 23:23:48.321530 1387 replica.cpp:676] Persisted action at 0 I0527 23:23:48.321584 1381 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 122910ns I0527 23:23:48.321602 1381 replica.cpp:676] Persisted action at 0 I0527 23:23:48.321775 1383 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151, log-replica(23)@67.195.138.8:35151 } I0527 23:23:48.321961 1381 replica.cpp:508] Replica received write request for position 0 I0527 23:23:48.321984 1381 leveldb.cpp:438] Reading position from leveldb took 7813ns I0527 23:23:48.322064 1380 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151, log-replica(23)@67.195.138.8:35151 } I0527 23:23:48.322073 1381 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 78683ns I0527 23:23:48.322077 1383 replica.cpp:508] Replica received write request for position 0 I0527 23:23:48.322084 1381 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322111 1383 leveldb.cpp:438] Reading position from leveldb took 17416ns I0527 23:23:48.322330 1383 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 157199ns I0527 23:23:48.322345 1383 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322522 1386 replica.cpp:655] Replica received learned notice for position 0 I0527 23:23:48.322523 1382 replica.cpp:655] Replica received learned notice for position 0 I0527 23:23:48.322638 1386 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 86907ns I0527 23:23:48.322661 1386 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322670 1386 replica.cpp:661] Replica learned NOP action at position 0 I0527 23:23:48.322682 1382 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 85031ns I0527 23:23:48.322693 1382 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322700 1382 replica.cpp:661] Replica learned NOP action at position 0 I0527 23:23:48.322790 1380 log.cpp:672] Writer started with ending position 0 I0527 23:23:48.322898 1380 log.cpp:680] Attempting to append 11 bytes to the log I0527 23:23:48.322978 1383 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0527 23:23:48.323122 1380 replica.cpp:508] Replica received write request for position 1 I0527 23:23:48.323158 1381 replica.cpp:508] Replica received write request for position 1 I0527 23:23:48.323202 1380 leveldb.cpp:343] Persisting action (27 bytes) to leveldb took 66527ns I0527 23:23:48.323215 1380 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323238 1381 leveldb.cpp:343] Persisting action (27 bytes) to leveldb took 67074ns I0527 23:23:48.323252 1381 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323354 1380 replica.cpp:655] Replica received learned notice for position 1 I0527 23:23:48.323362 1382 replica.cpp:655] Replica received learned notice for position 1 I0527 23:23:48.323443 1380 leveldb.cpp:343] Persisting action (29 bytes) to leveldb took 77398ns I0527 23:23:48.323461 1380 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323463 1382 leveldb.cpp:343] Persisting action (29 bytes) to leveldb took 90567ns I0527 23:23:48.323467 1380 replica.cpp:661] Replica learned APPEND action at position 1 I0527 23:23:48.323477 1382 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323484 1382 replica.cpp:661] Replica learned APPEND action at position 1 I0527 23:23:48.323729 1380 leveldb.cpp:438] Reading position from leveldb took 7224ns 2014-05-27 23:23:48,324:1352(0x2b1173c2a700):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x1463fff34bd0003 to [127.0.0.1:39446] 2014-05-27 23:23:48,324:1352(0x2b117301ff80):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x1463fff34bd0002 to [127.0.0.1:39446] I0527 23:23:48.326591 1386 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.326690 1382 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.327450 1384 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } 2014-05-27 23:23:48,446:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:23:51,782:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:23:55,118:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client I0527 23:23:57.002908 1381 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:57.003042 1381 network.hpp:461] ZooKeeper group PIDs: { } 2014-05-27 23:23:58,455:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:01,791:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:05,127:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:08,464:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:11,800:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:15,136:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:18,473:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:21,809:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:25,146:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:28,482:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:31,818:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:35,155:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:38,491:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:41,827:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retc...
Story points: 1