Project: mesos
Issue MESOS-2283: SlaveRecoveryTest.ReconcileKillTask is flaky.
Saw this on an internal CI: {noformat} [ RUN ] SlaveRecoveryTest/0.ReconcileKillTask Using temporary directory '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_D5wSwg' I0126 19:10:52.005317 13291 leveldb.cpp:176] Opened db in 978670ns I0126 19:10:52.006155 13291 leveldb.cpp:183] Compacted db in 541346ns I0126 19:10:52.006494 13291 leveldb.cpp:198] Created db iterator in 24562ns I0126 19:10:52.006798 13291 leveldb.cpp:204] Seeked to beginning of db in 3254ns I0126 19:10:52.007036 13291 leveldb.cpp:273] Iterated through 0 keys in the db in 949ns I0126 19:10:52.007369 13291 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0126 19:10:52.008362 13308 recover.cpp:449] Starting replica recovery I0126 19:10:52.009141 13308 recover.cpp:475] Replica is in EMPTY status I0126 19:10:52.016494 13308 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0126 19:10:52.017333 13309 recover.cpp:195] Received a recover response from a replica in EMPTY status I0126 19:10:52.018244 13309 recover.cpp:566] Updating replica status to STARTING I0126 19:10:52.019064 13305 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 113577ns I0126 19:10:52.019487 13305 replica.cpp:323] Persisted replica status to STARTING I0126 19:10:52.019937 13309 recover.cpp:475] Replica is in STARTING status I0126 19:10:52.021492 13307 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0126 19:10:52.022665 13309 recover.cpp:195] Received a recover response from a replica in STARTING status I0126 19:10:52.027971 13312 recover.cpp:566] Updating replica status to VOTING I0126 19:10:52.028590 13312 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 78452ns I0126 19:10:52.028869 13312 replica.cpp:323] Persisted replica status to VOTING I0126 19:10:52.029252 13312 recover.cpp:580] Successfully joined the Paxos group I0126 19:10:52.030828 13307 recover.cpp:464] Recover process terminated I0126 19:10:52.049947 13306 master.cpp:262] Master 20150126-191052-2272962752-35545-13291 (fedora-19) started on 192.168.122.135:35545 I0126 19:10:52.050499 13306 master.cpp:308] Master only allowing authenticated frameworks to register I0126 19:10:52.050765 13306 master.cpp:313] Master only allowing authenticated slaves to register I0126 19:10:52.051048 13306 credentials.hpp:36] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_D5wSwg/credentials' I0126 19:10:52.051589 13306 master.cpp:357] Authorization enabled I0126 19:10:52.052531 13305 hierarchical_allocator_process.hpp:285] Initialized hierarchical allocator process I0126 19:10:52.052881 13311 whitelist_watcher.cpp:65] No whitelist given I0126 19:10:52.055524 13306 master.cpp:1219] The newly elected leader is master@192.168.122.135:35545 with id 20150126-191052-2272962752-35545-13291 I0126 19:10:52.056226 13306 master.cpp:1232] Elected as the leading master! I0126 19:10:52.056639 13306 master.cpp:1050] Recovering from registrar I0126 19:10:52.057045 13307 registrar.cpp:313] Recovering registrar I0126 19:10:52.058554 13312 log.cpp:660] Attempting to start the writer I0126 19:10:52.060868 13309 replica.cpp:477] Replica received implicit promise request with proposal 1 I0126 19:10:52.061691 13309 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 91680ns I0126 19:10:52.062261 13309 replica.cpp:345] Persisted promised to 1 I0126 19:10:52.064559 13310 coordinator.cpp:230] Coordinator attemping to fill missing position I0126 19:10:52.069105 13311 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0126 19:10:52.069860 13311 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 94858ns I0126 19:10:52.070350 13311 replica.cpp:679] Persisted action at 0 I0126 19:10:52.080348 13305 replica.cpp:511] Replica received write request for position 0 I0126 19:10:52.081153 13305 leveldb.cpp:438] Reading position from leveldb took 62247ns I0126 19:10:52.081676 13305 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 81487ns I0126 19:10:52.082053 13305 replica.cpp:679] Persisted action at 0 I0126 19:10:52.083566 13309 replica.cpp:658] Replica received learned notice for position 0 I0126 19:10:52.085734 13309 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 283144ns I0126 19:10:52.086067 13309 replica.cpp:679] Persisted action at 0 I0126 19:10:52.086448 13309 replica.cpp:664] Replica learned NOP action at position 0 I0126 19:10:52.089784 13306 log.cpp:676] Writer started with ending position 0 I0126 19:10:52.093415 13309 leveldb.cpp:438] Reading position from leveldb took 66744ns I0126 19:10:52.104814 13306 registrar.cpp:346] Successfully fetched the registry (0B) in 47.451136ms I0126 19:10:52.105731 13306 registrar.cpp:445] Applied 1 operations in 42124ns; attempting to update the 'registry' I0126 19:10:52.111935 13305 log.cpp:684] Attempting to append 131 bytes to the log I0126 19:10:52.112754 13305 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0126 19:10:52.114297 13308 replica.cpp:511] Replica received write request for position 1 I0126 19:10:52.114908 13308 leveldb.cpp:343] Persisting action (150 bytes) to leveldb took 98332ns I0126 19:10:52.115387 13308 replica.cpp:679] Persisted action at 1 I0126 19:10:52.117277 13305 replica.cpp:658] Replica received learned notice for position 1 I0126 19:10:52.118142 13305 leveldb.cpp:343] Persisting action (152 bytes) to leveldb took 227799ns I0126 19:10:52.118621 13305 replica.cpp:679] Persisted action at 1 I0126 19:10:52.118979 13305 replica.cpp:664] Replica learned APPEND action at position 1 I0126 19:10:52.121311 13305 registrar.cpp:490] Successfully updated the 'registry' in 15.161088ms I0126 19:10:52.121548 13311 log.cpp:703] Attempting to truncate the log to 1 I0126 19:10:52.122697 13311 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0126 19:10:52.124316 13307 replica.cpp:511] Replica received write request for position 2 I0126 19:10:52.124913 13307 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 87281ns I0126 19:10:52.125334 13307 replica.cpp:679] Persisted action at 2 I0126 19:10:52.127018 13311 replica.cpp:658] Replica received learned notice for position 2 I0126 19:10:52.127835 13311 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 201050ns I0126 19:10:52.128232 13311 leveldb.cpp:401] Deleting ~1 keys from leveldb took 78012ns I0126 19:10:52.128835 13305 registrar.cpp:376] Successfully recovered registrar I0126 19:10:52.128551 13311 replica.cpp:679] Persisted action at 2 I0126 19:10:52.130105 13311 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0126 19:10:52.131479 13312 master.cpp:1077] Recovered 0 slaves from the Registry (95B) ; allowing 10mins for slaves to re-register I0126 19:10:52.143465 13291 containerizer.cpp:103] Using isolation: posix/cpu,posix/mem I0126 19:10:52.170471 13309 slave.cpp:173] Slave started on 101)@192.168.122.135:35545 I0126 19:10:52.171723 13309 credentials.hpp:84] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_qbguuM/credential' I0126 19:10:52.172286 13309 slave.cpp:282] Slave using credential for: test-principal I0126 19:10:52.172821 13309 slave.cpp:300] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0126 19:10:52.173982 13309 slave.cpp:329] Slave hostname: fedora-19 I0126 19:10:52.174505 13309 slave.cpp:330] Slave checkpoint: true I0126 19:10:52.179308 13309 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_qbguuM/meta' I0126 19:10:52.180075 13308 status_update_manager.cpp:197] Recovering status update manager I0126 19:10:52.180611 13308 containerizer.cpp:300] Recovering containerizer I0126 19:10:52.182473 13309 slave.cpp:3519] Finished recovery I0126 19:10:52.184403 13312 slave.cpp:613] New master detected at master@192.168.122.135:35545 I0126 19:10:52.184916 13312 slave.cpp:676] Authenticating with master master@192.168.122.135:35545 I0126 19:10:52.185230 13312 slave.cpp:681] Using default CRAM-MD5 authenticatee I0126 19:10:52.185715 13312 slave.cpp:649] Detecting new master I0126 19:10:52.186420 13312 authenticatee.hpp:138] Creating new client SASL connection I0126 19:10:52.186002 13311 status_update_manager.cpp:171] Pausing sending status updates I0126 19:10:52.188293 13312 master.cpp:4129] Authenticating slave(101)@192.168.122.135:35545 I0126 19:10:52.188748 13312 master.cpp:4140] Using default CRAM-MD5 authenticator I0126 19:10:52.189525 13312 authenticator.hpp:170] Creating new server SASL connection I0126 19:10:52.191082 13305 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0126 19:10:52.191550 13305 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0126 19:10:52.191990 13312 authenticator.hpp:276] Received SASL authentication start I0126 19:10:52.192365 13312 authenticator.hpp:398] Authentication requires more steps I0126 19:10:52.192800 13311 authenticatee.hpp:275] Received SASL authentication step I0126 19:10:52.193244 13312 authenticator.hpp:304] Received SASL authentication step I0126 19:10:52.193565 13312 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0126 19:10:52.193902 13312 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0126 19:10:52.194301 13312 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0126 19:10:52.195669 13312 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0126 19:10:52.196048 13312 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0126 19:10:52.196395 13312 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0126 19:10:52.196723 13312 authenticator.hpp:390] Authentication success I0126 19:10:52.197206 13305 authenticatee.hpp:315] Authentication success I0126 19:10:52.204121 13305 slave.cpp:747] Successfully authenticated with master master@192.168.122.135:35545 I0126 19:10:52.204676 13310 master.cpp:4187] Successfully authenticated principal 'test-principal' at slave(101)@192.168.122.135:35545 I0126 19:10:52.205729 13305 slave.cpp:1075] Will retry registration in 5.608661ms if necessary I0126 19:10:52.206451 13310 master.cpp:3275] Registering slave at slave(101)@192.168.122.135:35545 (fedora-19) with id 20150126-191052-2272962752-35545-13291-S0 I0126 19:10:52.210019 13310 registrar.cpp:445] Applied 1 operations in 235087ns; attempting to update the 'registry' I0126 19:10:52.220736 13308 slave.cpp:1075] Will retry registration in 9.28397ms if necessary I0126 19:10:52.221309 13311 master.cpp:3263] Ignoring register slave message from slave(101)@192.168.122.135:35545 (fedora-19) as admission is already in progress I0126 19:10:52.224818 13307 log.cpp:684] Attempting to append 302 bytes to the log I0126 19:10:52.225554 13307 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0126 19:10:52.227422 13305 replica.cpp:511] Replica received write request for position 3 I0126 19:10:52.227969 13305 leveldb.cpp:343] Persisting action (321 bytes) to leveldb took 100350ns I0126 19:10:52.228276 13305 replica.cpp:679] Persisted action at 3 I0126 19:10:52.232475 13312 replica.cpp:658] Replica received learned notice for position 3 I0126 19:10:52.233280 13312 leveldb.cpp:343] Persisting action (323 bytes) to leveldb took 546567ns I0126 19:10:52.233726 13312 replica.cpp:679] Persisted action at 3 I0126 19:10:52.234035 13312 replica.cpp:664] Replica learned APPEND action at position 3 I0126 19:10:52.236556 13310 registrar.cpp:490] Successfully updated the 'registry' in 26.040064ms I0126 19:10:52.237330 13305 log.cpp:703] Attempting to truncate the log to 3 I0126 19:10:52.238056 13311 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0126 19:10:52.239594 13311 replica.cpp:511] Replica received write request for position 4 I0126 19:10:52.240129 13311 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 92868ns I0126 19:10:52.240458 13311 replica.cpp:679] Persisted action at 4 I0126 19:10:52.241976 13308 replica.cpp:658] Replica received learned notice for position 4 I0126 19:10:52.242645 13308 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 95635ns I0126 19:10:52.242990 13308 leveldb.cpp:401] Deleting ~2 keys from leveldb took 58066ns I0126 19:10:52.243337 13308 replica.cpp:679] Persisted action at 4 I0126 19:10:52.243695 13308 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0126 19:10:52.245657 13291 sched.cpp:151] Version: 0.22.0 I0126 19:10:52.247625 13305 master.cpp:3329] Registered slave 20150126-191052-2272962752-35545-13291-S0 at slave(101)@192.168.122.135:35545 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0126 19:10:52.248942 13307 slave.cpp:781] Registered with master master@192.168.122.135:35545; given slave ID 20150126-191052-2272962752-35545-13291-S0 I0126 19:10:52.250396 13307 slave.cpp:797] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_qbguuM/meta/slaves/20150126-191052-2272962752-35545-13291-S0/slave.info' I0126 19:10:52.250731 13309 status_update_manager.cpp:178] Resuming sending status updates I0126 19:10:52.251765 13307 slave.cpp:2588] Received ping from slave-observer(99)@192.168.122.135:35545 I0126 19:10:52.247951 13310 hierarchical_allocator_process.hpp:453] Added slave 20150126-191052-2272962752-35545-13291-S0 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0126 19:10:52.252810 13310 hierarchical_allocator_process.hpp:831] No resources available to allocate! I0126 19:10:52.254365 13310 hierarchical_allocator_process.hpp:756] Performed allocation for slave 20150126-191052-2272962752-35545-13291-S0 in 1.732701ms I0126 19:10:52.254137 13307 sched.cpp:248] New master detected at master@192.168.122.135:35545 I0126 19:10:52.257863 13307 sched.cpp:304] Authenticating with master master@192.168.122.135:35545 I0126 19:10:52.258249 13307 sched.cpp:311] Using default CRAM-MD5 authenticatee I0126 19:10:52.258908 13306 authenticatee.hpp:138] Creating new client SASL connection I0126 19:10:52.261397 13309 master.cpp:4129] Authenticating scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.261776 13309 master.cpp:4140] Using default CRAM-MD5 authenticator I0126 19:10:52.264528 13309 authenticator.hpp:170] Creating new server SASL connection I0126 19:10:52.266248 13312 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0126 19:10:52.266749 13312 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0126 19:10:52.267143 13312 authenticator.hpp:276] Received SASL authentication start I0126 19:10:52.267525 13312 authenticator.hpp:398] Authentication requires more steps I0126 19:10:52.267917 13312 authenticatee.hpp:275] Received SASL authentication step I0126 19:10:52.268404 13312 authenticator.hpp:304] Received SASL authentication step I0126 19:10:52.268725 13312 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0126 19:10:52.269078 13312 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0126 19:10:52.269498 13312 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0126 19:10:52.269881 13312 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0126 19:10:52.270385 13312 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0126 19:10:52.271015 13312 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0126 19:10:52.271599 13312 authenticator.hpp:390] Authentication success I0126 19:10:52.272126 13312 authenticatee.hpp:315] Authentication success I0126 19:10:52.272415 13305 master.cpp:4187] Successfully authenticated principal 'test-principal' at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.273998 13307 sched.cpp:392] Successfully authenticated with master master@192.168.122.135:35545 I0126 19:10:52.274415 13307 sched.cpp:515] Sending registration request to master@192.168.122.135:35545 I0126 19:10:52.274842 13307 sched.cpp:548] Will retry registration in 674.656506ms if necessary I0126 19:10:52.275235 13305 master.cpp:1420] Received registration request for framework 'default' at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.276017 13305 master.cpp:1298] Authorizing framework principal 'test-principal' to receive offers for role '*' I0126 19:10:52.277027 13305 master.cpp:1484] Registering framework 20150126-191052-2272962752-35545-13291-0000 (default) at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.278285 13308 hierarchical_allocator_process.hpp:319] Added framework 20150126-191052-2272962752-35545-13291-0000 I0126 19:10:52.279575 13308 hierarchical_allocator_process.hpp:738] Performed allocation for 1 slaves in 697902ns I0126 19:10:52.287966 13305 master.cpp:4071] Sending 1 offers to framework 20150126-191052-2272962752-35545-13291-0000 (default) at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.288776 13307 sched.cpp:442] Framework registered with 20150126-191052-2272962752-35545-13291-0000 I0126 19:10:52.289373 13307 sched.cpp:456] Scheduler::registered took 21674ns I0126 19:10:52.289932 13307 sched.cpp:605] Scheduler::resourceOffers took 76147ns I0126 19:10:52.293220 13311 master.cpp:2677] Processing ACCEPT call for offers: [ 20150126-191052-2272962752-35545-13291-O0 ] on slave 20150126-191052-2272962752-35545-13291-S0 at slave(101)@192.168.122.135:35545 (fedora-19) for framework 20150126-191052-2272962752-35545-13291-0000 (default) at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 I0126 19:10:52.293586 13311 master.cpp:2513] Authorizing framework principal 'test-principal' to launch task 61eaeec3-e8ca-4e15-82d6-284c05c3bb6e as user 'jenkins' I0126 19:10:52.295825 13311 master.hpp:782] Adding task 61eaeec3-e8ca-4e15-82d6-284c05c3bb6e with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20150126-191052-2272962752-35545-13291-S0 (fedora-19) I0126 19:10:52.296272 13311 master.cpp:2885] Launching task 61eaeec3-e8ca-4e15-82d6-284c05c3bb6e of framework 20150126-191052-2272962752-35545-13291-0000 (default) at scheduler-6da85b48-b57f-4202-b630-c45f8f652321@192.168.122.135:35545 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20150126-191052-2272962752-35545-13291-S0 at slave(101)@192.168.122.135:35545 (fedora-19) I0126 19:10:52.296886 13309 slave.cpp:1130] Got assigned task 61eaeec3-e8ca-4e15-82d6-284c05c3bb6e for framework 20150126-191052-2272962752-35545-13291-0000 I0126 19:10:52.297324 13309 slave.cpp:3846] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_qbguuM/meta/slaves/20150126-191052-2272962752-35545-13291-S0/frameworks/20150126-191052-22...
Story points: 1