Project: mesos
Issue MESOS-1594: SlaveRecoveryTest/0.ReconcileKillTask is flaky
Observed this on Jenkins. {code} [ RUN ] SlaveRecoveryTest/0.ReconcileKillTask Using temporary directory '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_3zJ6DG' I0714 15:08:43.915114 27216 leveldb.cpp:176] Opened db in 474.695188ms I0714 15:08:43.933645 27216 leveldb.cpp:183] Compacted db in 18.068942ms I0714 15:08:43.934129 27216 leveldb.cpp:198] Created db iterator in 7860ns I0714 15:08:43.934439 27216 leveldb.cpp:204] Seeked to beginning of db in 2560ns I0714 15:08:43.934779 27216 leveldb.cpp:273] Iterated through 0 keys in the db in 1400ns I0714 15:08:43.935098 27216 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0714 15:08:43.936027 27238 recover.cpp:425] Starting replica recovery I0714 15:08:43.936225 27238 recover.cpp:451] Replica is in EMPTY status I0714 15:08:43.936867 27238 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0714 15:08:43.937049 27238 recover.cpp:188] Received a recover response from a replica in EMPTY status I0714 15:08:43.937232 27238 recover.cpp:542] Updating replica status to STARTING I0714 15:08:43.945600 27235 master.cpp:288] Master 20140714-150843-16842879-55850-27216 (quantal) started on 127.0.1.1:55850 I0714 15:08:43.945643 27235 master.cpp:325] Master only allowing authenticated frameworks to register I0714 15:08:43.945651 27235 master.cpp:330] Master only allowing authenticated slaves to register I0714 15:08:43.945658 27235 credentials.hpp:36] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_3zJ6DG/credentials' I0714 15:08:43.945808 27235 master.cpp:359] Authorization enabled I0714 15:08:43.946369 27235 hierarchical_allocator_process.hpp:301] Initializing hierarchical allocator process with master : master@127.0.1.1:55850 I0714 15:08:43.946419 27235 master.cpp:122] No whitelist given. Advertising offers for all slaves I0714 15:08:43.946614 27235 master.cpp:1128] The newly elected leader is master@127.0.1.1:55850 with id 20140714-150843-16842879-55850-27216 I0714 15:08:43.946630 27235 master.cpp:1141] Elected as the leading master! I0714 15:08:43.946637 27235 master.cpp:959] Recovering from registrar I0714 15:08:43.946707 27235 registrar.cpp:313] Recovering registrar I0714 15:08:43.957895 27238 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 20.529301ms I0714 15:08:43.957978 27238 replica.cpp:320] Persisted replica status to STARTING I0714 15:08:43.958142 27238 recover.cpp:451] Replica is in STARTING status I0714 15:08:43.958664 27238 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0714 15:08:43.958762 27238 recover.cpp:188] Received a recover response from a replica in STARTING status I0714 15:08:43.958945 27238 recover.cpp:542] Updating replica status to VOTING I0714 15:08:43.975685 27238 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 16.646136ms I0714 15:08:43.976367 27238 replica.cpp:320] Persisted replica status to VOTING I0714 15:08:43.976824 27241 recover.cpp:556] Successfully joined the Paxos group I0714 15:08:43.977072 27242 recover.cpp:440] Recover process terminated I0714 15:08:43.980590 27236 log.cpp:656] Attempting to start the writer I0714 15:08:43.981385 27236 replica.cpp:474] Replica received implicit promise request with proposal 1 I0714 15:08:43.999141 27236 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 17.705787ms I0714 15:08:43.999222 27236 replica.cpp:342] Persisted promised to 1 I0714 15:08:44.004451 27240 coordinator.cpp:230] Coordinator attemping to fill missing position I0714 15:08:44.004914 27240 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0714 15:08:44.021456 27240 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 16.499775ms I0714 15:08:44.021533 27240 replica.cpp:676] Persisted action at 0 I0714 15:08:44.022006 27240 replica.cpp:508] Replica received write request for position 0 I0714 15:08:44.022043 27240 leveldb.cpp:438] Reading position from leveldb took 21376ns I0714 15:08:44.035969 27240 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 13.885907ms I0714 15:08:44.036365 27240 replica.cpp:676] Persisted action at 0 I0714 15:08:44.040156 27238 replica.cpp:655] Replica received learned notice for position 0 I0714 15:08:44.058082 27238 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 17.860707ms I0714 15:08:44.058161 27238 replica.cpp:676] Persisted action at 0 I0714 15:08:44.058176 27238 replica.cpp:661] Replica learned NOP action at position 0 I0714 15:08:44.058526 27238 log.cpp:672] Writer started with ending position 0 I0714 15:08:44.058872 27238 leveldb.cpp:438] Reading position from leveldb took 25660ns I0714 15:08:44.060556 27238 registrar.cpp:346] Successfully fetched the registry (0B) I0714 15:08:44.060845 27238 registrar.cpp:422] Attempting to update the 'registry' I0714 15:08:44.062304 27238 log.cpp:680] Attempting to append 120 bytes to the log I0714 15:08:44.062866 27236 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0714 15:08:44.063154 27236 replica.cpp:508] Replica received write request for position 1 I0714 15:08:44.082813 27236 leveldb.cpp:343] Persisting action (137 bytes) to leveldb took 19.61683ms I0714 15:08:44.082890 27236 replica.cpp:676] Persisted action at 1 I0714 15:08:44.083256 27236 replica.cpp:655] Replica received learned notice for position 1 I0714 15:08:44.097398 27236 leveldb.cpp:343] Persisting action (139 bytes) to leveldb took 14.104796ms I0714 15:08:44.097475 27236 replica.cpp:676] Persisted action at 1 I0714 15:08:44.097488 27236 replica.cpp:661] Replica learned APPEND action at position 1 I0714 15:08:44.098569 27236 registrar.cpp:479] Successfully updated 'registry' I0714 15:08:44.098906 27240 log.cpp:699] Attempting to truncate the log to 1 I0714 15:08:44.099608 27240 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0714 15:08:44.100005 27240 replica.cpp:508] Replica received write request for position 2 I0714 15:08:44.100566 27236 registrar.cpp:372] Successfully recovered registrar I0714 15:08:44.101227 27239 master.cpp:986] Recovered 0 slaves from the Registry (84B) ; allowing 10mins for slaves to re-register I0714 15:08:44.118376 27240 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 18.329495ms I0714 15:08:44.118455 27240 replica.cpp:676] Persisted action at 2 I0714 15:08:44.122258 27242 replica.cpp:655] Replica received learned notice for position 2 I0714 15:08:44.137336 27242 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 15.023553ms I0714 15:08:44.137460 27242 leveldb.cpp:401] Deleting ~1 keys from leveldb took 55049ns I0714 15:08:44.137480 27242 replica.cpp:676] Persisted action at 2 I0714 15:08:44.137492 27242 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0714 15:08:44.143729 27216 containerizer.cpp:124] Using isolation: posix/cpu,posix/mem I0714 15:08:44.145934 27242 slave.cpp:168] Slave started on 43)@127.0.1.1:55850 I0714 15:08:44.145953 27242 credentials.hpp:84] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/credential' I0714 15:08:44.146040 27242 slave.cpp:266] Slave using credential for: test-principal I0714 15:08:44.146136 27242 slave.cpp:279] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0714 15:08:44.146198 27242 slave.cpp:324] Slave hostname: quantal I0714 15:08:44.146209 27242 slave.cpp:325] Slave checkpoint: true I0714 15:08:44.146708 27242 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta' I0714 15:08:44.146824 27242 status_update_manager.cpp:193] Recovering status update manager I0714 15:08:44.146901 27242 containerizer.cpp:287] Recovering containerizer I0714 15:08:44.147228 27242 slave.cpp:3126] Finished recovery I0714 15:08:44.147531 27242 slave.cpp:599] New master detected at master@127.0.1.1:55850 I0714 15:08:44.147562 27242 slave.cpp:675] Authenticating with master master@127.0.1.1:55850 I0714 15:08:44.147614 27242 slave.cpp:648] Detecting new master I0714 15:08:44.147652 27242 status_update_manager.cpp:167] New master detected at master@127.0.1.1:55850 I0714 15:08:44.147691 27242 authenticatee.hpp:128] Creating new client SASL connection I0714 15:08:44.148533 27235 master.cpp:3507] Authenticating slave(43)@127.0.1.1:55850 I0714 15:08:44.148666 27235 authenticator.hpp:156] Creating new server SASL connection I0714 15:08:44.149054 27242 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0714 15:08:44.149447 27242 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0714 15:08:44.149917 27236 authenticator.hpp:262] Received SASL authentication start I0714 15:08:44.149974 27236 authenticator.hpp:384] Authentication requires more steps I0714 15:08:44.150208 27242 authenticatee.hpp:265] Received SASL authentication step I0714 15:08:44.150720 27239 authenticator.hpp:290] Received SASL authentication step I0714 15:08:44.150749 27239 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0714 15:08:44.150758 27239 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0714 15:08:44.150771 27239 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0714 15:08:44.150781 27239 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0714 15:08:44.150787 27239 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.150792 27239 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.150804 27239 authenticator.hpp:376] Authentication success I0714 15:08:44.150848 27239 master.cpp:3547] Successfully authenticated principal 'test-principal' at slave(43)@127.0.1.1:55850 I0714 15:08:44.157696 27242 authenticatee.hpp:305] Authentication success I0714 15:08:44.158855 27242 slave.cpp:732] Successfully authenticated with master master@127.0.1.1:55850 I0714 15:08:44.158936 27242 slave.cpp:970] Will retry registration in 10.352612ms if necessary I0714 15:08:44.161813 27216 sched.cpp:139] Version: 0.20.0 I0714 15:08:44.162608 27236 sched.cpp:235] New master detected at master@127.0.1.1:55850 I0714 15:08:44.162637 27236 sched.cpp:285] Authenticating with master master@127.0.1.1:55850 I0714 15:08:44.162747 27236 authenticatee.hpp:128] Creating new client SASL connection I0714 15:08:44.163506 27239 master.cpp:2789] Registering slave at slave(43)@127.0.1.1:55850 (quantal) with id 20140714-150843-16842879-55850-27216-0 I0714 15:08:44.164086 27238 registrar.cpp:422] Attempting to update the 'registry' I0714 15:08:44.165694 27238 log.cpp:680] Attempting to append 295 bytes to the log I0714 15:08:44.166231 27240 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0714 15:08:44.166517 27240 replica.cpp:508] Replica received write request for position 3 I0714 15:08:44.167199 27239 master.cpp:3507] Authenticating scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.167867 27241 authenticator.hpp:156] Creating new server SASL connection I0714 15:08:44.168058 27241 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0714 15:08:44.168081 27241 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0714 15:08:44.168107 27241 authenticator.hpp:262] Received SASL authentication start I0714 15:08:44.168149 27241 authenticator.hpp:384] Authentication requires more steps I0714 15:08:44.168176 27241 authenticatee.hpp:265] Received SASL authentication step I0714 15:08:44.168215 27241 authenticator.hpp:290] Received SASL authentication step I0714 15:08:44.168233 27241 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0714 15:08:44.168793 27241 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0714 15:08:44.168820 27241 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0714 15:08:44.168834 27241 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0714 15:08:44.168840 27241 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.168845 27241 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.168858 27241 authenticator.hpp:376] Authentication success I0714 15:08:44.168895 27241 authenticatee.hpp:305] Authentication success I0714 15:08:44.168970 27241 sched.cpp:359] Successfully authenticated with master master@127.0.1.1:55850 I0714 15:08:44.168987 27241 sched.cpp:478] Sending registration request to master@127.0.1.1:55850 I0714 15:08:44.169426 27239 master.cpp:1239] Queuing up registration request from scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 because authentication is still in progress I0714 15:08:44.169958 27239 master.cpp:3547] Successfully authenticated principal 'test-principal' at scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.170440 27241 slave.cpp:970] Will retry registration in 8.76707ms if necessary I0714 15:08:44.175359 27239 master.cpp:2777] Ignoring register slave message from slave(43)@127.0.1.1:55850 (quantal) as admission is already in progress I0714 15:08:44.175916 27239 master.cpp:1247] Received registration request from scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.176298 27239 master.cpp:1207] Authorizing framework principal 'test-principal' to receive offers for role '*' I0714 15:08:44.176858 27239 master.cpp:1306] Registering framework 20140714-150843-16842879-55850-27216-0000 at scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.177408 27236 sched.cpp:409] Framework registered with 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.177443 27236 sched.cpp:423] Scheduler::registered took 12527ns I0714 15:08:44.177727 27241 hierarchical_allocator_process.hpp:331] Added framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.177747 27241 hierarchical_allocator_process.hpp:724] No resources available to allocate! I0714 15:08:44.177753 27241 hierarchical_allocator_process.hpp:686] Performed allocation for 0 slaves in 8120ns I0714 15:08:44.179908 27241 slave.cpp:970] Will retry registration in 66.781028ms if necessary I0714 15:08:44.180007 27241 master.cpp:2777] Ignoring register slave message from slave(43)@127.0.1.1:55850 (quantal) as admission is already in progress I0714 15:08:44.183082 27240 leveldb.cpp:343] Persisting action (314 bytes) to leveldb took 16.533189ms I0714 15:08:44.183125 27240 replica.cpp:676] Persisted action at 3 I0714 15:08:44.183465 27240 replica.cpp:655] Replica received learned notice for position 3 I0714 15:08:44.203276 27240 leveldb.cpp:343] Persisting action (316 bytes) to leveldb took 19.768951ms I0714 15:08:44.203376 27240 replica.cpp:676] Persisted action at 3 I0714 15:08:44.203392 27240 replica.cpp:661] Replica learned APPEND action at position 3 I0714 15:08:44.204033 27240 registrar.cpp:479] Successfully updated 'registry' I0714 15:08:44.204138 27240 log.cpp:699] Attempting to truncate the log to 3 I0714 15:08:44.204221 27240 master.cpp:2829] Registered slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) I0714 15:08:44.204241 27240 master.cpp:3975] Adding slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0714 15:08:44.204387 27240 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0714 15:08:44.204489 27240 slave.cpp:766] Registered with master master@127.0.1.1:55850; given slave ID 20140714-150843-16842879-55850-27216-0 I0714 15:08:44.204745 27240 slave.cpp:779] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/slave.info' I0714 15:08:44.204954 27240 hierarchical_allocator_process.hpp:444] Added slave 20140714-150843-16842879-55850-27216-0 (quantal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0714 15:08:44.205023 27240 hierarchical_allocator_process.hpp:750] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 to framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.205122 27240 hierarchical_allocator_process.hpp:706] Performed allocation for slave 20140714-150843-16842879-55850-27216-0 in 131192ns I0714 15:08:44.205189 27240 slave.cpp:2323] Received ping from slave-observer(32)@127.0.1.1:55850 I0714 15:08:44.205258 27240 master.hpp:801] Adding offer 20140714-150843-16842879-55850-27216-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.205303 27240 master.cpp:3454] Sending 1 offers to framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.205469 27240 sched.cpp:546] Scheduler::resourceOffers took 23591ns I0714 15:08:44.206351 27241 replica.cpp:508] Replica received write request for position 4 I0714 15:08:44.208353 27237 master.hpp:811] Removing offer 20140714-150843-16842879-55850-27216-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.208436 27237 master.cpp:2133] Processing reply for offers: [ 20140714-150843-16842879-55850-27216-0 ] on slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) for framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.208472 27237 master.cpp:2219] Authorizing framework principal 'test-principal' to launch task 4a6783aa-8d07-46e3-8399-2a5d047f0021 as user 'jenkins' I0714 15:08:44.208909 27237 master.hpp:773] Adding task 4a6783aa-8d07-46e3-8399-2a5d047f0021 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.208947 27237 master.cpp:2285] Launching task 4a6783aa-8d07-46e3-8399-2a5d047f0021 of framework 20140714-150843-16842879-55850-27216-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) I0714 15:08:44.209090 27237 slave.cpp:1001] Got assigned task 4a6783aa-8d07-46e3-8399-2a5d047f0021 for framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.209190 27237 slave.cpp:3398] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/frameworks/20140714-150843-16842879-55850-27216-0000/framework.info' I0714 15:08:44.209413 27237 slave.cpp:3405] Checkpointing framework pid 'scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850' to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/frameworks/20140714-150843-16842879-55850-27216-0000/framework.pid' I0714 15:08...
Story points: 1