Project: mesos
Issue MESOS-2302: FaultToleranceTest.SchedulerFailoverFrameworkMessage is flaky.
Bad Run: {noformat} [ RUN ] FaultToleranceTest.SchedulerFailoverFrameworkMessage Using temporary directory '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_f3jYkr' I0123 18:50:11.669674 15688 leveldb.cpp:176] Opened db in 31.920683ms I0123 18:50:11.678328 15688 leveldb.cpp:183] Compacted db in 8.580569ms I0123 18:50:11.678455 15688 leveldb.cpp:198] Created db iterator in 38478ns I0123 18:50:11.678478 15688 leveldb.cpp:204] Seeked to beginning of db in 3057ns I0123 18:50:11.678489 15688 leveldb.cpp:273] Iterated through 0 keys in the db in 427ns I0123 18:50:11.678539 15688 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0123 18:50:11.682271 15705 recover.cpp:449] Starting replica recovery I0123 18:50:11.682634 15705 recover.cpp:475] Replica is in EMPTY status I0123 18:50:11.684389 15708 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0123 18:50:11.685132 15708 recover.cpp:195] Received a recover response from a replica in EMPTY status I0123 18:50:11.689842 15708 recover.cpp:566] Updating replica status to STARTING I0123 18:50:11.702548 15708 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 12.484558ms I0123 18:50:11.702615 15708 replica.cpp:323] Persisted replica status to STARTING I0123 18:50:11.703531 15708 recover.cpp:475] Replica is in STARTING status I0123 18:50:11.705080 15704 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0123 18:50:11.712587 15708 recover.cpp:195] Received a recover response from a replica in STARTING status I0123 18:50:11.722898 15708 recover.cpp:566] Updating replica status to VOTING I0123 18:50:11.725427 15703 master.cpp:262] Master 20150123-185011-16777343-37526-15688 (localhost.localdomain) started on 127.0.0.1:37526 W0123 18:50:11.725464 15703 master.cpp:266] ************************************************** Master bound to loopback interface! Cannot communicate with remote schedulers or slaves. You might want to set '--ip' flag to a routable IP address. ************************************************** I0123 18:50:11.725502 15703 master.cpp:308] Master only allowing authenticated frameworks to register I0123 18:50:11.725513 15703 master.cpp:313] Master only allowing authenticated slaves to register I0123 18:50:11.725543 15703 credentials.hpp:36] Loading credentials for authentication from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_f3jYkr/credentials' I0123 18:50:11.725774 15703 master.cpp:357] Authorization enabled I0123 18:50:11.728428 15707 whitelist_watcher.cpp:65] No whitelist given I0123 18:50:11.729169 15707 master.cpp:1219] The newly elected leader is master@127.0.0.1:37526 with id 20150123-185011-16777343-37526-15688 I0123 18:50:11.729200 15707 master.cpp:1232] Elected as the leading master! I0123 18:50:11.729223 15707 master.cpp:1050] Recovering from registrar I0123 18:50:11.729595 15706 registrar.cpp:313] Recovering registrar I0123 18:50:11.730715 15703 hierarchical_allocator_process.hpp:285] Initialized hierarchical allocator process I0123 18:50:11.737431 15708 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 14.259597ms I0123 18:50:11.737511 15708 replica.cpp:323] Persisted replica status to VOTING I0123 18:50:11.737768 15708 recover.cpp:580] Successfully joined the Paxos group I0123 18:50:11.737977 15708 recover.cpp:464] Recover process terminated I0123 18:50:11.739083 15706 log.cpp:660] Attempting to start the writer I0123 18:50:11.741236 15706 replica.cpp:477] Replica received implicit promise request with proposal 1 I0123 18:50:11.750435 15706 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 8.813783ms I0123 18:50:11.750514 15706 replica.cpp:345] Persisted promised to 1 I0123 18:50:11.752239 15708 coordinator.cpp:230] Coordinator attemping to fill missing position I0123 18:50:11.754176 15706 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0123 18:50:11.763464 15706 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 8.799822ms I0123 18:50:11.763535 15706 replica.cpp:679] Persisted action at 0 I0123 18:50:11.765697 15709 replica.cpp:511] Replica received write request for position 0 I0123 18:50:11.766293 15709 leveldb.cpp:438] Reading position from leveldb took 54028ns I0123 18:50:11.776468 15709 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 9.789169ms I0123 18:50:11.776561 15709 replica.cpp:679] Persisted action at 0 I0123 18:50:11.777515 15709 replica.cpp:658] Replica received learned notice for position 0 I0123 18:50:11.785459 15709 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 7.897242ms I0123 18:50:11.785531 15709 replica.cpp:679] Persisted action at 0 I0123 18:50:11.785565 15709 replica.cpp:664] Replica learned NOP action at position 0 I0123 18:50:11.786633 15709 log.cpp:676] Writer started with ending position 0 I0123 18:50:11.788460 15709 leveldb.cpp:438] Reading position from leveldb took 266087ns I0123 18:50:11.801141 15709 registrar.cpp:346] Successfully fetched the registry (0B) in 71.491072ms I0123 18:50:11.801300 15709 registrar.cpp:445] Applied 1 operations in 41795ns; attempting to update the 'registry' I0123 18:50:11.805186 15707 log.cpp:684] Attempting to append 136 bytes to the log I0123 18:50:11.805454 15707 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0123 18:50:11.806677 15703 replica.cpp:511] Replica received write request for position 1 I0123 18:50:11.815621 15703 leveldb.cpp:343] Persisting action (155 bytes) to leveldb took 8.89177ms I0123 18:50:11.815692 15703 replica.cpp:679] Persisted action at 1 I0123 18:50:11.817358 15704 replica.cpp:658] Replica received learned notice for position 1 I0123 18:50:11.825014 15704 leveldb.cpp:343] Persisting action (157 bytes) to leveldb took 7.578558ms I0123 18:50:11.825088 15704 replica.cpp:679] Persisted action at 1 I0123 18:50:11.825124 15704 replica.cpp:664] Replica learned APPEND action at position 1 I0123 18:50:11.827008 15705 registrar.cpp:490] Successfully updated the 'registry' in 25.629952ms I0123 18:50:11.827143 15705 registrar.cpp:376] Successfully recovered registrar I0123 18:50:11.827517 15705 master.cpp:1077] Recovered 0 slaves from the Registry (98B) ; allowing 10mins for slaves to re-register I0123 18:50:11.828515 15704 log.cpp:703] Attempting to truncate the log to 1 I0123 18:50:11.829074 15704 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0123 18:50:11.830546 15709 replica.cpp:511] Replica received write request for position 2 I0123 18:50:11.837752 15709 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 7.142431ms I0123 18:50:11.837826 15709 replica.cpp:679] Persisted action at 2 I0123 18:50:11.839334 15709 replica.cpp:658] Replica received learned notice for position 2 I0123 18:50:11.847069 15709 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 7.116607ms I0123 18:50:11.847214 15709 leveldb.cpp:401] Deleting ~1 keys from leveldb took 74008ns I0123 18:50:11.847241 15709 replica.cpp:679] Persisted action at 2 I0123 18:50:11.847295 15709 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0123 18:50:11.870337 15710 slave.cpp:173] Slave started on 94)@127.0.0.1:37526 W0123 18:50:11.870980 15710 slave.cpp:176] ************************************************** Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address. ************************************************** I0123 18:50:11.871412 15710 credentials.hpp:84] Loading credential for authentication from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_TB8Rh3/credential' I0123 18:50:11.871819 15710 slave.cpp:282] Slave using credential for: test-principal I0123 18:50:11.873178 15710 slave.cpp:300] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0123 18:50:11.873620 15710 slave.cpp:329] Slave hostname: localhost.localdomain I0123 18:50:11.873837 15710 slave.cpp:330] Slave checkpoint: false W0123 18:50:11.874068 15710 slave.cpp:332] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0123 18:50:11.879103 15705 state.cpp:33] Recovering state from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_TB8Rh3/meta' W0123 18:50:11.882972 15688 sched.cpp:1246] ************************************************** Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address. ************************************************** I0123 18:50:11.884106 15709 status_update_manager.cpp:197] Recovering status update manager I0123 18:50:11.884703 15710 slave.cpp:3519] Finished recovery I0123 18:50:11.892076 15704 status_update_manager.cpp:171] Pausing sending status updates I0123 18:50:11.892590 15710 slave.cpp:613] New master detected at master@127.0.0.1:37526 I0123 18:50:11.892937 15710 slave.cpp:676] Authenticating with master master@127.0.0.1:37526 I0123 18:50:11.893165 15710 slave.cpp:681] Using default CRAM-MD5 authenticatee I0123 18:50:11.893754 15708 authenticatee.hpp:138] Creating new client SASL connection I0123 18:50:11.894120 15708 master.cpp:4129] Authenticating slave(94)@127.0.0.1:37526 I0123 18:50:11.894153 15708 master.cpp:4140] Using default CRAM-MD5 authenticator I0123 18:50:11.894628 15708 authenticator.hpp:170] Creating new server SASL connection I0123 18:50:11.894913 15708 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0123 18:50:11.894942 15708 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0123 18:50:11.895043 15708 authenticator.hpp:276] Received SASL authentication start I0123 18:50:11.895095 15708 authenticator.hpp:398] Authentication requires more steps I0123 18:50:11.895165 15708 authenticatee.hpp:275] Received SASL authentication step I0123 18:50:11.895261 15708 authenticator.hpp:304] Received SASL authentication step I0123 18:50:11.895292 15708 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0123 18:50:11.895305 15708 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0123 18:50:11.895354 15708 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0123 18:50:11.895881 15710 slave.cpp:649] Detecting new master I0123 18:50:11.898449 15708 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0123 18:50:11.899024 15708 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0123 18:50:11.899106 15708 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0123 18:50:11.899190 15708 authenticator.hpp:390] Authentication success I0123 18:50:11.899569 15706 authenticatee.hpp:315] Authentication success I0123 18:50:11.902299 15706 slave.cpp:747] Successfully authenticated with master master@127.0.0.1:37526 I0123 18:50:11.902847 15706 slave.cpp:1075] Will retry registration in 19.809649ms if necessary I0123 18:50:11.903264 15705 master.cpp:3214] Queuing up registration request from slave(94)@127.0.0.1:37526 because authentication is still in progress I0123 18:50:11.903497 15705 master.cpp:4187] Successfully authenticated principal 'test-principal' at slave(94)@127.0.0.1:37526 I0123 18:50:11.903940 15705 master.cpp:3275] Registering slave at slave(94)@127.0.0.1:37526 (localhost.localdomain) with id 20150123-185011-16777343-37526-15688-S0 I0123 18:50:11.904398 15705 registrar.cpp:445] Applied 1 operations in 63679ns; attempting to update the 'registry' I0123 18:50:11.917883 15688 sched.cpp:151] Version: 0.22.0 I0123 18:50:11.919347 15703 log.cpp:684] Attempting to append 315 bytes to the log I0123 18:50:11.921039 15703 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0123 18:50:11.919992 15706 sched.cpp:248] New master detected at master@127.0.0.1:37526 I0123 18:50:11.921352 15706 sched.cpp:304] Authenticating with master master@127.0.0.1:37526 I0123 18:50:11.921408 15706 sched.cpp:311] Using default CRAM-MD5 authenticatee I0123 18:50:11.921773 15706 authenticatee.hpp:138] Creating new client SASL connection I0123 18:50:11.922266 15706 master.cpp:4129] Authenticating scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.922301 15706 master.cpp:4140] Using default CRAM-MD5 authenticator I0123 18:50:11.923928 15703 replica.cpp:511] Replica received write request for position 3 I0123 18:50:11.924285 15707 authenticator.hpp:170] Creating new server SASL connection I0123 18:50:11.925091 15707 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0123 18:50:11.925122 15707 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0123 18:50:11.925194 15707 authenticator.hpp:276] Received SASL authentication start I0123 18:50:11.925257 15707 authenticator.hpp:398] Authentication requires more steps I0123 18:50:11.925325 15707 authenticatee.hpp:275] Received SASL authentication step I0123 18:50:11.925442 15707 authenticator.hpp:304] Received SASL authentication step I0123 18:50:11.925473 15707 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0123 18:50:11.925487 15707 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0123 18:50:11.925532 15707 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0123 18:50:11.925559 15707 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0123 18:50:11.925571 15707 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0123 18:50:11.925580 15707 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0123 18:50:11.925595 15707 authenticator.hpp:390] Authentication success I0123 18:50:11.925695 15707 authenticatee.hpp:315] Authentication success I0123 18:50:11.925792 15707 master.cpp:4187] Successfully authenticated principal 'test-principal' at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.926127 15707 sched.cpp:392] Successfully authenticated with master master@127.0.0.1:37526 I0123 18:50:11.926154 15707 sched.cpp:515] Sending registration request to master@127.0.0.1:37526 I0123 18:50:11.926215 15707 sched.cpp:548] Will retry registration in 866.81063ms if necessary I0123 18:50:11.926640 15707 master.cpp:1420] Received registration request for framework 'default' at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.926960 15707 master.cpp:1298] Authorizing framework principal 'test-principal' to receive offers for role '*' I0123 18:50:11.927691 15707 master.cpp:1484] Registering framework 20150123-185011-16777343-37526-15688-0000 (default) at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.928292 15708 hierarchical_allocator_process.hpp:319] Added framework 20150123-185011-16777343-37526-15688-0000 I0123 18:50:11.928326 15708 hierarchical_allocator_process.hpp:839] No resources available to allocate! I0123 18:50:11.928340 15708 hierarchical_allocator_process.hpp:746] Performed allocation for 0 slaves in 21080ns I0123 18:50:11.934458 15707 sched.cpp:442] Framework registered with 20150123-185011-16777343-37526-15688-0000 I0123 18:50:11.934927 15707 sched.cpp:456] Scheduler::registered took 112885ns I0123 18:50:11.935747 15709 slave.cpp:1075] Will retry registration in 19.609252ms if necessary I0123 18:50:11.935981 15709 master.cpp:3263] Ignoring register slave message from slave(94)@127.0.0.1:37526 (localhost.localdomain) as admission is already in progress I0123 18:50:11.938997 15703 leveldb.cpp:343] Persisting action (334 bytes) to leveldb took 10.171709ms I0123 18:50:11.939049 15703 replica.cpp:679] Persisted action at 3 I0123 18:50:11.940630 15709 replica.cpp:658] Replica received learned notice for position 3 I0123 18:50:11.945473 15709 leveldb.cpp:343] Persisting action (336 bytes) to leveldb took 4.804742ms I0123 18:50:11.945521 15709 replica.cpp:679] Persisted action at 3 I0123 18:50:11.945550 15709 replica.cpp:664] Replica learned APPEND action at position 3 I0123 18:50:11.947105 15709 registrar.cpp:490] Successfully updated the 'registry' in 42.637056ms I0123 18:50:11.948020 15703 master.cpp:3329] Registered slave 20150123-185011-16777343-37526-15688-S0 at slave(94)@127.0.0.1:37526 (localhost.localdomain) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0123 18:50:11.948318 15703 hierarchical_allocator_process.hpp:453] Added slave 20150123-185011-16777343-37526-15688-S0 (localhost.localdomain) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0123 18:50:11.948719 15703 hierarchical_allocator_process.hpp:764] Performed allocation for slave 20150123-185011-16777343-37526-15688-S0 in 355831ns I0123 18:50:11.948813 15703 slave.cpp:781] Registered with master master@127.0.0.1:37526; given slave ID 20150123-185011-16777343-37526-15688-S0 I0123 18:50:11.948969 15703 slave.cpp:2588] Received ping from slave-observer(92)@127.0.0.1:37526 I0123 18:50:11.949324 15703 master.cpp:4071] Sending 1 offers to framework 20150123-185011-16777343-37526-15688-0000 (default) at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.949571 15706 status_update_manager.cpp:178] Resuming sending status updates I0123 18:50:11.950023 15709 log.cpp:703] Attempting to truncate the log to 3 I0123 18:50:11.950810 15705 sched.cpp:605] Scheduler::resourceOffers took 135580ns I0123 18:50:11.952793 15708 master.cpp:2677] Processing ACCEPT call for offers: [ 20150123-185011-16777343-37526-15688-O0 ] on slave 20150123-185011-16777343-37526-15688-S0 at slave(94)@127.0.0.1:37526 (localhost.localdomain) for framework 20150123-185011-16777343-37526-15688-0000 (default) at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 I0123 18:50:11.952852 15708 master.cpp:2513] Authorizing framework principal 'test-principal' to launch task 1 as user 'jenkins' W0123 18:50:11.954649 15708 master.cpp:2130] Executor default for task 1 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0123 18:50:11.954988 15708 master.cpp:2142] Executor default for task 1 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0123 18:50:11.955579 15708 master.hpp:782] Adding task 1 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20150123-185011-16777343-37526-15688-S0 (localhost.localdomain) I0123 18:50:11.956035 15703 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0123 18:50:11.957592 15704 replica.cpp:511] Replica received write request for position 4 I0123 18:50:11.958485 15708 master.cpp:2885] Launching task 1 of framework 20150123-185011-16777343-37526-15688-0000 (default) at scheduler-2cecb105-ca23-4048-9707-12b1e4422e11@127.0.0.1:37526 with resources cpus(...
Story points: 1