Project: mesos
Issue MESOS-2324: MasterAllocatorTest/0.OutOfOrderDispatch is flaky
{noformat:title=} [ RUN ] MasterAllocatorTest/0.OutOfOrderDispatch Using temporary directory '/tmp/MasterAllocatorTest_0_OutOfOrderDispatch_kjLb9b' I0206 07:55:44.084333 15065 leveldb.cpp:175] Opened db in 25.006293ms I0206 07:55:44.089635 15065 leveldb.cpp:182] Compacted db in 5.256332ms I0206 07:55:44.089695 15065 leveldb.cpp:197] Created db iterator in 23534ns I0206 07:55:44.089710 15065 leveldb.cpp:203] Seeked to beginning of db in 2175ns I0206 07:55:44.089720 15065 leveldb.cpp:272] Iterated through 0 keys in the db in 417ns I0206 07:55:44.089781 15065 replica.cpp:743] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0206 07:55:44.093750 15086 recover.cpp:448] Starting replica recovery I0206 07:55:44.094044 15086 recover.cpp:474] Replica is in EMPTY status I0206 07:55:44.095473 15086 replica.cpp:640] Replica in EMPTY status received a broadcasted recover request I0206 07:55:44.095724 15086 recover.cpp:194] Received a recover response from a replica in EMPTY status I0206 07:55:44.096097 15086 recover.cpp:565] Updating replica status to STARTING I0206 07:55:44.106575 15086 leveldb.cpp:305] Persisting metadata (8 bytes) to leveldb took 10.289939ms I0206 07:55:44.106613 15086 replica.cpp:322] Persisted replica status to STARTING I0206 07:55:44.108144 15086 recover.cpp:474] Replica is in STARTING status I0206 07:55:44.109122 15086 replica.cpp:640] Replica in STARTING status received a broadcasted recover request I0206 07:55:44.110879 15091 recover.cpp:194] Received a recover response from a replica in STARTING status I0206 07:55:44.117267 15087 recover.cpp:565] Updating replica status to VOTING I0206 07:55:44.124771 15087 leveldb.cpp:305] Persisting metadata (8 bytes) to leveldb took 2.66794ms I0206 07:55:44.124814 15087 replica.cpp:322] Persisted replica status to VOTING I0206 07:55:44.124948 15087 recover.cpp:579] Successfully joined the Paxos group I0206 07:55:44.125095 15087 recover.cpp:463] Recover process terminated I0206 07:55:44.126204 15087 master.cpp:344] Master 20150206-075544-16842879-38895-15065 (utopic) started on 127.0.1.1:38895 I0206 07:55:44.126268 15087 master.cpp:390] Master only allowing authenticated frameworks to register I0206 07:55:44.126281 15087 master.cpp:395] Master only allowing authenticated slaves to register I0206 07:55:44.126307 15087 credentials.hpp:35] Loading credentials for authentication from '/tmp/MasterAllocatorTest_0_OutOfOrderDispatch_kjLb9b/credentials' I0206 07:55:44.126683 15087 master.cpp:439] Authorization enabled I0206 07:55:44.129329 15086 master.cpp:1350] The newly elected leader is master@127.0.1.1:38895 with id 20150206-075544-16842879-38895-15065 I0206 07:55:44.129361 15086 master.cpp:1363] Elected as the leading master! I0206 07:55:44.129389 15086 master.cpp:1181] Recovering from registrar I0206 07:55:44.129653 15088 registrar.cpp:312] Recovering registrar I0206 07:55:44.130859 15088 log.cpp:659] Attempting to start the writer I0206 07:55:44.132334 15088 replica.cpp:476] Replica received implicit promise request with proposal 1 I0206 07:55:44.135187 15088 leveldb.cpp:305] Persisting metadata (8 bytes) to leveldb took 2.825465ms I0206 07:55:44.135390 15088 replica.cpp:344] Persisted promised to 1 I0206 07:55:44.138062 15091 coordinator.cpp:229] Coordinator attemping to fill missing position I0206 07:55:44.139576 15091 replica.cpp:377] Replica received explicit promise request for position 0 with proposal 2 I0206 07:55:44.142156 15091 leveldb.cpp:342] Persisting action (8 bytes) to leveldb took 2.545543ms I0206 07:55:44.142189 15091 replica.cpp:678] Persisted action at 0 I0206 07:55:44.143414 15091 replica.cpp:510] Replica received write request for position 0 I0206 07:55:44.143468 15091 leveldb.cpp:437] Reading position from leveldb took 28872ns I0206 07:55:44.145982 15091 leveldb.cpp:342] Persisting action (14 bytes) to leveldb took 2.480277ms I0206 07:55:44.146015 15091 replica.cpp:678] Persisted action at 0 I0206 07:55:44.147050 15089 replica.cpp:657] Replica received learned notice for position 0 I0206 07:55:44.154364 15089 leveldb.cpp:342] Persisting action (16 bytes) to leveldb took 7.281644ms I0206 07:55:44.154400 15089 replica.cpp:678] Persisted action at 0 I0206 07:55:44.154422 15089 replica.cpp:663] Replica learned NOP action at position 0 I0206 07:55:44.155506 15091 log.cpp:675] Writer started with ending position 0 I0206 07:55:44.156746 15091 leveldb.cpp:437] Reading position from leveldb took 30248ns I0206 07:55:44.173681 15091 registrar.cpp:345] Successfully fetched the registry (0B) in 43.977984ms I0206 07:55:44.173821 15091 registrar.cpp:444] Applied 1 operations in 30768ns; attempting to update the 'registry' I0206 07:55:44.176213 15086 log.cpp:683] Attempting to append 119 bytes to the log I0206 07:55:44.176426 15086 coordinator.cpp:339] Coordinator attempting to write APPEND action at position 1 I0206 07:55:44.177608 15088 replica.cpp:510] Replica received write request for position 1 I0206 07:55:44.180059 15088 leveldb.cpp:342] Persisting action (136 bytes) to leveldb took 2.415145ms I0206 07:55:44.180094 15088 replica.cpp:678] Persisted action at 1 I0206 07:55:44.181324 15084 replica.cpp:657] Replica received learned notice for position 1 I0206 07:55:44.183831 15084 leveldb.cpp:342] Persisting action (138 bytes) to leveldb took 2.473724ms I0206 07:55:44.183866 15084 replica.cpp:678] Persisted action at 1 I0206 07:55:44.183887 15084 replica.cpp:663] Replica learned APPEND action at position 1 I0206 07:55:44.185510 15084 registrar.cpp:489] Successfully updated the 'registry' in 11.619072ms I0206 07:55:44.185678 15086 log.cpp:702] Attempting to truncate the log to 1 I0206 07:55:44.186111 15086 coordinator.cpp:339] Coordinator attempting to write TRUNCATE action at position 2 I0206 07:55:44.186944 15086 replica.cpp:510] Replica received write request for position 2 I0206 07:55:44.187492 15084 registrar.cpp:375] Successfully recovered registrar I0206 07:55:44.188016 15087 master.cpp:1208] Recovered 0 slaves from the Registry (83B) ; allowing 10mins for slaves to re-register I0206 07:55:44.189678 15086 leveldb.cpp:342] Persisting action (16 bytes) to leveldb took 2.702559ms I0206 07:55:44.189713 15086 replica.cpp:678] Persisted action at 2 I0206 07:55:44.190620 15086 replica.cpp:657] Replica received learned notice for position 2 I0206 07:55:44.193383 15086 leveldb.cpp:342] Persisting action (18 bytes) to leveldb took 2.737088ms I0206 07:55:44.193455 15086 leveldb.cpp:400] Deleting ~1 keys from leveldb took 37762ns I0206 07:55:44.193475 15086 replica.cpp:678] Persisted action at 2 I0206 07:55:44.193496 15086 replica.cpp:663] Replica learned TRUNCATE action at position 2 I0206 07:55:44.200028 15065 containerizer.cpp:102] Using isolation: posix/cpu,posix/mem I0206 07:55:44.212924 15088 slave.cpp:172] Slave started on 46)@127.0.1.1:38895 I0206 07:55:44.213762 15088 credentials.hpp:83] Loading credential for authentication from '/tmp/MasterAllocatorTest_0_OutOfOrderDispatch_RuNyVQ/credential' I0206 07:55:44.214251 15088 slave.cpp:281] Slave using credential for: test-principal I0206 07:55:44.214653 15088 slave.cpp:299] Slave resources: cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000] I0206 07:55:44.214918 15088 slave.cpp:328] Slave hostname: utopic I0206 07:55:44.215116 15088 slave.cpp:329] Slave checkpoint: false W0206 07:55:44.215332 15088 slave.cpp:331] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0206 07:55:44.217061 15090 state.cpp:32] Recovering state from '/tmp/MasterAllocatorTest_0_OutOfOrderDispatch_RuNyVQ/meta' I0206 07:55:44.235409 15088 status_update_manager.cpp:196] Recovering status update manager I0206 07:55:44.235601 15088 containerizer.cpp:299] Recovering containerizer I0206 07:55:44.236486 15088 slave.cpp:3526] Finished recovery I0206 07:55:44.237709 15087 status_update_manager.cpp:170] Pausing sending status updates I0206 07:55:44.237890 15088 slave.cpp:620] New master detected at master@127.0.1.1:38895 I0206 07:55:44.241575 15088 slave.cpp:683] Authenticating with master master@127.0.1.1:38895 I0206 07:55:44.247459 15088 slave.cpp:688] Using default CRAM-MD5 authenticatee I0206 07:55:44.248617 15089 authenticatee.hpp:137] Creating new client SASL connection I0206 07:55:44.249099 15089 master.cpp:3788] Authenticating slave(46)@127.0.1.1:38895 I0206 07:55:44.249137 15089 master.cpp:3799] Using default CRAM-MD5 authenticator I0206 07:55:44.249728 15089 authenticator.hpp:169] Creating new server SASL connection I0206 07:55:44.250285 15089 authenticatee.hpp:228] Received SASL authentication mechanisms: CRAM-MD5 I0206 07:55:44.250496 15089 authenticatee.hpp:254] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 07:55:44.250452 15088 slave.cpp:656] Detecting new master I0206 07:55:44.251063 15091 authenticator.hpp:275] Received SASL authentication start I0206 07:55:44.251124 15091 authenticator.hpp:397] Authentication requires more steps I0206 07:55:44.251256 15089 authenticatee.hpp:274] Received SASL authentication step I0206 07:55:44.251451 15090 authenticator.hpp:303] Received SASL authentication step I0206 07:55:44.251575 15090 authenticator.hpp:389] Authentication success I0206 07:55:44.251687 15090 master.cpp:3846] Successfully authenticated principal 'test-principal' at slave(46)@127.0.1.1:38895 I0206 07:55:44.253306 15089 authenticatee.hpp:314] Authentication success I0206 07:55:44.258015 15089 slave.cpp:754] Successfully authenticated with master master@127.0.1.1:38895 I0206 07:55:44.258468 15089 master.cpp:2913] Registering slave at slave(46)@127.0.1.1:38895 (utopic) with id 20150206-075544-16842879-38895-15065-S0 I0206 07:55:44.259028 15089 registrar.cpp:444] Applied 1 operations in 88902ns; attempting to update the 'registry' I0206 07:55:44.269492 15065 sched.cpp:149] Version: 0.22.0 I0206 07:55:44.270539 15090 sched.cpp:246] New master detected at master@127.0.1.1:38895 I0206 07:55:44.270614 15090 sched.cpp:302] Authenticating with master master@127.0.1.1:38895 I0206 07:55:44.270634 15090 sched.cpp:309] Using default CRAM-MD5 authenticatee I0206 07:55:44.270900 15090 authenticatee.hpp:137] Creating new client SASL connection I0206 07:55:44.272300 15089 log.cpp:683] Attempting to append 285 bytes to the log I0206 07:55:44.272552 15089 coordinator.cpp:339] Coordinator attempting to write APPEND action at position 3 I0206 07:55:44.273609 15086 master.cpp:3788] Authenticating scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.273643 15086 master.cpp:3799] Using default CRAM-MD5 authenticator I0206 07:55:44.273955 15086 authenticator.hpp:169] Creating new server SASL connection I0206 07:55:44.274617 15090 authenticatee.hpp:228] Received SASL authentication mechanisms: CRAM-MD5 I0206 07:55:44.274813 15090 authenticatee.hpp:254] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 07:55:44.275171 15088 authenticator.hpp:275] Received SASL authentication start I0206 07:55:44.275215 15088 authenticator.hpp:397] Authentication requires more steps I0206 07:55:44.275408 15090 authenticatee.hpp:274] Received SASL authentication step I0206 07:55:44.275696 15084 authenticator.hpp:303] Received SASL authentication step I0206 07:55:44.275774 15084 authenticator.hpp:389] Authentication success I0206 07:55:44.275876 15084 master.cpp:3846] Successfully authenticated principal 'test-principal' at scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.277593 15090 authenticatee.hpp:314] Authentication success I0206 07:55:44.278201 15086 sched.cpp:390] Successfully authenticated with master master@127.0.1.1:38895 I0206 07:55:44.278548 15086 master.cpp:1568] Received registration request for framework 'framework1' at scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.278642 15086 master.cpp:1429] Authorizing framework principal 'test-principal' to receive offers for role '*' I0206 07:55:44.279157 15086 master.cpp:1632] Registering framework 20150206-075544-16842879-38895-15065-0000 (framework1) at scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.280081 15086 sched.cpp:440] Framework registered with 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.280320 15086 hierarchical_allocator_process.hpp:318] Added framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.281411 15089 replica.cpp:510] Replica received write request for position 3 I0206 07:55:44.282289 15085 master.cpp:2901] Ignoring register slave message from slave(46)@127.0.1.1:38895 (utopic) as admission is already in progress I0206 07:55:44.284984 15089 leveldb.cpp:342] Persisting action (304 bytes) to leveldb took 3.368213ms I0206 07:55:44.285020 15089 replica.cpp:678] Persisted action at 3 I0206 07:55:44.285893 15089 replica.cpp:657] Replica received learned notice for position 3 I0206 07:55:44.288350 15089 leveldb.cpp:342] Persisting action (306 bytes) to leveldb took 2.430449ms I0206 07:55:44.288384 15089 replica.cpp:678] Persisted action at 3 I0206 07:55:44.288405 15089 replica.cpp:663] Replica learned APPEND action at position 3 I0206 07:55:44.290154 15089 registrar.cpp:489] Successfully updated the 'registry' in 31.046912ms I0206 07:55:44.290307 15085 log.cpp:702] Attempting to truncate the log to 3 I0206 07:55:44.290671 15085 coordinator.cpp:339] Coordinator attempting to write TRUNCATE action at position 4 I0206 07:55:44.291482 15085 replica.cpp:510] Replica received write request for position 4 I0206 07:55:44.292559 15087 master.cpp:2970] Registered slave 20150206-075544-16842879-38895-15065-S0 at slave(46)@127.0.1.1:38895 (utopic) with cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000] I0206 07:55:44.292940 15087 slave.cpp:788] Registered with master master@127.0.1.1:38895; given slave ID 20150206-075544-16842879-38895-15065-S0 I0206 07:55:44.293298 15087 hierarchical_allocator_process.hpp:450] Added slave 20150206-075544-16842879-38895-15065-S0 (utopic) with cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000] available) I0206 07:55:44.293684 15087 status_update_manager.cpp:177] Resuming sending status updates I0206 07:55:44.294085 15087 master.cpp:3730] Sending 1 offers to framework 20150206-075544-16842879-38895-15065-0000 (framework1) at scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.299957 15085 leveldb.cpp:342] Persisting action (16 bytes) to leveldb took 8.442691ms I0206 07:55:44.300165 15085 replica.cpp:678] Persisted action at 4 I0206 07:55:44.300698 15065 sched.cpp:1468] Asked to stop the driver I0206 07:55:44.301127 15090 sched.cpp:806] Stopping framework '20150206-075544-16842879-38895-15065-0000' I0206 07:55:44.301503 15090 master.cpp:1892] Asked to unregister framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.301535 15090 master.cpp:4158] Removing framework 20150206-075544-16842879-38895-15065-0000 (framework1) at scheduler-d6cac0a1-d461-4a05-b19d-5cbdae239eb0@127.0.1.1:38895 I0206 07:55:44.302376 15090 slave.cpp:1592] Asked to shut down framework 20150206-075544-16842879-38895-15065-0000 by master@127.0.1.1:38895 W0206 07:55:44.302407 15090 slave.cpp:1607] Cannot shut down unknown framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.302814 15090 hierarchical_allocator_process.hpp:397] Deactivated framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.302947 15090 hierarchical_allocator_process.hpp:351] Removed framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.309281 15086 hierarchical_allocator_process.hpp:642] Recovered cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000] (total allocatable: cpus(*):2; mem(*):1024; disk(*):24988; ports(*):[31000-32000]) on slave 20150206-075544-16842879-38895-15065-S0 from framework 20150206-075544-16842879-38895-15065-0000 I0206 07:55:44.310158 15084 replica.cpp:657] Replica received learned notice for position 4 I0206 07:55:44.313246 15084 leveldb.cpp:342] Persisting action (18 bytes) to leveldb took 3.055049ms I0206 07:55:44.313328 15084 leveldb.cpp:400] Deleting ~2 keys from leveldb took 45270ns I0206 07:55:44.313349 15084 replica.cpp:678] Persisted action at 4 I0206 07:55:44.313374 15084 replica.cpp:663] Replica learned TRUNCATE action at position 4 I0206 07:55:44.329591 15065 sched.cpp:149] Version: 0.22.0 I0206 07:55:44.330258 15088 sched.cpp:246] New master detected at master@127.0.1.1:38895 I0206 07:55:44.330346 15088 sched.cpp:302] Authenticating with master master@127.0.1.1:38895 I0206 07:55:44.330368 15088 sched.cpp:309] Using default CRAM-MD5 authenticatee I0206 07:55:44.330652 15088 authenticatee.hpp:137] Creating new client SASL connection I0206 07:55:44.331403 15088 master.cpp:3788] Authenticating scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 I0206 07:55:44.331717 15088 master.cpp:3799] Using default CRAM-MD5 authenticator I0206 07:55:44.332293 15088 authenticator.hpp:169] Creating new server SASL connection I0206 07:55:44.332655 15088 authenticatee.hpp:228] Received SASL authentication mechanisms: CRAM-MD5 I0206 07:55:44.332684 15088 authenticatee.hpp:254] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 07:55:44.332792 15088 authenticator.hpp:275] Received SASL authentication start I0206 07:55:44.332835 15088 authenticator.hpp:397] Authentication requires more steps I0206 07:55:44.332903 15088 authenticatee.hpp:274] Received SASL authentication step I0206 07:55:44.332983 15088 authenticator.hpp:303] Received SASL authentication step I0206 07:55:44.333056 15088 authenticator.hpp:389] Authentication success I0206 07:55:44.333153 15088 authenticatee.hpp:314] Authentication success I0206 07:55:44.333297 15091 master.cpp:3846] Successfully authenticated principal 'test-principal' at scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 I0206 07:55:44.334326 15087 sched.cpp:390] Successfully authenticated with master master@127.0.1.1:38895 I0206 07:55:44.334645 15087 master.cpp:1568] Received registration request for framework 'framework2' at scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 I0206 07:55:44.334722 15087 master.cpp:1429] Authorizing framework principal 'test-principal' to receive offers for role '*' I0206 07:55:44.335153 15087 master.cpp:1632] Registering framework 20150206-075544-16842879-38895-15065-0001 (framework2) at scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 I0206 07:55:44.336019 15087 sched.cpp:440] Framework registered with 20150206-075544-16842879-38895-15065-0001 I0206 07:55:44.336156 15087 hierarchical_allocator_process.hpp:318] Added framework 20150206-075544-16842879-38895-15065-0001 I0206 07:55:44.336796 15087 master.cpp:3730] Sending 1 offers to framework 20150206-075544-16842879-38895-15065-0001 (framework2) at scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 I0206 07:55:44.337725 15065 sched.cpp:1468] Asked to stop the driver I0206 07:55:44.338002 15086 sched.cpp:806] Stopping framework '20150206-075544-16842879-38895-15065-0001' I0206 07:55:44.338297 15090 master.cpp:1892] Asked to unregister framework 20150206-075544-16842879-38895-15065-0001 I0206 07:55:44.338353 15090 master.cpp:4158] Removing framework 20150206-075544-16842879-38895-15065-0001 (framework2) at scheduler-7bdaa90b-eb9f-4009-bd5a-d07fd3f24cec@127.0.1.1:38895 ../../src/tests/master_allocator_tests.cpp:300: Failure Mock function called more times than expected - taking default action specified at: ../../src/tests/mesos.hpp:713: Function call: deactivateFramework(@0x7fdb74008d70 20150206-075544-16842879-38895-15065-0001) Expected: to be called once Actual: called twice - over...
Story points: 1