Project: mesos
Issue MESOS-2225: FaultToleranceTest.ReregisterFrameworkExitedExecutor is flaky
Observed this on internal CI. {code} [ RUN ] FaultToleranceTest.ReregisterFrameworkExitedExecutor Using temporary directory '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_yNprKi' I0114 18:50:51.461186 4720 leveldb.cpp:176] Opened db in 4.866948ms I0114 18:50:51.462057 4720 leveldb.cpp:183] Compacted db in 472256ns I0114 18:50:51.462514 4720 leveldb.cpp:198] Created db iterator in 42905ns I0114 18:50:51.462784 4720 leveldb.cpp:204] Seeked to beginning of db in 21630ns I0114 18:50:51.463068 4720 leveldb.cpp:273] Iterated through 0 keys in the db in 19967ns I0114 18:50:51.463485 4720 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0114 18:50:51.464555 4737 recover.cpp:449] Starting replica recovery I0114 18:50:51.465188 4737 recover.cpp:475] Replica is in EMPTY status I0114 18:50:51.467324 4741 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0114 18:50:51.470118 4736 recover.cpp:195] Received a recover response from a replica in EMPTY status I0114 18:50:51.475424 4739 recover.cpp:566] Updating replica status to STARTING I0114 18:50:51.476553 4739 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 107545ns I0114 18:50:51.476862 4739 replica.cpp:323] Persisted replica status to STARTING I0114 18:50:51.477309 4739 recover.cpp:475] Replica is in STARTING status I0114 18:50:51.479109 4734 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0114 18:50:51.481274 4738 recover.cpp:195] Received a recover response from a replica in STARTING status I0114 18:50:51.482324 4738 recover.cpp:566] Updating replica status to VOTING I0114 18:50:51.482913 4738 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 66011ns I0114 18:50:51.483186 4738 replica.cpp:323] Persisted replica status to VOTING I0114 18:50:51.483608 4738 recover.cpp:580] Successfully joined the Paxos group I0114 18:50:51.484031 4738 recover.cpp:464] Recover process terminated I0114 18:50:51.554949 4734 master.cpp:262] Master 20150114-185051-2272962752-57018-4720 (fedora-19) started on 192.168.122.135:57018 I0114 18:50:51.555785 4734 master.cpp:308] Master only allowing authenticated frameworks to register I0114 18:50:51.556046 4734 master.cpp:313] Master only allowing authenticated slaves to register I0114 18:50:51.556426 4734 credentials.hpp:36] Loading credentials for authentication from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_yNprKi/credentials' I0114 18:50:51.557003 4734 master.cpp:357] Authorization enabled I0114 18:50:51.558007 4737 hierarchical_allocator_process.hpp:285] Initialized hierarchical allocator process I0114 18:50:51.558521 4741 whitelist_watcher.cpp:65] No whitelist given I0114 18:50:51.562185 4734 master.cpp:1219] The newly elected leader is master@192.168.122.135:57018 with id 20150114-185051-2272962752-57018-4720 I0114 18:50:51.562680 4734 master.cpp:1232] Elected as the leading master! I0114 18:50:51.562950 4734 master.cpp:1050] Recovering from registrar I0114 18:50:51.564506 4736 registrar.cpp:313] Recovering registrar I0114 18:50:51.566162 4737 log.cpp:660] Attempting to start the writer I0114 18:50:51.568691 4741 replica.cpp:477] Replica received implicit promise request with proposal 1 I0114 18:50:51.569154 4741 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 106885ns I0114 18:50:51.569504 4741 replica.cpp:345] Persisted promised to 1 I0114 18:50:51.573277 4740 coordinator.cpp:230] Coordinator attemping to fill missing position I0114 18:50:51.575623 4739 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0114 18:50:51.576133 4739 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 86360ns I0114 18:50:51.576449 4739 replica.cpp:679] Persisted action at 0 I0114 18:50:51.586966 4736 replica.cpp:511] Replica received write request for position 0 I0114 18:50:51.587666 4736 leveldb.cpp:438] Reading position from leveldb took 60621ns I0114 18:50:51.588043 4736 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 81094ns I0114 18:50:51.588374 4736 replica.cpp:679] Persisted action at 0 I0114 18:50:51.589418 4736 replica.cpp:658] Replica received learned notice for position 0 I0114 18:50:51.590428 4736 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 106648ns I0114 18:50:51.590840 4736 replica.cpp:679] Persisted action at 0 I0114 18:50:51.591104 4736 replica.cpp:664] Replica learned NOP action at position 0 I0114 18:50:51.592260 4734 log.cpp:676] Writer started with ending position 0 I0114 18:50:51.594172 4739 leveldb.cpp:438] Reading position from leveldb took 52163ns I0114 18:50:51.600744 4736 registrar.cpp:346] Successfully fetched the registry (0B) in 35968us I0114 18:50:51.601646 4736 registrar.cpp:445] Applied 1 operations in 184502ns; attempting to update the 'registry' I0114 18:50:51.604329 4737 log.cpp:684] Attempting to append 130 bytes to the log I0114 18:50:51.604966 4737 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0114 18:50:51.606449 4737 replica.cpp:511] Replica received write request for position 1 I0114 18:50:51.606937 4737 leveldb.cpp:343] Persisting action (149 bytes) to leveldb took 84877ns I0114 18:50:51.607199 4737 replica.cpp:679] Persisted action at 1 I0114 18:50:51.611934 4741 replica.cpp:658] Replica received learned notice for position 1 I0114 18:50:51.612423 4741 leveldb.cpp:343] Persisting action (151 bytes) to leveldb took 113059ns I0114 18:50:51.612794 4741 replica.cpp:679] Persisted action at 1 I0114 18:50:51.613056 4741 replica.cpp:664] Replica learned APPEND action at position 1 I0114 18:50:51.614598 4741 log.cpp:703] Attempting to truncate the log to 1 I0114 18:50:51.615157 4741 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0114 18:50:51.616458 4737 replica.cpp:511] Replica received write request for position 2 I0114 18:50:51.616902 4737 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 71716ns I0114 18:50:51.617168 4737 replica.cpp:679] Persisted action at 2 I0114 18:50:51.618505 4740 replica.cpp:658] Replica received learned notice for position 2 I0114 18:50:51.619031 4740 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 78481ns I0114 18:50:51.619567 4740 leveldb.cpp:401] Deleting ~1 keys from leveldb took 59638ns I0114 18:50:51.619832 4740 replica.cpp:679] Persisted action at 2 I0114 18:50:51.620101 4740 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0114 18:50:51.621757 4736 registrar.cpp:490] Successfully updated the 'registry' in 19.78496ms I0114 18:50:51.622658 4736 registrar.cpp:376] Successfully recovered registrar I0114 18:50:51.623261 4736 master.cpp:1077] Recovered 0 slaves from the Registry (94B) ; allowing 10mins for slaves to re-register I0114 18:50:51.670349 4739 slave.cpp:173] Slave started on 115)@192.168.122.135:57018 I0114 18:50:51.671133 4739 credentials.hpp:84] Loading credential for authentication from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_ONrVug/credential' I0114 18:50:51.671685 4739 slave.cpp:282] Slave using credential for: test-principal I0114 18:50:51.672245 4739 slave.cpp:300] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.673360 4739 slave.cpp:329] Slave hostname: fedora-19 I0114 18:50:51.673660 4739 slave.cpp:330] Slave checkpoint: false W0114 18:50:51.674052 4739 slave.cpp:332] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0114 18:50:51.677234 4737 state.cpp:33] Recovering state from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_ONrVug/meta' I0114 18:50:51.684973 4739 status_update_manager.cpp:197] Recovering status update manager I0114 18:50:51.687644 4739 slave.cpp:3519] Finished recovery I0114 18:50:51.688698 4737 slave.cpp:613] New master detected at master@192.168.122.135:57018 I0114 18:50:51.688902 4734 status_update_manager.cpp:171] Pausing sending status updates I0114 18:50:51.689482 4737 slave.cpp:676] Authenticating with master master@192.168.122.135:57018 I0114 18:50:51.689910 4737 slave.cpp:681] Using default CRAM-MD5 authenticatee I0114 18:50:51.690577 4741 authenticatee.hpp:138] Creating new client SASL connection I0114 18:50:51.691453 4737 slave.cpp:649] Detecting new master I0114 18:50:51.691864 4741 master.cpp:4130] Authenticating slave(115)@192.168.122.135:57018 I0114 18:50:51.692369 4741 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:50:51.693208 4741 authenticator.hpp:170] Creating new server SASL connection I0114 18:50:51.694598 4738 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:50:51.694893 4738 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:50:51.695329 4741 authenticator.hpp:276] Received SASL authentication start I0114 18:50:51.695641 4741 authenticator.hpp:398] Authentication requires more steps I0114 18:50:51.696028 4736 authenticatee.hpp:275] Received SASL authentication step I0114 18:50:51.696486 4741 authenticator.hpp:304] Received SASL authentication step I0114 18:50:51.696753 4741 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:50:51.697041 4741 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:50:51.697343 4741 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:50:51.697685 4741 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:50:51.697998 4741 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.698251 4741 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.698580 4741 authenticator.hpp:390] Authentication success I0114 18:50:51.698927 4735 authenticatee.hpp:315] Authentication success I0114 18:50:51.705123 4741 master.cpp:4188] Successfully authenticated principal 'test-principal' at slave(115)@192.168.122.135:57018 I0114 18:50:51.705847 4720 sched.cpp:151] Version: 0.22.0 I0114 18:50:51.707159 4736 sched.cpp:248] New master detected at master@192.168.122.135:57018 I0114 18:50:51.707523 4736 sched.cpp:304] Authenticating with master master@192.168.122.135:57018 I0114 18:50:51.707792 4736 sched.cpp:311] Using default CRAM-MD5 authenticatee I0114 18:50:51.708412 4736 authenticatee.hpp:138] Creating new client SASL connection I0114 18:50:51.709316 4735 slave.cpp:747] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:50:51.709723 4737 master.cpp:4130] Authenticating scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.710274 4737 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:50:51.710739 4735 slave.cpp:1075] Will retry registration in 17.028024ms if necessary I0114 18:50:51.711304 4737 master.cpp:3276] Registering slave at slave(115)@192.168.122.135:57018 (fedora-19) with id 20150114-185051-2272962752-57018-4720-S0 I0114 18:50:51.711459 4738 authenticator.hpp:170] Creating new server SASL connection I0114 18:50:51.713142 4739 registrar.cpp:445] Applied 1 operations in 100530ns; attempting to update the 'registry' I0114 18:50:51.713465 4738 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:50:51.715435 4738 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:50:51.715963 4740 authenticator.hpp:276] Received SASL authentication start I0114 18:50:51.716258 4740 authenticator.hpp:398] Authentication requires more steps I0114 18:50:51.716524 4740 authenticatee.hpp:275] Received SASL authentication step I0114 18:50:51.716784 4740 authenticator.hpp:304] Received SASL authentication step I0114 18:50:51.716979 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:50:51.717139 4740 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:50:51.717315 4740 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:50:51.717542 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:50:51.717703 4740 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.717864 4740 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.718040 4740 authenticator.hpp:390] Authentication success I0114 18:50:51.718292 4740 authenticatee.hpp:315] Authentication success I0114 18:50:51.718454 4738 master.cpp:4188] Successfully authenticated principal 'test-principal' at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.719012 4740 sched.cpp:392] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:50:51.719364 4740 sched.cpp:515] Sending registration request to master@192.168.122.135:57018 I0114 18:50:51.719702 4740 sched.cpp:548] Will retry registration in 746.539282ms if necessary I0114 18:50:51.719902 4735 master.cpp:1417] Received registration request for framework 'default' at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.720232 4735 master.cpp:1298] Authorizing framework principal 'test-principal' to receive offers for role '*' I0114 18:50:51.722206 4735 master.cpp:1481] Registering framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.720927 4737 log.cpp:684] Attempting to append 300 bytes to the log I0114 18:50:51.722924 4737 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0114 18:50:51.724269 4737 replica.cpp:511] Replica received write request for position 3 I0114 18:50:51.724817 4737 leveldb.cpp:343] Persisting action (319 bytes) to leveldb took 116638ns I0114 18:50:51.728560 4737 replica.cpp:679] Persisted action at 3 I0114 18:50:51.726066 4736 sched.cpp:442] Framework registered with 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.728879 4736 sched.cpp:456] Scheduler::registered took 34885ns I0114 18:50:51.725520 4735 hierarchical_allocator_process.hpp:319] Added framework 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.731864 4735 hierarchical_allocator_process.hpp:839] No resources available to allocate! I0114 18:50:51.732038 4735 hierarchical_allocator_process.hpp:746] Performed allocation for 0 slaves in 214728ns I0114 18:50:51.733106 4738 replica.cpp:658] Replica received learned notice for position 3 I0114 18:50:51.733340 4738 leveldb.cpp:343] Persisting action (321 bytes) to leveldb took 83165ns I0114 18:50:51.733538 4738 replica.cpp:679] Persisted action at 3 I0114 18:50:51.733705 4738 replica.cpp:664] Replica learned APPEND action at position 3 I0114 18:50:51.735610 4738 registrar.cpp:490] Successfully updated the 'registry' in 21.936128ms I0114 18:50:51.735805 4739 log.cpp:703] Attempting to truncate the log to 3 I0114 18:50:51.736445 4739 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0114 18:50:51.737664 4739 replica.cpp:511] Replica received write request for position 4 I0114 18:50:51.738013 4739 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 72906ns I0114 18:50:51.738255 4739 replica.cpp:679] Persisted action at 4 I0114 18:50:51.743397 4734 replica.cpp:658] Replica received learned notice for position 4 I0114 18:50:51.743628 4734 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 78832ns I0114 18:50:51.743837 4734 leveldb.cpp:401] Deleting ~2 keys from leveldb took 63991ns I0114 18:50:51.744004 4734 replica.cpp:679] Persisted action at 4 I0114 18:50:51.744168 4734 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0114 18:50:51.745537 4738 master.cpp:3330] Registered slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.745968 4734 hierarchical_allocator_process.hpp:453] Added slave 20150114-185051-2272962752-57018-4720-S0 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0114 18:50:51.746070 4735 slave.cpp:781] Registered with master master@192.168.122.135:57018; given slave ID 20150114-185051-2272962752-57018-4720-S0 I0114 18:50:51.751437 4741 status_update_manager.cpp:178] Resuming sending status updates I0114 18:50:51.752428 4740 master.cpp:4072] Sending 1 offers to framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.753764 4740 sched.cpp:605] Scheduler::resourceOffers took 751714ns I0114 18:50:51.754812 4740 master.cpp:2541] Processing reply for offers: [ 20150114-185051-2272962752-57018-4720-O0 ] on slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) for framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.755040 4740 master.cpp:2647] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' W0114 18:50:51.756431 4741 master.cpp:2124] Executor default for task 0 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0114 18:50:51.756652 4741 master.cpp:2136] Executor default for task 0 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0114 18:50:51.757284 4741 master.hpp:766] Adding task 0 with resources cpus(*):1; mem(*):16 on slave 20150114-185051-2272962752-57018-4720-S0 (fedora-19) I0114 18:50:51.757733 4734 hierarchical_allocator_process.hpp:764] Performed allocation for slave 20150114-185051-2272962752-57018-4720-S0 in 9.535066ms I0114 18:50:51.758117 4735 slave.cpp:2588] Received ping from slave-observer(95)@192.168.122.135:57018 I0114 18:50:51.758630 4741 master.cpp:2897] Launching task 0 of framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 with resources cpus(*):1; mem(*):16 on slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) I0114 18:50:51.759526 4741 hierarchical_allocator_process.hpp:610] Updated allocation of framework 20150114-185051-2272962752-57018-4720-0000 on slave 20150114-185051-2272962752-57018-4720-S0 from cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] to cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.759796 4737 slave.cpp:1130] Got assigned task 0 for framework 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.761184 4737 slave.cpp:1245] Launching task 0 for framework 20150...
Story points: 2