Project: mesos
Issue MESOS-1545: SlaveRecoveryTest/0.MultipleFrameworks is flaky
{code} [ RUN ] SlaveRecoveryTest/0.MultipleFrameworks Using temporary directory '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_6dJqxr' I0626 00:04:39.557339 5450 leveldb.cpp:176] Opened db in 179.857593ms I0626 00:04:39.565433 5450 leveldb.cpp:183] Compacted db in 8.071041ms I0626 00:04:39.565457 5450 leveldb.cpp:198] Created db iterator in 4065ns I0626 00:04:39.565466 5450 leveldb.cpp:204] Seeked to beginning of db in 596ns I0626 00:04:39.565474 5450 leveldb.cpp:273] Iterated through 0 keys in the db in 396ns I0626 00:04:39.565490 5450 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0626 00:04:39.565827 5476 recover.cpp:425] Starting replica recovery I0626 00:04:39.566033 5474 recover.cpp:451] Replica is in EMPTY status I0626 00:04:39.566504 5474 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0626 00:04:39.566686 5477 recover.cpp:188] Received a recover response from a replica in EMPTY status I0626 00:04:39.566905 5472 recover.cpp:542] Updating replica status to STARTING I0626 00:04:39.568307 5471 master.cpp:288] Master 20140626-000439-1032504131-55423-5450 (juno.apache.org) started on 67.195.138.61:55423 I0626 00:04:39.568332 5471 master.cpp:325] Master only allowing authenticated frameworks to register I0626 00:04:39.568339 5471 master.cpp:330] Master only allowing authenticated slaves to register I0626 00:04:39.568348 5471 credentials.hpp:35] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_6dJqxr/credentials' I0626 00:04:39.568461 5471 master.cpp:356] Authorization enabled I0626 00:04:39.568739 5478 master.cpp:122] No whitelist given. Advertising offers for all slaves I0626 00:04:39.568814 5475 hierarchical_allocator_process.hpp:301] Initializing hierarchical allocator process with master : master@67.195.138.61:55423 I0626 00:04:39.569206 5478 master.cpp:1122] The newly elected leader is master@67.195.138.61:55423 with id 20140626-000439-1032504131-55423-5450 I0626 00:04:39.569223 5478 master.cpp:1135] Elected as the leading master! I0626 00:04:39.569231 5478 master.cpp:953] Recovering from registrar I0626 00:04:39.569286 5475 registrar.cpp:313] Recovering registrar I0626 00:04:39.600639 5477 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 33.682136ms I0626 00:04:39.600661 5477 replica.cpp:320] Persisted replica status to STARTING I0626 00:04:39.600790 5476 recover.cpp:451] Replica is in STARTING status I0626 00:04:39.601184 5474 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0626 00:04:39.601274 5477 recover.cpp:188] Received a recover response from a replica in STARTING status I0626 00:04:39.601465 5471 recover.cpp:542] Updating replica status to VOTING I0626 00:04:39.610605 5471 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 9.076262ms I0626 00:04:39.610638 5471 replica.cpp:320] Persisted replica status to VOTING I0626 00:04:39.610683 5471 recover.cpp:556] Successfully joined the Paxos group I0626 00:04:39.610780 5471 recover.cpp:440] Recover process terminated I0626 00:04:39.610946 5474 log.cpp:656] Attempting to start the writer I0626 00:04:39.611486 5475 replica.cpp:474] Replica received implicit promise request with proposal 1 I0626 00:04:39.618924 5475 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 7.418789ms I0626 00:04:39.618942 5475 replica.cpp:342] Persisted promised to 1 I0626 00:04:39.619220 5476 coordinator.cpp:230] Coordinator attemping to fill missing position I0626 00:04:39.619763 5476 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0626 00:04:39.627267 5476 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 7.485492ms I0626 00:04:39.627295 5476 replica.cpp:676] Persisted action at 0 I0626 00:04:39.627822 5473 replica.cpp:508] Replica received write request for position 0 I0626 00:04:39.627861 5473 leveldb.cpp:438] Reading position from leveldb took 17132ns I0626 00:04:39.635592 5473 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 7.714322ms I0626 00:04:39.635612 5473 replica.cpp:676] Persisted action at 0 I0626 00:04:39.635797 5473 replica.cpp:655] Replica received learned notice for position 0 I0626 00:04:39.643941 5473 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 8.129347ms I0626 00:04:39.643960 5473 replica.cpp:676] Persisted action at 0 I0626 00:04:39.643970 5473 replica.cpp:661] Replica learned NOP action at position 0 I0626 00:04:39.644207 5473 log.cpp:672] Writer started with ending position 0 I0626 00:04:39.644625 5471 leveldb.cpp:438] Reading position from leveldb took 9128ns I0626 00:04:39.646010 5476 registrar.cpp:346] Successfully fetched the registry (0B) I0626 00:04:39.646044 5476 registrar.cpp:422] Attempting to update the 'registry' I0626 00:04:39.647274 5471 log.cpp:680] Attempting to append 136 bytes to the log I0626 00:04:39.647337 5471 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0626 00:04:39.647687 5476 replica.cpp:508] Replica received write request for position 1 I0626 00:04:39.655206 5476 leveldb.cpp:343] Persisting action (155 bytes) to leveldb took 7.499736ms I0626 00:04:39.655225 5476 replica.cpp:676] Persisted action at 1 I0626 00:04:39.655467 5476 replica.cpp:655] Replica received learned notice for position 1 I0626 00:04:39.663534 5476 leveldb.cpp:343] Persisting action (157 bytes) to leveldb took 8.054929ms I0626 00:04:39.663554 5476 replica.cpp:676] Persisted action at 1 I0626 00:04:39.663563 5476 replica.cpp:661] Replica learned APPEND action at position 1 I0626 00:04:39.663890 5478 registrar.cpp:479] Successfully updated 'registry' I0626 00:04:39.663947 5478 registrar.cpp:372] Successfully recovered registrar I0626 00:04:39.663969 5476 log.cpp:699] Attempting to truncate the log to 1 I0626 00:04:39.664044 5478 master.cpp:980] Recovered 0 slaves from the Registry (98B) ; allowing 10mins for slaves to re-register I0626 00:04:39.664057 5476 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0626 00:04:39.664341 5476 replica.cpp:508] Replica received write request for position 2 I0626 00:04:39.664681 5450 containerizer.cpp:124] Using isolation: posix/cpu,posix/mem I0626 00:04:39.666721 5471 slave.cpp:168] Slave started on 173)@67.195.138.61:55423 I0626 00:04:39.666741 5471 credentials.hpp:35] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/credential' I0626 00:04:39.666806 5471 slave.cpp:268] Slave using credential for: test-principal I0626 00:04:39.666936 5471 slave.cpp:281] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0626 00:04:39.667000 5471 slave.cpp:326] Slave hostname: juno.apache.org I0626 00:04:39.667009 5471 slave.cpp:327] Slave checkpoint: true I0626 00:04:39.667572 5478 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta' I0626 00:04:39.667703 5475 status_update_manager.cpp:193] Recovering status update manager I0626 00:04:39.667840 5475 containerizer.cpp:287] Recovering containerizer I0626 00:04:39.668478 5471 slave.cpp:3128] Finished recovery I0626 00:04:39.668712 5471 slave.cpp:601] New master detected at master@67.195.138.61:55423 I0626 00:04:39.668738 5471 slave.cpp:677] Authenticating with master master@67.195.138.61:55423 I0626 00:04:39.668802 5471 slave.cpp:650] Detecting new master I0626 00:04:39.668861 5471 status_update_manager.cpp:167] New master detected at master@67.195.138.61:55423 I0626 00:04:39.668916 5471 authenticatee.hpp:128] Creating new client SASL connection I0626 00:04:39.669087 5471 master.cpp:3499] Authenticating slave(173)@67.195.138.61:55423 I0626 00:04:39.669203 5471 authenticator.hpp:156] Creating new server SASL connection I0626 00:04:39.669340 5471 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0626 00:04:39.669359 5471 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0626 00:04:39.669386 5471 authenticator.hpp:262] Received SASL authentication start I0626 00:04:39.669414 5471 authenticator.hpp:384] Authentication requires more steps I0626 00:04:39.669457 5471 authenticatee.hpp:265] Received SASL authentication step I0626 00:04:39.669514 5471 authenticator.hpp:290] Received SASL authentication step I0626 00:04:39.669534 5471 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0626 00:04:39.669543 5471 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0626 00:04:39.669567 5471 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0626 00:04:39.669580 5471 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0626 00:04:39.669589 5471 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.669594 5471 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.669606 5471 authenticator.hpp:376] Authentication success I0626 00:04:39.669641 5471 authenticatee.hpp:305] Authentication success I0626 00:04:39.669669 5471 master.cpp:3539] Successfully authenticated principal 'test-principal' at slave(173)@67.195.138.61:55423 I0626 00:04:39.669761 5450 sched.cpp:139] Version: 0.20.0 I0626 00:04:39.669764 5478 slave.cpp:734] Successfully authenticated with master master@67.195.138.61:55423 I0626 00:04:39.669826 5478 slave.cpp:972] Will retry registration in 3.190666ms if necessary I0626 00:04:39.669950 5471 master.cpp:2781] Registering slave at slave(173)@67.195.138.61:55423 (juno.apache.org) with id 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.669960 5475 sched.cpp:235] New master detected at master@67.195.138.61:55423 I0626 00:04:39.669977 5475 sched.cpp:285] Authenticating with master master@67.195.138.61:55423 I0626 00:04:39.670073 5471 registrar.cpp:422] Attempting to update the 'registry' I0626 00:04:39.670114 5475 authenticatee.hpp:128] Creating new client SASL connection I0626 00:04:39.670263 5475 master.cpp:3499] Authenticating scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670361 5474 authenticator.hpp:156] Creating new server SASL connection I0626 00:04:39.670506 5475 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0626 00:04:39.670526 5475 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0626 00:04:39.670559 5475 authenticator.hpp:262] Received SASL authentication start I0626 00:04:39.670590 5475 authenticator.hpp:384] Authentication requires more steps I0626 00:04:39.670619 5475 authenticatee.hpp:265] Received SASL authentication step I0626 00:04:39.670650 5475 authenticator.hpp:290] Received SASL authentication step I0626 00:04:39.670670 5475 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0626 00:04:39.670677 5475 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0626 00:04:39.670687 5475 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0626 00:04:39.670697 5475 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0626 00:04:39.670706 5475 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.670712 5475 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.670723 5475 authenticator.hpp:376] Authentication success I0626 00:04:39.670749 5475 authenticatee.hpp:305] Authentication success I0626 00:04:39.670773 5475 master.cpp:3539] Successfully authenticated principal 'test-principal' at scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670845 5475 sched.cpp:359] Successfully authenticated with master master@67.195.138.61:55423 I0626 00:04:39.670858 5475 sched.cpp:478] Sending registration request to master@67.195.138.61:55423 I0626 00:04:39.670899 5475 master.cpp:1241] Received registration request from scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670922 5475 master.cpp:1201] Authorizing framework principal 'test-principal' to receive offers for role '*' I0626 00:04:39.671052 5475 master.cpp:1300] Registering framework 20140626-000439-1032504131-55423-5450-0000 at scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.671159 5474 sched.cpp:409] Framework registered with 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.671185 5474 sched.cpp:423] Scheduler::registered took 10223ns I0626 00:04:39.671226 5474 hierarchical_allocator_process.hpp:331] Added framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.671241 5474 hierarchical_allocator_process.hpp:724] No resources available to allocate! I0626 00:04:39.671247 5474 hierarchical_allocator_process.hpp:686] Performed allocation for 0 slaves in 8574ns I0626 00:04:39.671879 5476 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 7.48781ms I0626 00:04:39.671900 5476 replica.cpp:676] Persisted action at 2 I0626 00:04:39.672164 5471 replica.cpp:655] Replica received learned notice for position 2 I0626 00:04:39.674092 5472 slave.cpp:972] Will retry registration in 25.467893ms if necessary I0626 00:04:39.674108 5476 master.cpp:2769] Ignoring register slave message from slave(173)@67.195.138.61:55423 (juno.apache.org) as admission is already in progress I0626 00:04:39.680193 5471 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 8.01285ms I0626 00:04:39.680223 5471 leveldb.cpp:401] Deleting ~1 keys from leveldb took 11393ns I0626 00:04:39.680234 5471 replica.cpp:676] Persisted action at 2 I0626 00:04:39.680245 5471 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0626 00:04:39.680585 5472 log.cpp:680] Attempting to append 326 bytes to the log I0626 00:04:39.680670 5477 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0626 00:04:39.680953 5474 replica.cpp:508] Replica received write request for position 3 I0626 00:04:39.688521 5474 leveldb.cpp:343] Persisting action (345 bytes) to leveldb took 7.548316ms I0626 00:04:39.688542 5474 replica.cpp:676] Persisted action at 3 I0626 00:04:39.688750 5474 replica.cpp:655] Replica received learned notice for position 3 I0626 00:04:39.696851 5474 leveldb.cpp:343] Persisting action (347 bytes) to leveldb took 8.088289ms I0626 00:04:39.696869 5474 replica.cpp:676] Persisted action at 3 I0626 00:04:39.696878 5474 replica.cpp:661] Replica learned APPEND action at position 3 I0626 00:04:39.697268 5474 registrar.cpp:479] Successfully updated 'registry' I0626 00:04:39.697350 5474 log.cpp:699] Attempting to truncate the log to 3 I0626 00:04:39.697412 5474 master.cpp:2821] Registered slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) I0626 00:04:39.697423 5474 master.cpp:3967] Adding slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0626 00:04:39.697535 5474 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0626 00:04:39.697618 5474 slave.cpp:768] Registered with master master@67.195.138.61:55423; given slave ID 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.697754 5474 slave.cpp:781] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta/slaves/20140626-000439-1032504131-55423-5450-0/slave.info' I0626 00:04:39.697762 5471 hierarchical_allocator_process.hpp:444] Added slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0626 00:04:39.697845 5471 hierarchical_allocator_process.hpp:750] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 to framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.697854 5474 slave.cpp:2325] Received ping from slave-observer(142)@67.195.138.61:55423 I0626 00:04:39.698040 5471 hierarchical_allocator_process.hpp:706] Performed allocation for slave 20140626-000439-1032504131-55423-5450-0 in 231333ns I0626 00:04:39.698051 5474 replica.cpp:508] Replica received write request for position 4 I0626 00:04:39.698118 5471 master.hpp:794] Adding offer 20140626-000439-1032504131-55423-5450-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.698170 5471 master.cpp:3446] Sending 1 offers to framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.698318 5471 sched.cpp:546] Scheduler::resourceOffers took 24371ns I0626 00:04:39.699718 5477 master.hpp:804] Removing offer 20140626-000439-1032504131-55423-5450-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.699787 5477 master.cpp:2125] Processing reply for offers: [ 20140626-000439-1032504131-55423-5450-0 ] on slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) for framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.699812 5477 master.cpp:2211] Authorizing framework principal 'test-principal' to launch task 897522cc-4ec5-4904-aed0-00b6b8c41028 as user 'jenkins' I0626 00:04:39.700160 5477 master.hpp:766] Adding task 897522cc-4ec5-4904-aed0-00b6b8c41028 with resources cpus(*):1; mem(*):512 on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.700188 5477 master.cpp:2277] Launching task 897522cc-4ec5-4904-aed0-00b6b8c41028 of framework 20140626-000439-1032504131-55423-5450-0000 with resources cpus(*):1; mem(*):512 on slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) I0626 00:04:39.700392 5471 slave.cpp:1003] Got assigned task 897522cc-4ec5-4904-aed0-00b6b8c41028 for framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.700479 5477 hierarchical_allocator_process.hpp:546] Framework 20140626-000439-1032504131-55423-5450-0000 left cpus(*):1; mem(*):512; disk(*):1024; ports(*):[31000-32000] unused on slave 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.700505 5471 slave.cpp:3400] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta/slaves/20140626-000439-1032504131-55423-5450-0/frameworks/20140626-000439-1032504131-55423-5450-0000/framework.info' I0626 00:04:39.700597 5477 hierarchical_allocator_process.hpp:588] Framework 20140626-000439-1032504131-55423-5450-0000 filtered slave 20140626-000439-1032504131-55423-5450-0 for 5secs I0626 00:04:39.700686 5471 slave.cpp:3407] Checkpointing framework pid 'scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423' to '...
Story points: 1