Project: mesos
Issue MESOS-2403: MasterAllocatorTest/0.FrameworkReregistersFirst is flaky
{code} [ RUN ] MasterAllocatorTest/0.FrameworkReregistersFirst Using temporary directory '/tmp/MasterAllocatorTest_0_FrameworkReregistersFirst_Vy5Nml' I0224 23:22:31.681670 30589 leveldb.cpp:176] Opened db in 2.943518ms I0224 23:22:31.682152 30619 process.cpp:2117] Dropped / Lost event for PID: slave(65)@67.195.81.187:38391 I0224 23:22:31.682732 30589 leveldb.cpp:183] Compacted db in 1.029469ms I0224 23:22:31.682777 30589 leveldb.cpp:198] Created db iterator in 15460ns I0224 23:22:31.682792 30589 leveldb.cpp:204] Seeked to beginning of db in 1832ns I0224 23:22:31.682802 30589 leveldb.cpp:273] Iterated through 0 keys in the db in 319ns I0224 23:22:31.682833 30589 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0224 23:22:31.683228 30605 recover.cpp:449] Starting replica recovery I0224 23:22:31.683537 30605 recover.cpp:475] Replica is in 4 status I0224 23:22:31.684624 30615 replica.cpp:641] Replica in 4 status received a broadcasted recover request I0224 23:22:31.684978 30616 recover.cpp:195] Received a recover response from a replica in 4 status I0224 23:22:31.685405 30610 recover.cpp:566] Updating replica status to 3 I0224 23:22:31.686249 30609 master.cpp:349] Master 20150224-232231-3142697795-38391-30589 (pomona.apache.org) started on 67.195.81.187:38391 I0224 23:22:31.686265 30617 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 717897ns I0224 23:22:31.686319 30617 replica.cpp:323] Persisted replica status to 3 I0224 23:22:31.686336 30609 master.cpp:395] Master only allowing authenticated frameworks to register I0224 23:22:31.686357 30609 master.cpp:400] Master only allowing authenticated slaves to register I0224 23:22:31.686390 30609 credentials.hpp:37] Loading credentials for authentication from '/tmp/MasterAllocatorTest_0_FrameworkReregistersFirst_Vy5Nml/credentials' I0224 23:22:31.686511 30606 recover.cpp:475] Replica is in 3 status I0224 23:22:31.686563 30609 master.cpp:442] Authorization enabled I0224 23:22:31.686929 30607 whitelist_watcher.cpp:79] No whitelist given I0224 23:22:31.686954 30603 hierarchical.hpp:287] Initialized hierarchical allocator process I0224 23:22:31.687134 30605 replica.cpp:641] Replica in 3 status received a broadcasted recover request I0224 23:22:31.687731 30609 master.cpp:1356] The newly elected leader is master@67.195.81.187:38391 with id 20150224-232231-3142697795-38391-30589 I0224 23:22:31.839818 30609 master.cpp:1369] Elected as the leading master! I0224 23:22:31.839834 30609 master.cpp:1187] Recovering from registrar I0224 23:22:31.839926 30605 registrar.cpp:313] Recovering registrar I0224 23:22:31.840000 30613 recover.cpp:195] Received a recover response from a replica in 3 status I0224 23:22:31.840504 30606 recover.cpp:566] Updating replica status to 1 I0224 23:22:31.841599 30611 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 990330ns I0224 23:22:31.841627 30611 replica.cpp:323] Persisted replica status to 1 I0224 23:22:31.841743 30611 recover.cpp:580] Successfully joined the Paxos group I0224 23:22:31.841904 30611 recover.cpp:464] Recover process terminated I0224 23:22:31.842366 30608 log.cpp:660] Attempting to start the writer I0224 23:22:31.843557 30607 replica.cpp:477] Replica received implicit promise request with proposal 1 I0224 23:22:31.844312 30607 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 722368ns I0224 23:22:31.844337 30607 replica.cpp:345] Persisted promised to 1 I0224 23:22:31.844889 30615 coordinator.cpp:230] Coordinator attemping to fill missing position I0224 23:22:31.846043 30614 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0224 23:22:31.846729 30614 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 660024ns I0224 23:22:31.846746 30614 replica.cpp:679] Persisted action at 0 I0224 23:22:31.847671 30611 replica.cpp:511] Replica received write request for position 0 I0224 23:22:31.847723 30611 leveldb.cpp:438] Reading position from leveldb took 27349ns I0224 23:22:31.848429 30611 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 671461ns I0224 23:22:31.848454 30611 replica.cpp:679] Persisted action at 0 I0224 23:22:31.849041 30615 replica.cpp:658] Replica received learned notice for position 0 I0224 23:22:31.849762 30615 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 690386ns I0224 23:22:31.849787 30615 replica.cpp:679] Persisted action at 0 I0224 23:22:31.849808 30615 replica.cpp:664] Replica learned 1 action at position 0 I0224 23:22:31.850416 30612 log.cpp:676] Writer started with ending position 0 I0224 23:22:31.851490 30615 leveldb.cpp:438] Reading position from leveldb took 30659ns I0224 23:22:31.854452 30610 registrar.cpp:346] Successfully fetched the registry (0B) in 14.491136ms I0224 23:22:31.854543 30610 registrar.cpp:445] Applied 1 operations in 18024ns; attempting to update the 'registry' I0224 23:22:31.857095 30604 log.cpp:684] Attempting to append 139 bytes to the log I0224 23:22:31.857208 30608 coordinator.cpp:340] Coordinator attempting to write 2 action at position 1 I0224 23:22:31.858073 30609 replica.cpp:511] Replica received write request for position 1 I0224 23:22:31.858808 30609 leveldb.cpp:343] Persisting action (158 bytes) to leveldb took 701708ns I0224 23:22:31.858835 30609 replica.cpp:679] Persisted action at 1 I0224 23:22:31.859508 30618 replica.cpp:658] Replica received learned notice for position 1 I0224 23:22:31.860267 30618 leveldb.cpp:343] Persisting action (160 bytes) to leveldb took 731035ns I0224 23:22:31.860309 30618 replica.cpp:679] Persisted action at 1 I0224 23:22:31.860332 30618 replica.cpp:664] Replica learned 2 action at position 1 I0224 23:22:31.860983 30609 registrar.cpp:490] Successfully updated the 'registry' in 6.39616ms I0224 23:22:31.861071 30609 registrar.cpp:376] Successfully recovered registrar I0224 23:22:31.861126 30608 log.cpp:703] Attempting to truncate the log to 1 I0224 23:22:31.861249 30603 coordinator.cpp:340] Coordinator attempting to write 3 action at position 2 I0224 23:22:31.861248 30617 master.cpp:1214] Recovered 0 slaves from the Registry (101B) ; allowing 10mins for slaves to re-register I0224 23:22:31.861831 30613 replica.cpp:511] Replica received write request for position 2 I0224 23:22:31.862504 30613 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 648125ns I0224 23:22:31.862531 30613 replica.cpp:679] Persisted action at 2 I0224 23:22:31.863067 30603 replica.cpp:658] Replica received learned notice for position 2 I0224 23:22:31.863689 30603 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 602784ns I0224 23:22:31.863737 30603 leveldb.cpp:401] Deleting ~1 keys from leveldb took 28697ns I0224 23:22:31.863751 30603 replica.cpp:679] Persisted action at 2 I0224 23:22:31.863767 30603 replica.cpp:664] Replica learned 3 action at position 2 I0224 23:22:31.875962 30610 slave.cpp:174] Slave started on 66)@67.195.81.187:38391 I0224 23:22:31.876008 30610 credentials.hpp:85] Loading credential for authentication from '/tmp/MasterAllocatorTest_0_FrameworkReregistersFirst_ikVXQM/credential' I0224 23:22:31.876144 30610 slave.cpp:281] Slave using credential for: test-principal I0224 23:22:31.876404 30610 slave.cpp:299] Slave resources: cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] I0224 23:22:31.876489 30610 slave.cpp:328] Slave hostname: pomona.apache.org I0224 23:22:31.876502 30610 slave.cpp:329] Slave checkpoint: false W0224 23:22:31.876507 30610 slave.cpp:331] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0224 23:22:31.877014 30603 state.cpp:35] Recovering state from '/tmp/MasterAllocatorTest_0_FrameworkReregistersFirst_ikVXQM/meta' I0224 23:22:31.877230 30610 status_update_manager.cpp:197] Recovering status update manager I0224 23:22:31.877495 30609 slave.cpp:3776] Finished recovery I0224 23:22:31.877879 30607 status_update_manager.cpp:171] Pausing sending status updates I0224 23:22:31.877879 30604 slave.cpp:624] New master detected at master@67.195.81.187:38391 I0224 23:22:31.877959 30604 slave.cpp:687] Authenticating with master master@67.195.81.187:38391 I0224 23:22:31.877975 30604 slave.cpp:692] Using default CRAM-MD5 authenticatee I0224 23:22:31.878069 30604 slave.cpp:660] Detecting new master I0224 23:22:31.878093 30608 authenticatee.hpp:139] Creating new client SASL connection I0224 23:22:31.878223 30604 master.cpp:3813] Authenticating slave(66)@67.195.81.187:38391 I0224 23:22:31.878244 30604 master.cpp:3824] Using default CRAM-MD5 authenticator I0224 23:22:31.878412 30613 authenticator.hpp:170] Creating new server SASL connection I0224 23:22:31.878525 30603 authenticatee.hpp:230] Received SASL authentication mechanisms: CRAM-MD5 I0224 23:22:31.878551 30603 authenticatee.hpp:256] Attempting to authenticate with mechanism 'CRAM-MD5' I0224 23:22:31.878625 30617 authenticator.hpp:276] Received SASL authentication start I0224 23:22:31.878662 30617 authenticator.hpp:398] Authentication requires more steps I0224 23:22:31.878727 30603 authenticatee.hpp:276] Received SASL authentication step I0224 23:22:31.878815 30617 authenticator.hpp:304] Received SASL authentication step I0224 23:22:31.878839 30617 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0224 23:22:31.878847 30617 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0224 23:22:31.878875 30617 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0224 23:22:31.878891 30617 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0224 23:22:31.878900 30617 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0224 23:22:31.878906 30617 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0224 23:22:31.878916 30617 authenticator.hpp:390] Authentication success I0224 23:22:31.880717 30589 sched.cpp:157] Version: 0.23.0 I0224 23:22:32.017823 30611 authenticatee.hpp:316] Authentication success I0224 23:22:32.017901 30618 master.cpp:3871] Successfully authenticated principal 'test-principal' at slave(66)@67.195.81.187:38391 I0224 23:22:32.018156 30615 sched.cpp:254] New master detected at master@67.195.81.187:38391 I0224 23:22:32.018240 30615 sched.cpp:310] Authenticating with master master@67.195.81.187:38391 I0224 23:22:32.018263 30615 sched.cpp:317] Using default CRAM-MD5 authenticatee I0224 23:22:32.018496 30613 slave.cpp:758] Successfully authenticated with master master@67.195.81.187:38391 I0224 23:22:32.018579 30611 authenticatee.hpp:139] Creating new client SASL connection I0224 23:22:32.018620 30613 slave.cpp:1090] Will retry registration in 363167ns if necessary I0224 23:22:32.018811 30615 master.cpp:2938] Registering slave at slave(66)@67.195.81.187:38391 (pomona.apache.org) with id 20150224-232231-3142697795-38391-30589-S0 I0224 23:22:32.019122 30615 master.cpp:3813] Authenticating scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.019156 30615 master.cpp:3824] Using default CRAM-MD5 authenticator I0224 23:22:32.019232 30612 registrar.cpp:445] Applied 1 operations in 57599ns; attempting to update the 'registry' I0224 23:22:32.019394 30603 authenticator.hpp:170] Creating new server SASL connection I0224 23:22:32.019541 30611 authenticatee.hpp:230] Received SASL authentication mechanisms: CRAM-MD5 I0224 23:22:32.019568 30611 authenticatee.hpp:256] Attempting to authenticate with mechanism 'CRAM-MD5' I0224 23:22:32.019666 30605 authenticator.hpp:276] Received SASL authentication start I0224 23:22:32.019717 30605 authenticator.hpp:398] Authentication requires more steps I0224 23:22:32.019805 30615 authenticatee.hpp:276] Received SASL authentication step I0224 23:22:32.019942 30605 authenticator.hpp:304] Received SASL authentication step I0224 23:22:32.019979 30605 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0224 23:22:32.019994 30605 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0224 23:22:32.020025 30605 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0224 23:22:32.020036 30610 slave.cpp:1090] Will retry registration in 10.850555ms if necessary I0224 23:22:32.020053 30605 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0224 23:22:32.020102 30605 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0224 23:22:32.020117 30605 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0224 23:22:32.020133 30605 authenticator.hpp:390] Authentication success I0224 23:22:32.020151 30611 master.cpp:2926] Ignoring register slave message from slave(66)@67.195.81.187:38391 (pomona.apache.org) as admission is already in progress I0224 23:22:32.020226 30603 authenticatee.hpp:316] Authentication success I0224 23:22:32.020256 30611 master.cpp:3871] Successfully authenticated principal 'test-principal' at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.020534 30615 sched.cpp:398] Successfully authenticated with master master@67.195.81.187:38391 I0224 23:22:32.020561 30615 sched.cpp:521] Sending registration request to master@67.195.81.187:38391 I0224 23:22:32.020635 30615 sched.cpp:554] Will retry registration in 490.035142ms if necessary I0224 23:22:32.020720 30613 master.cpp:1574] Received registration request for framework 'default' at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.020787 30613 master.cpp:1435] Authorizing framework principal 'test-principal' to receive offers for role '*' I0224 23:22:32.021122 30607 master.cpp:1638] Registering framework 20150224-232231-3142697795-38391-30589-0000 (default) at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.021502 30611 hierarchical.hpp:321] Added framework 20150224-232231-3142697795-38391-30589-0000 I0224 23:22:32.021531 30611 hierarchical.hpp:834] No resources available to allocate! I0224 23:22:32.021543 30611 hierarchical.hpp:741] Performed allocation for 0 slaves in 18915ns I0224 23:22:32.021618 30609 sched.cpp:448] Framework registered with 20150224-232231-3142697795-38391-30589-0000 I0224 23:22:32.021673 30609 sched.cpp:462] Scheduler::registered took 26310ns I0224 23:22:32.022400 30613 log.cpp:684] Attempting to append 316 bytes to the log I0224 23:22:32.022523 30608 coordinator.cpp:340] Coordinator attempting to write 2 action at position 3 I0224 23:22:32.023232 30607 replica.cpp:511] Replica received write request for position 3 I0224 23:22:32.024055 30607 leveldb.cpp:343] Persisting action (335 bytes) to leveldb took 798548ns I0224 23:22:32.024073 30607 replica.cpp:679] Persisted action at 3 I0224 23:22:32.024651 30610 replica.cpp:658] Replica received learned notice for position 3 I0224 23:22:32.025252 30610 leveldb.cpp:343] Persisting action (337 bytes) to leveldb took 580525ns I0224 23:22:32.025271 30610 replica.cpp:679] Persisted action at 3 I0224 23:22:32.025297 30610 replica.cpp:664] Replica learned 2 action at position 3 I0224 23:22:32.025995 30618 registrar.cpp:490] Successfully updated the 'registry' in 6.586112ms I0224 23:22:32.026228 30604 log.cpp:703] Attempting to truncate the log to 3 I0224 23:22:32.026360 30609 coordinator.cpp:340] Coordinator attempting to write 3 action at position 4 I0224 23:22:32.026669 30609 slave.cpp:2831] Received ping from slave-observer(66)@67.195.81.187:38391 I0224 23:22:32.026772 30609 slave.cpp:792] Registered with master master@67.195.81.187:38391; given slave ID 20150224-232231-3142697795-38391-30589-S0 I0224 23:22:32.026737 30603 master.cpp:2995] Registered slave 20150224-232231-3142697795-38391-30589-S0 at slave(66)@67.195.81.187:38391 (pomona.apache.org) with cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] I0224 23:22:32.026867 30603 status_update_manager.cpp:178] Resuming sending status updates I0224 23:22:32.026868 30617 hierarchical.hpp:455] Added slave 20150224-232231-3142697795-38391-30589-S0 (pomona.apache.org) with cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] available) I0224 23:22:32.026921 30615 replica.cpp:511] Replica received write request for position 4 I0224 23:22:32.027276 30617 hierarchical.hpp:759] Performed allocation for slave 20150224-232231-3142697795-38391-30589-S0 in 351257ns I0224 23:22:32.027580 30615 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 624249ns I0224 23:22:32.027604 30615 replica.cpp:679] Persisted action at 4 I0224 23:22:32.027642 30618 master.cpp:3755] Sending 1 offers to framework 20150224-232231-3142697795-38391-30589-0000 (default) at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.028223 30617 replica.cpp:658] Replica received learned notice for position 4 I0224 23:22:32.028621 30607 sched.cpp:611] Scheduler::resourceOffers took 648326ns I0224 23:22:32.028916 30617 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 662416ns I0224 23:22:32.028991 30617 leveldb.cpp:401] Deleting ~2 keys from leveldb took 47386ns I0224 23:22:32.029021 30617 replica.cpp:679] Persisted action at 4 I0224 23:22:32.029044 30617 replica.cpp:664] Replica learned 3 action at position 4 I0224 23:22:32.029534 30613 master.cpp:2268] Processing ACCEPT call for offers: [ 20150224-232231-3142697795-38391-30589-O0 ] on slave 20150224-232231-3142697795-38391-30589-S0 at slave(66)@67.195.81.187:38391 (pomona.apache.org) for framework 20150224-232231-3142697795-38391-30589-0000 (default) at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 I0224 23:22:32.190521 30613 master.cpp:2112] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' W0224 23:22:32.191864 30604 validation.cpp:328] Executor default for task 0 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0224 23:22:32.191905 30604 validation.cpp:340] Executor default for task 0 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0224 23:22:32.192206 30604 master.hpp:822] Adding task 0 with resources cpus(*):1; mem(*):500 on slave 20150224-232231-3142697795-38391-30589-S0 (pomona.apache.org) I0224 23:22:32.192318 30604 master.cpp:2545] Launching task 0 of framework 20150224-232231-3142697795-38391-30589-0000 (default) at scheduler-9a3224cc-aef0-49a7-a240-4b85b913ff44@67.195.81.187:38391 with resources cpus(*):1; mem(*):500 on slave 20150224-232231-3142697795-38391-30589-S0 at slave(66)@67.195.81.187:38391 (pomona.apache.org) I0224 23:22:32.192659 30611 slave.cpp:1121] Got assigned task 0 for framework 20150224-232231-3142697795-38391-30589-0000 I0224 23:22:32.192847 30609 hierarchical.hpp:648] Recovered cpus(*):1; mem(*):524; disk(*):3.70122e+06; ports(*):[31000-32000] (total allocatable: cpus(*):1; mem(*):524; ...
Story points: 2