Project: mesos
Issue MESOS-2008: MasterAuthorizationTest.DuplicateReregistration is flaky
{noformat:title=} [ RUN ] MasterAuthorizationTest.DuplicateReregistration Using temporary directory '/tmp/MasterAuthorizationTest_DuplicateReregistration_DLOmYX' I1029 08:25:26.021766 32232 leveldb.cpp:176] Opened db in 3.066621ms I1029 08:25:26.022734 32232 leveldb.cpp:183] Compacted db in 935019ns I1029 08:25:26.022766 32232 leveldb.cpp:198] Created db iterator in 4350ns I1029 08:25:26.022785 32232 leveldb.cpp:204] Seeked to beginning of db in 902ns I1029 08:25:26.022799 32232 leveldb.cpp:273] Iterated through 0 keys in the db in 387ns I1029 08:25:26.022831 32232 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1029 08:25:26.023305 32248 recover.cpp:437] Starting replica recovery I1029 08:25:26.023598 32248 recover.cpp:463] Replica is in EMPTY status I1029 08:25:26.025059 32260 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I1029 08:25:26.025320 32247 recover.cpp:188] Received a recover response from a replica in EMPTY status I1029 08:25:26.025585 32256 recover.cpp:554] Updating replica status to STARTING I1029 08:25:26.026546 32249 master.cpp:312] Master 20141029-082526-3142697795-40696-32232 (pomona.apache.org) started on 67.195.81.187:40696 I1029 08:25:26.026561 32261 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 694444ns I1029 08:25:26.026592 32249 master.cpp:358] Master only allowing authenticated frameworks to register I1029 08:25:26.026592 32261 replica.cpp:320] Persisted replica status to STARTING I1029 08:25:26.026605 32249 master.cpp:363] Master only allowing authenticated slaves to register I1029 08:25:26.026639 32249 credentials.hpp:36] Loading credentials for authentication from '/tmp/MasterAuthorizationTest_DuplicateReregistration_DLOmYX/credentials' I1029 08:25:26.026877 32249 master.cpp:392] Authorization enabled I1029 08:25:26.026901 32260 recover.cpp:463] Replica is in STARTING status I1029 08:25:26.027498 32261 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:26.027541 32248 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@67.195.81.187:40696 I1029 08:25:26.028055 32252 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I1029 08:25:26.028451 32247 recover.cpp:188] Received a recover response from a replica in STARTING status I1029 08:25:26.028733 32249 master.cpp:1242] The newly elected leader is master@67.195.81.187:40696 with id 20141029-082526-3142697795-40696-32232 I1029 08:25:26.028764 32249 master.cpp:1255] Elected as the leading master! I1029 08:25:26.028781 32249 master.cpp:1073] Recovering from registrar I1029 08:25:26.028904 32246 recover.cpp:554] Updating replica status to VOTING I1029 08:25:26.029163 32257 registrar.cpp:313] Recovering registrar I1029 08:25:26.029556 32251 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 485711ns I1029 08:25:26.029588 32251 replica.cpp:320] Persisted replica status to VOTING I1029 08:25:26.029726 32253 recover.cpp:568] Successfully joined the Paxos group I1029 08:25:26.029932 32253 recover.cpp:452] Recover process terminated I1029 08:25:26.030436 32250 log.cpp:656] Attempting to start the writer I1029 08:25:26.032152 32248 replica.cpp:474] Replica received implicit promise request with proposal 1 I1029 08:25:26.032778 32248 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 597030ns I1029 08:25:26.032807 32248 replica.cpp:342] Persisted promised to 1 I1029 08:25:26.033481 32254 coordinator.cpp:230] Coordinator attemping to fill missing position I1029 08:25:26.035429 32247 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I1029 08:25:26.036154 32247 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 690208ns I1029 08:25:26.036181 32247 replica.cpp:676] Persisted action at 0 I1029 08:25:26.037344 32249 replica.cpp:508] Replica received write request for position 0 I1029 08:25:26.037395 32249 leveldb.cpp:438] Reading position from leveldb took 22607ns I1029 08:25:26.038074 32249 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 647429ns I1029 08:25:26.038105 32249 replica.cpp:676] Persisted action at 0 I1029 08:25:26.038683 32247 replica.cpp:655] Replica received learned notice for position 0 I1029 08:25:26.039378 32247 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 664911ns I1029 08:25:26.039407 32247 replica.cpp:676] Persisted action at 0 I1029 08:25:26.039433 32247 replica.cpp:661] Replica learned NOP action at position 0 I1029 08:25:26.040045 32252 log.cpp:672] Writer started with ending position 0 I1029 08:25:26.041378 32251 leveldb.cpp:438] Reading position from leveldb took 25625ns I1029 08:25:26.044642 32246 registrar.cpp:346] Successfully fetched the registry (0B) in 15.433984ms I1029 08:25:26.044742 32246 registrar.cpp:445] Applied 1 operations in 16444ns; attempting to update the 'registry' I1029 08:25:26.047538 32256 log.cpp:680] Attempting to append 139 bytes to the log I1029 08:25:26.156330 32247 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I1029 08:25:26.158460 32261 replica.cpp:508] Replica received write request for position 1 I1029 08:25:26.159277 32261 leveldb.cpp:343] Persisting action (158 bytes) to leveldb took 782308ns I1029 08:25:26.159328 32261 replica.cpp:676] Persisted action at 1 I1029 08:25:26.160267 32255 replica.cpp:655] Replica received learned notice for position 1 I1029 08:25:26.161070 32255 leveldb.cpp:343] Persisting action (160 bytes) to leveldb took 750259ns I1029 08:25:26.161100 32255 replica.cpp:676] Persisted action at 1 I1029 08:25:26.161125 32255 replica.cpp:661] Replica learned APPEND action at position 1 I1029 08:25:26.162199 32253 registrar.cpp:490] Successfully updated the 'registry' in 117.40416ms I1029 08:25:26.162400 32253 registrar.cpp:376] Successfully recovered registrar I1029 08:25:26.162724 32249 master.cpp:1100] Recovered 0 slaves from the Registry (101B) ; allowing 10mins for slaves to re-register I1029 08:25:26.162757 32253 log.cpp:699] Attempting to truncate the log to 1 I1029 08:25:26.162919 32256 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I1029 08:25:26.163949 32250 replica.cpp:508] Replica received write request for position 2 I1029 08:25:26.164589 32250 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 603175ns I1029 08:25:26.164618 32250 replica.cpp:676] Persisted action at 2 I1029 08:25:26.165385 32251 replica.cpp:655] Replica received learned notice for position 2 I1029 08:25:26.166007 32251 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 594003ns I1029 08:25:26.166056 32251 leveldb.cpp:401] Deleting ~1 keys from leveldb took 23309ns I1029 08:25:26.166077 32251 replica.cpp:676] Persisted action at 2 I1029 08:25:26.166100 32251 replica.cpp:661] Replica learned TRUNCATE action at position 2 I1029 08:25:26.178493 32232 sched.cpp:137] Version: 0.21.0 I1029 08:25:26.179029 32256 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:26.179078 32256 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:26.179424 32246 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:26.179678 32259 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.179970 32250 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:26.180165 32250 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:26.180191 32250 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:26.180272 32250 authenticator.hpp:267] Received SASL authentication start I1029 08:25:26.180378 32250 authenticator.hpp:389] Authentication requires more steps I1029 08:25:26.180557 32260 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:26.180704 32254 authenticator.hpp:295] Received SASL authentication step I1029 08:25:26.180737 32254 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:26.180748 32254 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:26.180780 32254 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:26.180804 32254 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:26.180816 32254 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.180824 32254 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.180841 32254 authenticator.hpp:381] Authentication success I1029 08:25:26.180937 32259 authenticatee.hpp:310] Authentication success I1029 08:25:26.180991 32260 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.181422 32259 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:26.181449 32259 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:26.181697 32260 master.cpp:1362] Received registration request for framework 'default' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.181758 32260 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:26.182063 32260 master.cpp:1426] Registering framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.182430 32248 hierarchical_allocator_process.hpp:329] Added framework 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:26.182462 32248 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:26.182462 32261 sched.cpp:407] Framework registered with 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:26.182473 32248 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 15372ns I1029 08:25:26.182554 32261 sched.cpp:421] Scheduler::registered took 60059ns I1029 08:25:26.185515 32260 sched.cpp:227] Scheduler::disconnected took 16607ns I1029 08:25:26.185538 32260 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:26.185567 32260 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:26.185783 32246 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:26.186218 32250 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.186456 32247 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:26.186594 32250 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:26.186621 32250 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:26.186745 32259 authenticator.hpp:267] Received SASL authentication start I1029 08:25:26.186800 32259 authenticator.hpp:389] Authentication requires more steps I1029 08:25:26.186936 32260 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:26.187062 32249 authenticator.hpp:295] Received SASL authentication step I1029 08:25:26.187095 32249 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:26.187108 32249 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:26.187137 32249 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:26.187162 32249 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:26.187175 32249 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.187182 32249 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.187199 32249 authenticator.hpp:381] Authentication success I1029 08:25:26.187327 32249 authenticatee.hpp:310] Authentication success I1029 08:25:26.187366 32260 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.187631 32249 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:26.187659 32249 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:27.028445 32251 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:28.045682 32251 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 1.017231941secs I1029 08:25:28.045760 32249 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:28.045900 32253 master.cpp:1499] Received re-registration request from framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.045989 32253 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:28.046455 32253 master.cpp:1499] Received re-registration request from framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.046529 32253 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:28.050155 32247 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:28.050217 32247 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:28.050405 32252 master.cpp:1552] Re-registering framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.050509 32253 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:28.050566 32252 master.cpp:1592] Allowing framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 to re-register with an already used id I1029 08:25:28.051084 32257 sched.cpp:449] Framework re-registered with 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:28.051151 32252 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.051167 32257 sched.cpp:463] Scheduler::reregistered took 52801ns I1029 08:25:28.051723 32261 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:28.052042 32249 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:28.052077 32249 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:28.052170 32249 master.cpp:1534] Dropping re-registration request of framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 because new authentication attempt is in progress I1029 08:25:28.052218 32257 authenticator.hpp:267] Received SASL authentication start I1029 08:25:28.052325 32257 authenticator.hpp:389] Authentication requires more steps I1029 08:25:28.052428 32257 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:28.052641 32246 authenticator.hpp:295] Received SASL authentication step I1029 08:25:28.052685 32246 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:28.052701 32246 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:28.052739 32246 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:28.052767 32246 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:28.052779 32246 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:28.052788 32246 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:28.052804 32246 authenticator.hpp:381] Authentication success I1029 08:25:28.052947 32252 authenticatee.hpp:310] Authentication success I1029 08:25:28.053020 32246 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.053462 32247 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:29.046855 32261 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:29.046880 32261 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 35632ns I1029 08:25:30.047458 32253 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:30.047487 32253 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 43031ns I1029 08:25:31.028373 32261 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:31.048673 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:31.048702 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 44769ns I1029 08:25:32.049576 32259 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:32.049604 32259 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 51919ns I1029 08:25:33.050864 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:33.050896 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38019ns I1029 08:25:34.051961 32251 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:34.051993 32251 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 64619ns I1029 08:25:35.052196 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:35.052223 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 34475ns I1029 08:25:36.029101 32259 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:36.053067 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:36.053095 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38354ns I1029 08:25:37.053506 32259 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:37.053536 32259 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38249ns tests/master_authorization_tests.cpp:877: Failure Failed to wait 10secs for frameworkReregisteredMessage I1029 08:25:38.053241 32259 master.cpp:768] Framework 2014...
Story points: 2