Project: mesos
Issue MESOS-3311: SlaveTest.HTTPSchedulerSlaveRestart
Observed on ASF CI {code} [ RUN ] SlaveTest.HTTPSchedulerSlaveRestart Using temporary directory '/tmp/SlaveTest_HTTPSchedulerSlaveRestart_CXyDrA' I0825 22:07:36.809872 27610 leveldb.cpp:176] Opened db in 3.751801ms I0825 22:07:36.811115 27610 leveldb.cpp:183] Compacted db in 1.2194ms I0825 22:07:36.811175 27610 leveldb.cpp:198] Created db iterator in 30669ns I0825 22:07:36.811197 27610 leveldb.cpp:204] Seeked to beginning of db in 7829ns I0825 22:07:36.811208 27610 leveldb.cpp:273] Iterated through 0 keys in the db in 6017ns I0825 22:07:36.811245 27610 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0825 22:07:36.811722 27638 recover.cpp:449] Starting replica recovery I0825 22:07:36.811980 27638 recover.cpp:475] Replica is in EMPTY status I0825 22:07:36.813033 27641 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0825 22:07:36.813355 27635 recover.cpp:195] Received a recover response from a replica in EMPTY status I0825 22:07:36.813756 27628 recover.cpp:566] Updating replica status to STARTING I0825 22:07:36.814434 27636 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 570160ns I0825 22:07:36.814471 27636 replica.cpp:323] Persisted replica status to STARTING I0825 22:07:36.814743 27642 recover.cpp:475] Replica is in STARTING status I0825 22:07:36.814965 27638 master.cpp:378] Master 20150825-220736-234885548-51219-27610 (09c6504e3a31) started on 172.17.0.14:51219 I0825 22:07:36.814999 27638 master.cpp:380] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/SlaveTest_HTTPSchedulerSlaveRestart_CXyDrA/credentials" --framework_sorter="drf" --help="false" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.25.0/_inst/share/mesos/webui" --work_dir="/tmp/SlaveTest_HTTPSchedulerSlaveRestart_CXyDrA/master" --zk_session_timeout="10secs" I0825 22:07:36.815347 27638 master.cpp:425] Master only allowing authenticated frameworks to register I0825 22:07:36.815371 27638 master.cpp:430] Master only allowing authenticated slaves to register I0825 22:07:36.815402 27638 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveTest_HTTPSchedulerSlaveRestart_CXyDrA/credentials' I0825 22:07:36.815634 27632 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0825 22:07:36.815752 27638 master.cpp:469] Using default 'crammd5' authenticator I0825 22:07:36.815904 27638 master.cpp:506] Authorization enabled I0825 22:07:36.815979 27643 recover.cpp:195] Received a recover response from a replica in STARTING status I0825 22:07:36.816185 27637 whitelist_watcher.cpp:79] No whitelist given I0825 22:07:36.816186 27641 hierarchical.hpp:346] Initialized hierarchical allocator process I0825 22:07:36.816519 27630 recover.cpp:566] Updating replica status to VOTING I0825 22:07:36.817258 27639 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 475231ns I0825 22:07:36.817296 27639 replica.cpp:323] Persisted replica status to VOTING I0825 22:07:36.817420 27637 master.cpp:1525] The newly elected leader is master@172.17.0.14:51219 with id 20150825-220736-234885548-51219-27610 I0825 22:07:36.817467 27637 master.cpp:1538] Elected as the leading master! I0825 22:07:36.817483 27637 master.cpp:1308] Recovering from registrar I0825 22:07:36.817509 27635 recover.cpp:580] Successfully joined the Paxos group I0825 22:07:36.817708 27633 registrar.cpp:311] Recovering registrar I0825 22:07:36.817844 27635 recover.cpp:464] Recover process terminated I0825 22:07:36.818439 27631 log.cpp:661] Attempting to start the writer I0825 22:07:36.819694 27636 replica.cpp:477] Replica received implicit promise request with proposal 1 I0825 22:07:36.820133 27636 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 421255ns I0825 22:07:36.820168 27636 replica.cpp:345] Persisted promised to 1 I0825 22:07:36.820804 27630 coordinator.cpp:231] Coordinator attemping to fill missing position I0825 22:07:36.822105 27638 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0825 22:07:36.822597 27638 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 468065ns I0825 22:07:36.822625 27638 replica.cpp:679] Persisted action at 0 I0825 22:07:36.823737 27637 replica.cpp:511] Replica received write request for position 0 I0825 22:07:36.823796 27637 leveldb.cpp:438] Reading position from leveldb took 39603ns I0825 22:07:36.824267 27637 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 446655ns I0825 22:07:36.824296 27637 replica.cpp:679] Persisted action at 0 I0825 22:07:36.824961 27634 replica.cpp:658] Replica received learned notice for position 0 I0825 22:07:36.825340 27634 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 362236ns I0825 22:07:36.825369 27634 replica.cpp:679] Persisted action at 0 I0825 22:07:36.825388 27634 replica.cpp:664] Replica learned NOP action at position 0 I0825 22:07:36.825975 27642 log.cpp:677] Writer started with ending position 0 I0825 22:07:36.826997 27628 leveldb.cpp:438] Reading position from leveldb took 56us I0825 22:07:36.829946 27639 registrar.cpp:344] Successfully fetched the registry (0B) in 12.187136ms I0825 22:07:36.830077 27639 registrar.cpp:443] Applied 1 operations in 40874ns; attempting to update the 'registry' I0825 22:07:36.832870 27635 log.cpp:685] Attempting to append 174 bytes to the log I0825 22:07:36.833088 27641 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 1 I0825 22:07:36.833845 27636 replica.cpp:511] Replica received write request for position 1 I0825 22:07:36.834293 27636 leveldb.cpp:343] Persisting action (193 bytes) to leveldb took 425175ns I0825 22:07:36.834324 27636 replica.cpp:679] Persisted action at 1 I0825 22:07:36.835077 27643 replica.cpp:658] Replica received learned notice for position 1 I0825 22:07:36.835500 27643 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 404831ns I0825 22:07:36.835532 27643 replica.cpp:679] Persisted action at 1 I0825 22:07:36.835574 27643 replica.cpp:664] Replica learned APPEND action at position 1 I0825 22:07:36.836545 27643 registrar.cpp:488] Successfully updated the 'registry' in 6.393088ms I0825 22:07:36.836707 27643 registrar.cpp:374] Successfully recovered registrar I0825 22:07:36.836874 27639 log.cpp:704] Attempting to truncate the log to 1 I0825 22:07:36.837174 27632 master.cpp:1335] Recovered 0 slaves from the Registry (135B) ; allowing 10mins for slaves to re-register I0825 22:07:36.837291 27634 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 2 I0825 22:07:36.838249 27639 replica.cpp:511] Replica received write request for position 2 I0825 22:07:36.838685 27639 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 412214ns I0825 22:07:36.838716 27639 replica.cpp:679] Persisted action at 2 I0825 22:07:36.839735 27628 replica.cpp:658] Replica received learned notice for position 2 I0825 22:07:36.840304 27628 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 547841ns I0825 22:07:36.840375 27628 leveldb.cpp:401] Deleting ~1 keys from leveldb took 51256ns I0825 22:07:36.840401 27628 replica.cpp:679] Persisted action at 2 I0825 22:07:36.840428 27628 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0825 22:07:36.849371 27610 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix I0825 22:07:36.856500 27633 slave.cpp:190] Slave started on 286)@172.17.0.14:51219 I0825 22:07:36.856541 27633 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveTest_HTTPSchedulerSlaveRestart_ukkA8L/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveTest_HTTPSchedulerSlaveRestart_ukkA8L/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.25.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/SlaveTest_HTTPSchedulerSlaveRestart_ukkA8L" I0825 22:07:36.857074 27633 credentials.hpp:85] Loading credential for authentication from '/tmp/SlaveTest_HTTPSchedulerSlaveRestart_ukkA8L/credential' I0825 22:07:36.857275 27633 slave.cpp:321] Slave using credential for: test-principal I0825 22:07:36.857822 27633 slave.cpp:354] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0825 22:07:36.857936 27633 slave.cpp:384] Slave hostname: 09c6504e3a31 I0825 22:07:36.857959 27633 slave.cpp:389] Slave checkpoint: true I0825 22:07:36.858886 27637 state.cpp:54] Recovering state from '/tmp/SlaveTest_HTTPSchedulerSlaveRestart_ukkA8L/meta' I0825 22:07:36.859130 27638 status_update_manager.cpp:202] Recovering status update manager I0825 22:07:36.859465 27636 containerizer.cpp:379] Recovering containerizer I0825 22:07:36.860631 27634 slave.cpp:4069] Finished recovery I0825 22:07:36.861034 27634 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0825 22:07:36.861239 27643 status_update_manager.cpp:176] Pausing sending status updates I0825 22:07:36.861240 27634 slave.cpp:684] New master detected at master@172.17.0.14:51219 I0825 22:07:36.861322 27634 slave.cpp:747] Authenticating with master master@172.17.0.14:51219 I0825 22:07:36.861343 27634 slave.cpp:752] Using default CRAM-MD5 authenticatee I0825 22:07:36.861450 27634 slave.cpp:720] Detecting new master I0825 22:07:36.861495 27628 authenticatee.cpp:115] Creating new client SASL connection I0825 22:07:36.861569 27634 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0825 22:07:36.861716 27632 master.cpp:4694] Authenticating slave(286)@172.17.0.14:51219 I0825 22:07:36.861799 27629 authenticator.cpp:407] Starting authentication session for crammd5_authenticatee(665)@172.17.0.14:51219 I0825 22:07:36.862045 27642 authenticator.cpp:92] Creating new server SASL connection I0825 22:07:36.862308 27635 authenticatee.cpp:206] Received SASL authentication mechanisms: CRAM-MD5 I0825 22:07:36.862337 27635 authenticatee.cpp:232] Attempting to authenticate with mechanism 'CRAM-MD5' I0825 22:07:36.862421 27629 authenticator.cpp:197] Received SASL authentication start I0825 22:07:36.862478 27629 authenticator.cpp:319] Authentication requires more steps I0825 22:07:36.862579 27633 authenticatee.cpp:252] Received SASL authentication step I0825 22:07:36.862679 27628 authenticator.cpp:225] Received SASL authentication step I0825 22:07:36.862707 27628 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: '09c6504e3a31' server FQDN: '09c6504e3a31' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0825 22:07:36.862717 27628 auxprop.cpp:174] Looking up auxiliary property '*userPassword' I0825 22:07:36.862754 27628 auxprop.cpp:174] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0825 22:07:36.862785 27628 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: '09c6504e3a31' server FQDN: '09c6504e3a31' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0825 22:07:36.862797 27628 auxprop.cpp:124] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0825 22:07:36.862802 27628 auxprop.cpp:124] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0825 22:07:36.862817 27628 authenticator.cpp:311] Authentication success I0825 22:07:36.862884 27629 authenticatee.cpp:292] Authentication success I0825 22:07:36.862921 27630 master.cpp:4724] Successfully authenticated principal 'test-principal' at slave(286)@172.17.0.14:51219 I0825 22:07:36.862969 27642 authenticator.cpp:425] Authentication session cleanup for crammd5_authenticatee(665)@172.17.0.14:51219 I0825 22:07:36.863139 27639 slave.cpp:815] Successfully authenticated with master master@172.17.0.14:51219 I0825 22:07:36.863256 27639 slave.cpp:1209] Will retry registration in 15.028678ms if necessary I0825 22:07:36.863382 27643 master.cpp:3636] Registering slave at slave(286)@172.17.0.14:51219 (09c6504e3a31) with id 20150825-220736-234885548-51219-27610-S0 I0825 22:07:36.863899 27610 sched.cpp:164] Version: 0.25.0 I0825 22:07:36.863940 27636 registrar.cpp:443] Applied 1 operations in 94492ns; attempting to update the 'registry' I0825 22:07:36.864670 27632 sched.cpp:262] New master detected at master@172.17.0.14:51219 I0825 22:07:36.864790 27632 sched.cpp:318] Authenticating with master master@172.17.0.14:51219 I0825 22:07:36.864821 27632 sched.cpp:325] Using default CRAM-MD5 authenticatee I0825 22:07:36.865095 27637 authenticatee.cpp:115] Creating new client SASL connection I0825 22:07:36.865453 27643 master.cpp:4694] Authenticating scheduler-6c5ddcdb-9dd1-4b38-b051-5f714d3c1c55@172.17.0.14:51219 I0825 22:07:36.865603 27629 authenticator.cpp:407] Starting authentication session for crammd5_authenticatee(666)@172.17.0.14:51219 I0825 22:07:36.865840 27638 authenticator.cpp:92] Creating new server SASL connection I0825 22:07:36.866217 27630 authenticatee.cpp:206] Received SASL authentication mechanisms: CRAM-MD5 I0825 22:07:36.866260 27630 authenticatee.cpp:232] Attempting to authenticate with mechanism 'CRAM-MD5' I0825 22:07:36.866433 27639 authenticator.cpp:197] Received SASL authentication start I0825 22:07:36.866513 27639 authenticator.cpp:319] Authentication requires more steps I0825 22:07:36.866710 27630 authenticatee.cpp:252] Received SASL authentication step I0825 22:07:36.866999 27638 authenticator.cpp:225] Received SASL authentication step I0825 22:07:36.867051 27638 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: '09c6504e3a31' server FQDN: '09c6504e3a31' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0825 22:07:36.867077 27638 auxprop.cpp:174] Looking up auxiliary property '*userPassword' I0825 22:07:36.867130 27638 auxprop.cpp:174] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0825 22:07:36.867162 27638 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: '09c6504e3a31' server FQDN: '09c6504e3a31' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0825 22:07:36.867175 27638 auxprop.cpp:124] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0825 22:07:36.867183 27638 auxprop.cpp:124] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0825 22:07:36.867202 27638 authenticator.cpp:311] Authentication success I0825 22:07:36.867426 27636 authenticatee.cpp:292] Authentication success I0825 22:07:36.867434 27633 authenticator.cpp:425] Authentication session cleanup for crammd5_authenticatee(666)@172.17.0.14:51219 I0825 22:07:36.867627 27630 master.cpp:4724] Successfully authenticated principal 'test-principal' at scheduler-6c5ddcdb-9dd1-4b38-b051-5f714d3c1c55@172.17.0.14:51219 I0825 22:07:36.867951 27641 sched.cpp:407] Successfully authenticated with master master@172.17.0.14:51219 I0825 22:07:36.867986 27641 sched.cpp:713] Sending SUBSCRIBE call to master@172.17.0.14:51219 I0825 22:07:36.868114 27641 sched.cpp:746] Will retry registration in 1.352726078secs if necessary I0825 22:07:36.868233 27634 log.cpp:685] Attempting to append 344 bytes to the log I0825 22:07:36.868268 27638 master.cpp:2094] Received SUBSCRIBE call for framework 'default' at scheduler-6c5ddcdb-9dd1-4b38-b051-5f714d3c1c55@172.17.0.14:51219 I0825 22:07:36.868305 27638 master.cpp:1564] Authorizing framework principal 'test-principal' to receive offers for role '*' I0825 22:07:36.868373 27631 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 3 I0825 22:07:36.868614 27642 master.cpp:2164] Subscribing framework default with checkpointing enabled and capabilities [ ] I0825 22:07:36.868999 27643 hierarchical.hpp:391] Added framework 20150825-220736-234885548-51219-27610-0000 I0825 22:07:36.869030 27643 hierarchical.hpp:1010] No resources available to allocate! I0825 22:07:36.869046 27643 hierarchical.hpp:910] Performed allocation for 0 slaves in 34654ns I0825 22:07:36.869215 27631 sched.cpp:640] Framework registered with 20150825-220736-234885548-51219-27610-0000 I0825 22:07:36.869215 27643 replica.cpp:511] Replica received write request for position 3 I0825 22:07:36.869268 27631 sched.cpp:654] Scheduler::registered took 29976ns I0825 22:07:36.869453 27643 leveldb.cpp:343] Persisting action (363 bytes) to leveldb took 181689ns I0825 22:07:36.869477 27643 replica.cpp:679] Persisted action at 3 I0825 22:07:36.870075 27629 replica.cpp:658] Replica received learned notice for position 3 I0825 22:07:36.870542 27629 leveldb.cpp:343] Persisting action (365 bytes) to leveldb took 469081ns I0825 22:07:36.870589 27629 replica.cpp:679] Persisted action at 3 I0825 22:07:36.870622 27629 replica.cpp:664] Replica learned APPEND action at position 3 I0825 22:07:36.872133 27632 registrar.cpp:488] Successfully updated the 'registry' in 8.113152ms I0825 22:07:36.872354 27639 log.cpp:704] Attempting to truncate the log to 3 I0825 22:07:36.872470 27632 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 4 I0825 22:07:36.872879 27637 slave.cpp:3058] Received ping from slave-observer(274)@172.17.0.14:51219 I0825 22:07:36.873015 27636 master.cpp:3699] Registered slave 20150825-220736-234885548-51219-27610-S0 at slave(286)@172.17.0.14:51219 (09c6504e3a31) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0825 22:07:36.873180 27637 slave.cpp:859] Registered with master master@172.17.0.14:51219; given slave ID 20150825-220736-234885548-51219-27610-S0 I0825 22:07:36.873219 27637 fetcher.cpp:77] Clearing fetcher cache I0825 22:07:36.873410 27634 status_update_manager.cpp:183] Resuming sending status updates I0825 22:07:36.873379 27628 hierarchical.hpp:542] Added slave 20150825-220736-234885548-51219-27610-S0 (09c6504e3a31) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0825 22:07:36.873482 27642 replica.cpp:511] Replica received write request for position 4 I0825 22:0...
Story points: 2