Project: mesos
Issue MESOS-4614: SlaveRecoveryTest/0.CleanupHTTPExecutor is flaky
Just saw this failure on the ASF CI: {code} [ RUN ] SlaveRecoveryTest/0.CleanupHTTPExecutor I0206 00:22:44.791671 2824 leveldb.cpp:174] Opened db in 2.539372ms I0206 00:22:44.792459 2824 leveldb.cpp:181] Compacted db in 740473ns I0206 00:22:44.792510 2824 leveldb.cpp:196] Created db iterator in 24164ns I0206 00:22:44.792532 2824 leveldb.cpp:202] Seeked to beginning of db in 1831ns I0206 00:22:44.792548 2824 leveldb.cpp:271] Iterated through 0 keys in the db in 342ns I0206 00:22:44.792605 2824 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0206 00:22:44.793256 2847 recover.cpp:447] Starting replica recovery I0206 00:22:44.793480 2847 recover.cpp:473] Replica is in EMPTY status I0206 00:22:44.794538 2847 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (9472)@172.17.0.2:43484 I0206 00:22:44.795040 2848 recover.cpp:193] Received a recover response from a replica in EMPTY status I0206 00:22:44.795644 2848 recover.cpp:564] Updating replica status to STARTING I0206 00:22:44.796519 2850 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 752810ns I0206 00:22:44.796545 2850 replica.cpp:320] Persisted replica status to STARTING I0206 00:22:44.796725 2848 recover.cpp:473] Replica is in STARTING status I0206 00:22:44.797828 2857 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (9473)@172.17.0.2:43484 I0206 00:22:44.798355 2850 recover.cpp:193] Received a recover response from a replica in STARTING status I0206 00:22:44.799193 2850 recover.cpp:564] Updating replica status to VOTING I0206 00:22:44.799583 2855 master.cpp:376] Master 0b206a40-a9c3-4d44-a5bd-8032d60a32ca (6632562f1ade) started on 172.17.0.2:43484 I0206 00:22:44.799609 2855 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/n2FxQV/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.28.0/_inst/share/mesos/webui" --work_dir="/tmp/n2FxQV/master" --zk_session_timeout="10secs" I0206 00:22:44.799991 2855 master.cpp:423] Master only allowing authenticated frameworks to register I0206 00:22:44.800009 2855 master.cpp:428] Master only allowing authenticated slaves to register I0206 00:22:44.800020 2855 credentials.hpp:35] Loading credentials for authentication from '/tmp/n2FxQV/credentials' I0206 00:22:44.800245 2850 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 679345ns I0206 00:22:44.800370 2850 replica.cpp:320] Persisted replica status to VOTING I0206 00:22:44.800397 2855 master.cpp:468] Using default 'crammd5' authenticator I0206 00:22:44.800693 2855 master.cpp:537] Using default 'basic' HTTP authenticator I0206 00:22:44.800815 2855 master.cpp:571] Authorization enabled I0206 00:22:44.801216 2850 recover.cpp:578] Successfully joined the Paxos group I0206 00:22:44.801604 2850 recover.cpp:462] Recover process terminated I0206 00:22:44.801759 2856 whitelist_watcher.cpp:77] No whitelist given I0206 00:22:44.801725 2847 hierarchical.cpp:144] Initialized hierarchical allocator process I0206 00:22:44.803982 2855 master.cpp:1712] The newly elected leader is master@172.17.0.2:43484 with id 0b206a40-a9c3-4d44-a5bd-8032d60a32ca I0206 00:22:44.804026 2855 master.cpp:1725] Elected as the leading master! I0206 00:22:44.804059 2855 master.cpp:1470] Recovering from registrar I0206 00:22:44.804424 2855 registrar.cpp:307] Recovering registrar I0206 00:22:44.805202 2855 log.cpp:659] Attempting to start the writer I0206 00:22:44.806782 2856 replica.cpp:493] Replica received implicit promise request from (9475)@172.17.0.2:43484 with proposal 1 I0206 00:22:44.807368 2856 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 547939ns I0206 00:22:44.807395 2856 replica.cpp:342] Persisted promised to 1 I0206 00:22:44.808375 2856 coordinator.cpp:238] Coordinator attempting to fill missing positions I0206 00:22:44.809460 2848 replica.cpp:388] Replica received explicit promise request from (9476)@172.17.0.2:43484 for position 0 with proposal 2 I0206 00:22:44.809929 2848 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 427561ns I0206 00:22:44.809967 2848 replica.cpp:712] Persisted action at 0 I0206 00:22:44.811035 2850 replica.cpp:537] Replica received write request for position 0 from (9477)@172.17.0.2:43484 I0206 00:22:44.811149 2850 leveldb.cpp:436] Reading position from leveldb took 36452ns I0206 00:22:44.811532 2850 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 318924ns I0206 00:22:44.811615 2850 replica.cpp:712] Persisted action at 0 I0206 00:22:44.812532 2850 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0206 00:22:44.813117 2850 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 476530ns I0206 00:22:44.813143 2850 replica.cpp:712] Persisted action at 0 I0206 00:22:44.813166 2850 replica.cpp:697] Replica learned NOP action at position 0 I0206 00:22:44.813984 2848 log.cpp:675] Writer started with ending position 0 I0206 00:22:44.815549 2848 leveldb.cpp:436] Reading position from leveldb took 31800ns I0206 00:22:44.817061 2848 registrar.cpp:340] Successfully fetched the registry (0B) in 12.591104ms I0206 00:22:44.817319 2848 registrar.cpp:439] Applied 1 operations in 63480ns; attempting to update the 'registry' I0206 00:22:44.818780 2845 log.cpp:683] Attempting to append 170 bytes to the log I0206 00:22:44.818981 2845 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0206 00:22:44.819941 2845 replica.cpp:537] Replica received write request for position 1 from (9478)@172.17.0.2:43484 I0206 00:22:44.820582 2845 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 600949ns I0206 00:22:44.820608 2845 replica.cpp:712] Persisted action at 1 I0206 00:22:44.821552 2845 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0206 00:22:44.821934 2845 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 352813ns I0206 00:22:44.821960 2845 replica.cpp:712] Persisted action at 1 I0206 00:22:44.821979 2845 replica.cpp:697] Replica learned APPEND action at position 1 I0206 00:22:44.823447 2845 registrar.cpp:484] Successfully updated the 'registry' in 5.987072ms I0206 00:22:44.823580 2845 registrar.cpp:370] Successfully recovered registrar I0206 00:22:44.823833 2845 log.cpp:702] Attempting to truncate the log to 1 I0206 00:22:44.824203 2845 master.cpp:1522] Recovered 0 slaves from the Registry (131B) ; allowing 10mins for slaves to re-register I0206 00:22:44.824291 2845 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0206 00:22:44.824645 2845 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover I0206 00:22:44.825222 2850 replica.cpp:537] Replica received write request for position 2 from (9479)@172.17.0.2:43484 I0206 00:22:44.825742 2850 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 481617ns I0206 00:22:44.825772 2850 replica.cpp:712] Persisted action at 2 I0206 00:22:44.826748 2852 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0206 00:22:44.827368 2852 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 588591ns I0206 00:22:44.827432 2852 leveldb.cpp:399] Deleting ~1 keys from leveldb took 33059ns I0206 00:22:44.827450 2852 replica.cpp:712] Persisted action at 2 I0206 00:22:44.827468 2852 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0206 00:22:44.838011 2824 containerizer.cpp:149] Using isolation: posix/cpu,posix/mem,filesystem/posix W0206 00:22:44.838873 2824 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I0206 00:22:44.843785 2857 slave.cpp:193] Slave started on 172.17.0.2:43484 I0206 00:22:44.843819 2857 slave.cpp:194] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_CleanupHTTPExecutor_kAXwvw/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="https://auth.docker.io" --docker_kill_orphans="true" --docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_CleanupHTTPExecutor_kAXwvw/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.28.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_CleanupHTTPExecutor_kAXwvw" I0206 00:22:44.844292 2857 credentials.hpp:83] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_CleanupHTTPExecutor_kAXwvw/credential' I0206 00:22:44.844518 2857 slave.cpp:324] Slave using credential for: test-principal I0206 00:22:44.844696 2857 resources.cpp:564] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0206 00:22:44.845243 2857 slave.cpp:464] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 00:22:44.845326 2857 slave.cpp:472] Slave attributes: [ ] I0206 00:22:44.845342 2857 slave.cpp:477] Slave hostname: 6632562f1ade I0206 00:22:44.845953 2824 sched.cpp:222] Version: 0.28.0 I0206 00:22:44.846853 2848 sched.cpp:326] New master detected at master@172.17.0.2:43484 I0206 00:22:44.846936 2848 sched.cpp:382] Authenticating with master master@172.17.0.2:43484 I0206 00:22:44.846958 2848 sched.cpp:389] Using default CRAM-MD5 authenticatee I0206 00:22:44.847692 2858 state.cpp:58] Recovering state from '/tmp/SlaveRecoveryTest_0_CleanupHTTPExecutor_kAXwvw/meta' I0206 00:22:44.848108 2850 status_update_manager.cpp:200] Recovering status update manager I0206 00:22:44.848325 2852 containerizer.cpp:397] Recovering containerizer I0206 00:22:44.848603 2845 authenticatee.cpp:121] Creating new client SASL connection I0206 00:22:44.849719 2845 master.cpp:5523] Authenticating scheduler-63899759-d7fc-42b2-8371-57484f352895@172.17.0.2:43484 I0206 00:22:44.850052 2852 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(662)@172.17.0.2:43484 I0206 00:22:44.850227 2854 provisioner.cpp:245] Provisioner recovery complete I0206 00:22:44.850410 2852 authenticator.cpp:98] Creating new server SASL connection I0206 00:22:44.850692 2852 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0206 00:22:44.850720 2852 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 00:22:44.850805 2852 authenticator.cpp:203] Received SASL authentication start I0206 00:22:44.850862 2852 authenticator.cpp:325] Authentication requires more steps I0206 00:22:44.850939 2852 authenticatee.cpp:258] Received SASL authentication step I0206 00:22:44.851027 2852 authenticator.cpp:231] Received SASL authentication step I0206 00:22:44.851052 2852 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '6632562f1ade' server FQDN: '6632562f1ade' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0206 00:22:44.851063 2852 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0206 00:22:44.851102 2852 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0206 00:22:44.851121 2852 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '6632562f1ade' server FQDN: '6632562f1ade' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0206 00:22:44.851130 2852 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0206 00:22:44.851136 2852 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0206 00:22:44.851150 2852 authenticator.cpp:317] Authentication success I0206 00:22:44.851219 2850 authenticatee.cpp:298] Authentication success I0206 00:22:44.851310 2850 master.cpp:5553] Successfully authenticated principal 'test-principal' at scheduler-63899759-d7fc-42b2-8371-57484f352895@172.17.0.2:43484 I0206 00:22:44.851485 2849 slave.cpp:4496] Finished recovery I0206 00:22:44.852154 2843 sched.cpp:471] Successfully authenticated with master master@172.17.0.2:43484 I0206 00:22:44.852175 2843 sched.cpp:776] Sending SUBSCRIBE call to master@172.17.0.2:43484 I0206 00:22:44.852262 2843 sched.cpp:809] Will retry registration in 939.183679ms if necessary I0206 00:22:44.852375 2844 master.cpp:2280] Received SUBSCRIBE call for framework 'default' at scheduler-63899759-d7fc-42b2-8371-57484f352895@172.17.0.2:43484 I0206 00:22:44.852448 2844 master.cpp:1751] Authorizing framework principal 'test-principal' to receive offers for role '*' I0206 00:22:44.852699 2852 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(662)@172.17.0.2:43484 I0206 00:22:44.852782 2844 master.cpp:2351] Subscribing framework default with checkpointing enabled and capabilities [ ] I0206 00:22:44.853056 2849 slave.cpp:4668] Querying resource estimator for oversubscribable resources I0206 00:22:44.853421 2856 hierarchical.cpp:265] Added framework 0b206a40-a9c3-4d44-a5bd-8032d60a32ca-0000 I0206 00:22:44.853513 2856 hierarchical.cpp:1403] No resources available to allocate! I0206 00:22:44.853582 2844 sched.cpp:703] Framework registered with 0b206a40-a9c3-4d44-a5bd-8032d60a32ca-0000 I0206 00:22:44.853613 2852 slave.cpp:4682] Received oversubscribable resources from the resource estimator I0206 00:22:44.853663 2844 sched.cpp:717] Scheduler::registered took 53762ns I0206 00:22:44.853899 2843 slave.cpp:796] New master detected at master@172.17.0.2:43484 I0206 00:22:44.853955 2854 status_update_manager.cpp:174] Pausing sending status updates I0206 00:22:44.853997 2856 hierarchical.cpp:1498] No inverse offers to send out! I0206 00:22:44.853960 2843 slave.cpp:859] Authenticating with master master@172.17.0.2:43484 I0206 00:22:44.854035 2843 slave.cpp:864] Using default CRAM-MD5 authenticatee I0206 00:22:44.854030 2856 hierarchical.cpp:1096] Performed allocation for 0 slaves in 581355ns I0206 00:22:44.854182 2843 slave.cpp:832] Detecting new master I0206 00:22:44.854277 2854 authenticatee.cpp:121] Creating new client SASL connection I0206 00:22:44.854517 2843 master.cpp:5523] Authenticating slave@172.17.0.2:43484 I0206 00:22:44.854603 2854 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(663)@172.17.0.2:43484 I0206 00:22:44.854836 2855 authenticator.cpp:98] Creating new server SASL connection I0206 00:22:44.855013 2852 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0206 00:22:44.855044 2852 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 00:22:44.855139 2855 authenticator.cpp:203] Received SASL authentication start I0206 00:22:44.855186 2855 authenticator.cpp:325] Authentication requires more steps I0206 00:22:44.855263 2855 authenticatee.cpp:258] Received SASL authentication step I0206 00:22:44.855352 2855 authenticator.cpp:231] Received SASL authentication step I0206 00:22:44.855381 2855 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '6632562f1ade' server FQDN: '6632562f1ade' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0206 00:22:44.855389 2855 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0206 00:22:44.855419 2855 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0206 00:22:44.855438 2855 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '6632562f1ade' server FQDN: '6632562f1ade' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0206 00:22:44.855448 2855 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0206 00:22:44.855453 2855 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0206 00:22:44.855464 2855 authenticator.cpp:317] Authentication success I0206 00:22:44.855540 2851 authenticatee.cpp:298] Authentication success I0206 00:22:44.855721 2851 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(663)@172.17.0.2:43484 I0206 00:22:44.855832 2852 slave.cpp:927] Successfully authenticated with master master@172.17.0.2:43484 I0206 00:22:44.855615 2855 master.cpp:5553] Successfully authenticated principal 'test-principal' at slave@172.17.0.2:43484 I0206 00:22:44.855973 2852 slave.cpp:1321] Will retry registration in 9.327708ms if necessary I0206 00:22:44.856145 2854 master.cpp:4237] Registering slave at slave@172.17.0.2:43484 (6632562f1ade) with id 0b206a40-a9c3-4d44-a5bd-8032d60a32ca-S0 I0206 00:22:44.856598 2851 registrar.cpp:439] Applied 1 operations in 59112ns; attempting to update the 'registry' I0206 00:22:44.857403 2851 log.cpp:683] Attempting to append 339 bytes to the log I0206 00:22:44.857525 2855 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0206 00:22:44.858482 2844 replica.cpp:537] Replica received write request for position 3 from (9493)@172.17.0.2:43484 I0206 00:22:44.858755 2844 leveldb.cpp:341] Persisting action (358 bytes) to leveldb took 228484ns I0206 00:22:44.858855 2844 replica.cpp:712] Persisted action at 3 I0206 00:22:44.859751 2852 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0206 00:22:44.860332 2852 leveldb.cpp:341] Persisting action (360 bytes) to leveldb took 549638ns I0206 00:22:44.860358 2852 replica.cpp:712] Persisted action at 3 I0206 00:22:44.860411 2852 replica.cpp:697] Replica learned APPEND action at position 3 I0206 00:22:44.862709 2856 registrar.cpp:484] Succe...
Story points: 3