Project: mesos
Issue MESOS-3579: FetcherCacheTest.LocalUncachedExtract is flaky
From ASF CI: https://builds.apache.org/job/Mesos/866/COMPILER=clang,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/console {code} [ RUN ] FetcherCacheTest.LocalUncachedExtract Using temporary directory '/tmp/FetcherCacheTest_LocalUncachedExtract_jHBfeA' I0925 19:15:39.541198 27410 leveldb.cpp:176] Opened db in 3.43934ms I0925 19:15:39.542362 27410 leveldb.cpp:183] Compacted db in 1.136184ms I0925 19:15:39.542428 27410 leveldb.cpp:198] Created db iterator in 35866ns I0925 19:15:39.542448 27410 leveldb.cpp:204] Seeked to beginning of db in 8807ns I0925 19:15:39.542459 27410 leveldb.cpp:273] Iterated through 0 keys in the db in 6325ns I0925 19:15:39.542505 27410 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0925 19:15:39.543143 27438 recover.cpp:449] Starting replica recovery I0925 19:15:39.543393 27438 recover.cpp:475] Replica is in EMPTY status I0925 19:15:39.544373 27436 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0925 19:15:39.544791 27433 recover.cpp:195] Received a recover response from a replica in EMPTY status I0925 19:15:39.545284 27433 recover.cpp:566] Updating replica status to STARTING I0925 19:15:39.546155 27436 master.cpp:376] Master c8bf1c95-50f4-4832-a570-c560f0b466ae (f57fd4291168) started on 172.17.1.195:41781 I0925 19:15:39.546257 27433 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 747249ns I0925 19:15:39.546288 27433 replica.cpp:323] Persisted replica status to STARTING I0925 19:15:39.546483 27434 recover.cpp:475] Replica is in STARTING status I0925 19:15:39.546187 27436 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/FetcherCacheTest_LocalUncachedExtract_jHBfeA/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/FetcherCacheTest_LocalUncachedExtract_jHBfeA/master" --zk_session_timeout="10secs" I0925 19:15:39.546567 27436 master.cpp:423] Master only allowing authenticated frameworks to register I0925 19:15:39.546617 27436 master.cpp:428] Master only allowing authenticated slaves to register I0925 19:15:39.546632 27436 credentials.hpp:37] Loading credentials for authentication from '/tmp/FetcherCacheTest_LocalUncachedExtract_jHBfeA/credentials' I0925 19:15:39.546931 27436 master.cpp:467] Using default 'crammd5' authenticator I0925 19:15:39.547044 27436 master.cpp:504] Authorization enabled I0925 19:15:39.547276 27441 whitelist_watcher.cpp:79] No whitelist given I0925 19:15:39.547320 27434 hierarchical.hpp:468] Initialized hierarchical allocator process I0925 19:15:39.547471 27438 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0925 19:15:39.548318 27443 recover.cpp:195] Received a recover response from a replica in STARTING status I0925 19:15:39.549067 27435 recover.cpp:566] Updating replica status to VOTING I0925 19:15:39.549115 27440 master.cpp:1603] The newly elected leader is master@172.17.1.195:41781 with id c8bf1c95-50f4-4832-a570-c560f0b466ae I0925 19:15:39.549162 27440 master.cpp:1616] Elected as the leading master! I0925 19:15:39.549190 27440 master.cpp:1376] Recovering from registrar I0925 19:15:39.549342 27434 registrar.cpp:309] Recovering registrar I0925 19:15:39.549666 27430 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 418187ns I0925 19:15:39.549753 27430 replica.cpp:323] Persisted replica status to VOTING I0925 19:15:39.550089 27442 recover.cpp:580] Successfully joined the Paxos group I0925 19:15:39.550320 27442 recover.cpp:464] Recover process terminated I0925 19:15:39.550904 27430 log.cpp:661] Attempting to start the writer I0925 19:15:39.551955 27434 replica.cpp:477] Replica received implicit promise request with proposal 1 I0925 19:15:39.552351 27434 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 380746ns I0925 19:15:39.552372 27434 replica.cpp:345] Persisted promised to 1 I0925 19:15:39.552896 27436 coordinator.cpp:231] Coordinator attemping to fill missing position I0925 19:15:39.554003 27432 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0925 19:15:39.554534 27432 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 510572ns I0925 19:15:39.554558 27432 replica.cpp:679] Persisted action at 0 I0925 19:15:39.555516 27443 replica.cpp:511] Replica received write request for position 0 I0925 19:15:39.555595 27443 leveldb.cpp:438] Reading position from leveldb took 65355ns I0925 19:15:39.556177 27443 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 542757ns I0925 19:15:39.556200 27443 replica.cpp:679] Persisted action at 0 I0925 19:15:39.556813 27429 replica.cpp:658] Replica received learned notice for position 0 I0925 19:15:39.557251 27429 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 422272ns I0925 19:15:39.557281 27429 replica.cpp:679] Persisted action at 0 I0925 19:15:39.557315 27429 replica.cpp:664] Replica learned NOP action at position 0 I0925 19:15:39.558061 27442 log.cpp:677] Writer started with ending position 0 I0925 19:15:39.559294 27434 leveldb.cpp:438] Reading position from leveldb took 56536ns I0925 19:15:39.560333 27432 registrar.cpp:342] Successfully fetched the registry (0B) in 10.936064ms I0925 19:15:39.560469 27432 registrar.cpp:441] Applied 1 operations in 41340ns; attempting to update the 'registry' I0925 19:15:39.561244 27441 log.cpp:685] Attempting to append 176 bytes to the log I0925 19:15:39.561378 27436 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 1 I0925 19:15:39.562126 27439 replica.cpp:511] Replica received write request for position 1 I0925 19:15:39.562515 27439 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 364968ns I0925 19:15:39.562539 27439 replica.cpp:679] Persisted action at 1 I0925 19:15:39.563160 27438 replica.cpp:658] Replica received learned notice for position 1 I0925 19:15:39.563699 27438 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 455933ns I0925 19:15:39.563730 27438 replica.cpp:679] Persisted action at 1 I0925 19:15:39.563753 27438 replica.cpp:664] Replica learned APPEND action at position 1 I0925 19:15:39.564749 27434 registrar.cpp:486] Successfully updated the 'registry' in 4.214016ms I0925 19:15:39.564893 27434 registrar.cpp:372] Successfully recovered registrar I0925 19:15:39.564950 27442 log.cpp:704] Attempting to truncate the log to 1 I0925 19:15:39.565039 27429 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 2 I0925 19:15:39.565172 27430 master.cpp:1413] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register I0925 19:15:39.565946 27429 replica.cpp:511] Replica received write request for position 2 I0925 19:15:39.566349 27429 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 375473ns I0925 19:15:39.566371 27429 replica.cpp:679] Persisted action at 2 I0925 19:15:39.566994 27431 replica.cpp:658] Replica received learned notice for position 2 I0925 19:15:39.567440 27431 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 437095ns I0925 19:15:39.567483 27431 leveldb.cpp:401] Deleting ~1 keys from leveldb took 31954ns I0925 19:15:39.567498 27431 replica.cpp:679] Persisted action at 2 I0925 19:15:39.567514 27431 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0925 19:15:39.576660 27410 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix W0925 19:15:39.577055 27410 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges I0925 19:15:39.583020 27443 slave.cpp:190] Slave started on 46)@172.17.1.195:41781 I0925 19:15:39.583062 27443 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resource_monitoring_interval="1secs" --resources="cpus(*):1000; mem(*):1000" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4" I0925 19:15:39.583472 27443 credentials.hpp:85] Loading credential for authentication from '/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4/credential' I0925 19:15:39.583752 27443 slave.cpp:321] Slave using credential for: test-principal I0925 19:15:39.584249 27443 slave.cpp:354] Slave resources: cpus(*):1000; mem(*):1000; disk(*):3.70122e+06; ports(*):[31000-32000] I0925 19:15:39.584344 27443 slave.cpp:390] Slave hostname: f57fd4291168 I0925 19:15:39.584362 27443 slave.cpp:395] Slave checkpoint: true I0925 19:15:39.585180 27428 state.cpp:54] Recovering state from '/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4/meta' I0925 19:15:39.585383 27440 status_update_manager.cpp:202] Recovering status update manager I0925 19:15:39.585636 27435 containerizer.cpp:386] Recovering containerizer I0925 19:15:39.586380 27438 slave.cpp:4110] Finished recovery I0925 19:15:39.586845 27438 slave.cpp:4267] Querying resource estimator for oversubscribable resources I0925 19:15:39.587059 27430 status_update_manager.cpp:176] Pausing sending status updates I0925 19:15:39.587064 27438 slave.cpp:705] New master detected at master@172.17.1.195:41781 I0925 19:15:39.587139 27438 slave.cpp:768] Authenticating with master master@172.17.1.195:41781 I0925 19:15:39.587163 27438 slave.cpp:773] Using default CRAM-MD5 authenticatee I0925 19:15:39.587321 27438 slave.cpp:741] Detecting new master I0925 19:15:39.587357 27434 authenticatee.cpp:115] Creating new client SASL connection I0925 19:15:39.587574 27438 slave.cpp:4281] Received oversubscribable resources from the resource estimator I0925 19:15:39.587739 27442 master.cpp:5138] Authenticating slave(46)@172.17.1.195:41781 I0925 19:15:39.587853 27441 authenticator.cpp:407] Starting authentication session for crammd5_authenticatee(139)@172.17.1.195:41781 I0925 19:15:39.588052 27439 authenticator.cpp:92] Creating new server SASL connection I0925 19:15:39.588248 27431 authenticatee.cpp:206] Received SASL authentication mechanisms: CRAM-MD5 I0925 19:15:39.588297 27431 authenticatee.cpp:232] Attempting to authenticate with mechanism 'CRAM-MD5' I0925 19:15:39.588443 27437 authenticator.cpp:197] Received SASL authentication start I0925 19:15:39.588506 27437 authenticator.cpp:319] Authentication requires more steps I0925 19:15:39.588677 27443 authenticatee.cpp:252] Received SASL authentication step I0925 19:15:39.588814 27436 authenticator.cpp:225] Received SASL authentication step I0925 19:15:39.588855 27436 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: 'f57fd4291168' server FQDN: 'f57fd4291168' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0925 19:15:39.588876 27436 auxprop.cpp:174] Looking up auxiliary property '*userPassword' I0925 19:15:39.588937 27436 auxprop.cpp:174] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0925 19:15:39.588979 27436 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: 'f57fd4291168' server FQDN: 'f57fd4291168' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0925 19:15:39.588997 27436 auxprop.cpp:124] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0925 19:15:39.589011 27436 auxprop.cpp:124] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0925 19:15:39.589036 27436 authenticator.cpp:311] Authentication success I0925 19:15:39.589126 27443 authenticatee.cpp:292] Authentication success I0925 19:15:39.589192 27437 master.cpp:5168] Successfully authenticated principal 'test-principal' at slave(46)@172.17.1.195:41781 I0925 19:15:39.589238 27433 authenticator.cpp:425] Authentication session cleanup for crammd5_authenticatee(139)@172.17.1.195:41781 I0925 19:15:39.589412 27440 slave.cpp:836] Successfully authenticated with master master@172.17.1.195:41781 I0925 19:15:39.589540 27440 slave.cpp:1230] Will retry registration in 13.562027ms if necessary I0925 19:15:39.589745 27436 master.cpp:3862] Registering slave at slave(46)@172.17.1.195:41781 (f57fd4291168) with id c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 I0925 19:15:39.590121 27438 registrar.cpp:441] Applied 1 operations in 70627ns; attempting to update the 'registry' I0925 19:15:39.590831 27430 log.cpp:685] Attempting to append 345 bytes to the log I0925 19:15:39.590927 27439 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 3 I0925 19:15:39.591809 27430 replica.cpp:511] Replica received write request for position 3 I0925 19:15:39.592072 27430 leveldb.cpp:343] Persisting action (364 bytes) to leveldb took 221734ns I0925 19:15:39.592099 27430 replica.cpp:679] Persisted action at 3 I0925 19:15:39.592643 27442 replica.cpp:658] Replica received learned notice for position 3 I0925 19:15:39.593215 27442 leveldb.cpp:343] Persisting action (366 bytes) to leveldb took 560946ns I0925 19:15:39.593237 27442 replica.cpp:679] Persisted action at 3 I0925 19:15:39.593255 27442 replica.cpp:664] Replica learned APPEND action at position 3 I0925 19:15:39.594663 27433 registrar.cpp:486] Successfully updated the 'registry' in 4.472832ms I0925 19:15:39.594874 27431 log.cpp:704] Attempting to truncate the log to 3 I0925 19:15:39.595407 27429 slave.cpp:3138] Received ping from slave-observer(45)@172.17.1.195:41781 I0925 19:15:39.595450 27433 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 4 I0925 19:15:39.596017 27442 replica.cpp:511] Replica received write request for position 4 I0925 19:15:39.596029 27429 hierarchical.hpp:675] Added slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 (f57fd4291168) with cpus(*):1000; mem(*):1000; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) I0925 19:15:39.595952 27441 master.cpp:3930] Registered slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 at slave(46)@172.17.1.195:41781 (f57fd4291168) with cpus(*):1000; mem(*):1000; disk(*):3.70122e+06; ports(*):[31000-32000] I0925 19:15:39.596240 27429 hierarchical.hpp:1326] No resources available to allocate! I0925 19:15:39.596263 27439 slave.cpp:880] Registered with master master@172.17.1.195:41781; given slave ID c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 I0925 19:15:39.596341 27439 fetcher.cpp:77] Clearing fetcher cache I0925 19:15:39.596345 27429 hierarchical.hpp:1421] No inverse offers to send out! I0925 19:15:39.596367 27429 hierarchical.hpp:1239] Performed allocation for slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 in 299337ns I0925 19:15:39.596524 27434 status_update_manager.cpp:183] Resuming sending status updates I0925 19:15:39.596571 27442 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 575374ns I0925 19:15:39.596662 27442 replica.cpp:679] Persisted action at 4 I0925 19:15:39.596984 27439 slave.cpp:903] Checkpointing SlaveInfo to '/tmp/FetcherCacheTest_LocalUncachedExtract_LwfzK4/meta/slaves/c8bf1c95-50f4-4832-a570-c560f0b466ae-S0/slave.info' I0925 19:15:39.597522 27434 replica.cpp:658] Replica received learned notice for position 4 I0925 19:15:39.597553 27410 sched.cpp:164] Version: 0.26.0 I0925 19:15:39.597746 27439 slave.cpp:939] Forwarding total oversubscribed resources I0925 19:15:39.598021 27429 master.cpp:4272] Received update of slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 at slave(46)@172.17.1.195:41781 (f57fd4291168) with total oversubscribed resources I0925 19:15:39.598070 27434 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 531503ns I0925 19:15:39.598162 27434 leveldb.cpp:401] Deleting ~2 keys from leveldb took 79081ns I0925 19:15:39.598170 27428 sched.cpp:262] New master detected at master@172.17.1.195:41781 I0925 19:15:39.598206 27434 replica.cpp:679] Persisted action at 4 I0925 19:15:39.598238 27434 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0925 19:15:39.598276 27428 sched.cpp:318] Authenticating with master master@172.17.1.195:41781 I0925 19:15:39.598296 27428 sched.cpp:325] Using default CRAM-MD5 authenticatee I0925 19:15:39.598950 27430 hierarchical.hpp:735] Slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 (f57fd4291168) updated with oversubscribed resources (total: cpus(*):1000; mem(*):1000; disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) I0925 19:15:39.599242 27430 hierarchical.hpp:1326] No resources available to allocate! I0925 19:15:39.599282 27430 hierarchical.hpp:1421] No inverse offers to send out! I0925 19:15:39.599341 27430 hierarchical.hpp:1239] Performed allocation for slave c8bf1c95-50f4-4832-a570-c560f0b466ae-S0 in 327742ns I0925 19:15:39.599632 27437 authenticatee.cpp:115] Creating new client SASL connection I0925 19:15:39.600005 27428 master.cpp:5138] Authenticating scheduler-dda30e8e-47b7-4b1d-9a96-32364754be63@172.17.1.195:41781 I0925 19:15:39.600170 27435 authenticator.cpp:407] Starting authentication session for crammd5_authenticatee(140)@172.17.1.195:41781 I0925 19:15:39.600518 27433 authenticator.cpp:92] Creating new server SASL connection I0925 19:15:39.600788 27436 authenticatee.cpp:206] Received SASL authentication mechanisms: CRAM-MD5 I0925 19:15:39.600831 27436 authenticatee.cpp:232] Attempting to authenticate with mechanism 'CRAM-MD5' I0925 19:15:39.600944 27433 authenticator.cpp:197] Received SASL authentication start I0925 19:15:39.601019 27433 authenticator.cpp:319] Authentication requires more steps I0925 19:15:39.601150 27436 authenticatee.cpp:252] Received SASL authentication step I0925 19:15:39.601284 27436 authenticator.cpp:225] Received SASL authentication step I0925 19:15:39.601326 27436 auxprop.cpp:102] Request to lookup properties for user: 'test-principal' realm: 'f57fd4291168' server FQDN: 'f...
Story points: 2