Project: mesos
Issue MESOS-4208: PersistentVolumeTest.BadACLDropCreateAndDestroy is flaky
{noformat} [ RUN ] PersistentVolumeTest.BadACLDropCreateAndDestroy I1219 09:51:32.623245 31878 leveldb.cpp:174] Opened db in 4.393596ms I1219 09:51:32.624084 31878 leveldb.cpp:181] Compacted db in 709447ns I1219 09:51:32.624186 31878 leveldb.cpp:196] Created db iterator in 21252ns I1219 09:51:32.624290 31878 leveldb.cpp:202] Seeked to beginning of db in 11391ns I1219 09:51:32.624378 31878 leveldb.cpp:271] Iterated through 0 keys in the db in 611ns I1219 09:51:32.624505 31878 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1219 09:51:32.625195 31904 recover.cpp:447] Starting replica recovery I1219 09:51:32.625641 31904 recover.cpp:473] Replica is in EMPTY status I1219 09:51:32.627305 31904 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (6740)@172.17.0.3:36408 I1219 09:51:32.627749 31904 recover.cpp:193] Received a recover response from a replica in EMPTY status I1219 09:51:32.628330 31904 recover.cpp:564] Updating replica status to STARTING I1219 09:51:32.629068 31906 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 410494ns I1219 09:51:32.629169 31906 replica.cpp:320] Persisted replica status to STARTING I1219 09:51:32.629598 31906 recover.cpp:473] Replica is in STARTING status I1219 09:51:32.630782 31912 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (6741)@172.17.0.3:36408 I1219 09:51:32.631166 31901 recover.cpp:193] Received a recover response from a replica in STARTING status I1219 09:51:32.632467 31902 recover.cpp:564] Updating replica status to VOTING I1219 09:51:32.633600 31907 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 311370ns I1219 09:51:32.633627 31907 replica.cpp:320] Persisted replica status to VOTING I1219 09:51:32.633719 31907 recover.cpp:578] Successfully joined the Paxos group I1219 09:51:32.633874 31907 recover.cpp:462] Recover process terminated I1219 09:51:32.636409 31909 master.cpp:365] Master bded856d-1c7f-4fad-a8bc-3629ba8c59d3 (60ab6e727501) started on 172.17.0.3:36408 I1219 09:51:32.636593 31909 master.cpp:367] Flags at startup: --acls="create_volumes { principals { values: "creator-principal" } volume_types { type: ANY } } create_volumes { principals { type: ANY } volume_types { type: NONE } } " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/SpPF7B/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.27.0/_inst/share/mesos/webui" --work_dir="/tmp/SpPF7B/master" --zk_session_timeout="10secs" I1219 09:51:32.637055 31909 master.cpp:414] Master allowing unauthenticated frameworks to register I1219 09:51:32.637068 31909 master.cpp:417] Master only allowing authenticated slaves to register I1219 09:51:32.637094 31909 credentials.hpp:35] Loading credentials for authentication from '/tmp/SpPF7B/credentials' I1219 09:51:32.637403 31909 master.cpp:456] Using default 'crammd5' authenticator I1219 09:51:32.637555 31909 master.cpp:493] Authorization enabled W1219 09:51:32.637575 31909 master.cpp:553] The '--roles' flag is deprecated. This flag will be removed in the future. See the Mesos 0.27 upgrade notes for more information I1219 09:51:32.637806 31897 whitelist_watcher.cpp:77] No whitelist given I1219 09:51:32.637820 31910 hierarchical.cpp:147] Initialized hierarchical allocator process I1219 09:51:32.639677 31909 master.cpp:1629] The newly elected leader is master@172.17.0.3:36408 with id bded856d-1c7f-4fad-a8bc-3629ba8c59d3 I1219 09:51:32.639768 31909 master.cpp:1642] Elected as the leading master! I1219 09:51:32.639892 31909 master.cpp:1387] Recovering from registrar I1219 09:51:32.640136 31907 registrar.cpp:307] Recovering registrar I1219 09:51:32.640929 31901 log.cpp:659] Attempting to start the writer I1219 09:51:32.642199 31912 replica.cpp:493] Replica received implicit promise request from (6742)@172.17.0.3:36408 with proposal 1 I1219 09:51:32.642719 31912 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 445876ns I1219 09:51:32.642755 31912 replica.cpp:342] Persisted promised to 1 I1219 09:51:32.643478 31904 coordinator.cpp:238] Coordinator attempting to fill missing positions I1219 09:51:32.645009 31909 replica.cpp:388] Replica received explicit promise request from (6743)@172.17.0.3:36408 for position 0 with proposal 2 I1219 09:51:32.645356 31909 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 310064ns I1219 09:51:32.645382 31909 replica.cpp:712] Persisted action at 0 I1219 09:51:32.646662 31909 replica.cpp:537] Replica received write request for position 0 from (6744)@172.17.0.3:36408 I1219 09:51:32.646721 31909 leveldb.cpp:436] Reading position from leveldb took 29298ns I1219 09:51:32.647047 31909 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 283424ns I1219 09:51:32.647073 31909 replica.cpp:712] Persisted action at 0 I1219 09:51:32.647722 31909 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I1219 09:51:32.648052 31909 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 300825ns I1219 09:51:32.648077 31909 replica.cpp:712] Persisted action at 0 I1219 09:51:32.648095 31909 replica.cpp:697] Replica learned NOP action at position 0 I1219 09:51:32.655295 31899 log.cpp:675] Writer started with ending position 0 I1219 09:51:32.656543 31905 leveldb.cpp:436] Reading position from leveldb took 32788ns I1219 09:51:32.658164 31905 registrar.cpp:340] Successfully fetched the registry (0B) in 0ns I1219 09:51:32.658604 31905 registrar.cpp:439] Applied 1 operations in 38183ns; attempting to update the 'registry' I1219 09:51:32.660102 31905 log.cpp:683] Attempting to append 170 bytes to the log I1219 09:51:32.660538 31906 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I1219 09:51:32.661872 31906 replica.cpp:537] Replica received write request for position 1 from (6745)@172.17.0.3:36408 I1219 09:51:32.662719 31906 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 483018ns I1219 09:51:32.663054 31906 replica.cpp:712] Persisted action at 1 I1219 09:51:32.664008 31902 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I1219 09:51:32.664330 31902 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 287310ns I1219 09:51:32.664355 31902 replica.cpp:712] Persisted action at 1 I1219 09:51:32.664376 31902 replica.cpp:697] Replica learned APPEND action at position 1 I1219 09:51:32.665365 31902 registrar.cpp:484] Successfully updated the 'registry' in 0ns I1219 09:51:32.665493 31902 registrar.cpp:370] Successfully recovered registrar I1219 09:51:32.665894 31902 master.cpp:1439] Recovered 0 slaves from the Registry (131B) ; allowing 10mins for slaves to re-register I1219 09:51:32.665990 31902 hierarchical.cpp:165] Skipping recovery of hierarchical allocator: nothing to recover I1219 09:51:32.666266 31902 log.cpp:702] Attempting to truncate the log to 1 I1219 09:51:32.666424 31902 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I1219 09:51:32.667181 31907 replica.cpp:537] Replica received write request for position 2 from (6746)@172.17.0.3:36408 I1219 09:51:32.667768 31907 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 335947ns I1219 09:51:32.668067 31907 replica.cpp:712] Persisted action at 2 I1219 09:51:32.668942 31906 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I1219 09:51:32.669240 31906 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 266566ns I1219 09:51:32.669292 31906 leveldb.cpp:399] Deleting ~1 keys from leveldb took 27852ns I1219 09:51:32.669314 31906 replica.cpp:712] Persisted action at 2 I1219 09:51:32.669334 31906 replica.cpp:697] Replica learned TRUNCATE action at position 2 I1219 09:51:32.691251 31878 containerizer.cpp:141] Using isolation: posix/cpu,posix/mem,filesystem/posix W1219 09:51:32.691759 31878 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I1219 09:51:32.697428 31901 slave.cpp:191] Slave started on 228)@172.17.0.3:36408 I1219 09:51:32.697459 31901 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.27.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):2048" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc" I1219 09:51:32.697963 31901 credentials.hpp:83] Loading credential for authentication from '/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc/credential' I1219 09:51:32.698210 31901 slave.cpp:322] Slave using credential for: test-principal I1219 09:51:32.698449 31901 resources.cpp:478] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):2048 Trying semicolon-delimited string format instead I1219 09:51:32.699065 31901 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] I1219 09:51:32.699137 31901 slave.cpp:400] Slave attributes: [ ] I1219 09:51:32.699151 31901 slave.cpp:405] Slave hostname: 60ab6e727501 I1219 09:51:32.699161 31901 slave.cpp:410] Slave checkpoint: true I1219 09:51:32.699364 31878 sched.cpp:164] Version: 0.27.0 I1219 09:51:32.700614 31911 sched.cpp:262] New master detected at master@172.17.0.3:36408 I1219 09:51:32.700703 31911 sched.cpp:272] No credentials provided. Attempting to register without authentication I1219 09:51:32.700724 31911 sched.cpp:714] Sending SUBSCRIBE call to master@172.17.0.3:36408 I1219 09:51:32.700839 31911 sched.cpp:747] Will retry registration in 620.399428ms if necessary I1219 09:51:32.701244 31903 master.cpp:2197] Received SUBSCRIBE call for framework 'default' at scheduler-0333dddc-4b41-40ed-8853-a1aadf1f1879@172.17.0.3:36408 I1219 09:51:32.701313 31903 master.cpp:1668] Authorizing framework principal 'test-principal' to receive offers for role 'role1' I1219 09:51:32.701625 31903 master.cpp:2268] Subscribing framework default with checkpointing disabled and capabilities [ ] I1219 09:51:32.702308 31903 hierarchical.cpp:260] Added framework bded856d-1c7f-4fad-a8bc-3629ba8c59d3-0000 I1219 09:51:32.702386 31903 hierarchical.cpp:1329] No resources available to allocate! I1219 09:51:32.702422 31903 hierarchical.cpp:1423] No inverse offers to send out! I1219 09:51:32.702448 31903 hierarchical.cpp:1079] Performed allocation for 0 slaves in 114358ns I1219 09:51:32.702638 31903 sched.cpp:641] Framework registered with bded856d-1c7f-4fad-a8bc-3629ba8c59d3-0000 I1219 09:51:32.702688 31903 sched.cpp:655] Scheduler::registered took 25558ns I1219 09:51:32.703553 31901 state.cpp:58] Recovering state from '/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc/meta' I1219 09:51:32.704118 31897 status_update_manager.cpp:200] Recovering status update manager I1219 09:51:32.704407 31907 containerizer.cpp:383] Recovering containerizer I1219 09:51:32.705373 31907 slave.cpp:4427] Finished recovery I1219 09:51:32.705991 31907 slave.cpp:4599] Querying resource estimator for oversubscribable resources I1219 09:51:32.706277 31907 slave.cpp:4613] Received oversubscribable resources from the resource estimator I1219 09:51:32.706666 31907 slave.cpp:729] New master detected at master@172.17.0.3:36408 I1219 09:51:32.706738 31907 slave.cpp:792] Authenticating with master master@172.17.0.3:36408 I1219 09:51:32.706760 31907 slave.cpp:797] Using default CRAM-MD5 authenticatee I1219 09:51:32.706886 31899 status_update_manager.cpp:174] Pausing sending status updates I1219 09:51:32.706941 31907 slave.cpp:765] Detecting new master I1219 09:51:32.707036 31899 authenticatee.cpp:121] Creating new client SASL connection I1219 09:51:32.707291 31910 master.cpp:5423] Authenticating slave(228)@172.17.0.3:36408 I1219 09:51:32.707479 31910 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(510)@172.17.0.3:36408 I1219 09:51:32.707849 31910 authenticator.cpp:98] Creating new server SASL connection I1219 09:51:32.708082 31910 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I1219 09:51:32.708112 31910 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I1219 09:51:32.708196 31910 authenticator.cpp:203] Received SASL authentication start I1219 09:51:32.708395 31910 authenticator.cpp:325] Authentication requires more steps I1219 09:51:32.708611 31902 authenticatee.cpp:258] Received SASL authentication step I1219 09:51:32.708773 31910 authenticator.cpp:231] Received SASL authentication step I1219 09:51:32.708889 31910 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '60ab6e727501' server FQDN: '60ab6e727501' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1219 09:51:32.708976 31910 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I1219 09:51:32.709096 31910 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1219 09:51:32.709200 31910 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '60ab6e727501' server FQDN: '60ab6e727501' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1219 09:51:32.709285 31910 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1219 09:51:32.709363 31910 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1219 09:51:32.709452 31910 authenticator.cpp:317] Authentication success I1219 09:51:32.709707 31910 authenticatee.cpp:298] Authentication success I1219 09:51:32.710252 31910 slave.cpp:860] Successfully authenticated with master master@172.17.0.3:36408 I1219 09:51:32.710525 31910 slave.cpp:1254] Will retry registration in 17.44437ms if necessary I1219 09:51:32.709839 31908 master.cpp:5453] Successfully authenticated principal 'test-principal' at slave(228)@172.17.0.3:36408 I1219 09:51:32.710985 31908 master.cpp:4132] Registering slave at slave(228)@172.17.0.3:36408 (60ab6e727501) with id bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 I1219 09:51:32.711645 31908 registrar.cpp:439] Applied 1 operations in 83191ns; attempting to update the 'registry' I1219 09:51:32.709908 31912 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(510)@172.17.0.3:36408 I1219 09:51:32.713407 31908 log.cpp:683] Attempting to append 343 bytes to the log I1219 09:51:32.713646 31912 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I1219 09:51:32.714884 31911 replica.cpp:537] Replica received write request for position 3 from (6758)@172.17.0.3:36408 I1219 09:51:32.715221 31911 leveldb.cpp:341] Persisting action (362 bytes) to leveldb took 288909ns I1219 09:51:32.715250 31911 replica.cpp:712] Persisted action at 3 I1219 09:51:32.716145 31912 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I1219 09:51:32.716689 31912 leveldb.cpp:341] Persisting action (364 bytes) to leveldb took 512217ns I1219 09:51:32.716716 31912 replica.cpp:712] Persisted action at 3 I1219 09:51:32.716737 31912 replica.cpp:697] Replica learned APPEND action at position 3 I1219 09:51:32.718426 31911 registrar.cpp:484] Successfully updated the 'registry' in 0ns I1219 09:51:32.719441 31902 slave.cpp:3371] Received ping from slave-observer(228)@172.17.0.3:36408 I1219 09:51:32.719843 31909 log.cpp:702] Attempting to truncate the log to 3 I1219 09:51:32.719908 31911 master.cpp:4200] Registered slave bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 at slave(228)@172.17.0.3:36408 (60ab6e727501) with cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] I1219 09:51:32.720064 31911 slave.cpp:904] Registered with master master@172.17.0.3:36408; given slave ID bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 I1219 09:51:32.720088 31911 fetcher.cpp:81] Clearing fetcher cache I1219 09:51:32.720491 31911 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_BadACLDropCreateAndDestroy_gWLtnc/meta/slaves/bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0/slave.info' I1219 09:51:32.720844 31909 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I1219 09:51:32.720929 31911 slave.cpp:963] Forwarding total oversubscribed resources I1219 09:51:32.721017 31903 status_update_manager.cpp:181] Resuming sending status updates I1219 09:51:32.721099 31911 master.cpp:4542] Received update of slave bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 at slave(228)@172.17.0.3:36408 (60ab6e727501) with total oversubscribed resources I1219 09:51:32.721141 31905 hierarchical.cpp:465] Added slave bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 (60ab6e727501) with cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] (allocated: ) I1219 09:51:32.721879 31911 replica.cpp:537] Replica received write request for position 4 from (6759)@172.17.0.3:36408 I1219 09:51:32.722293 31905 hierarchical.cpp:1423] No inverse offers to send out! I1219 09:51:32.722337 31905 hierarchical.cpp:1101] Performed allocation for slave bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 in 1.155563ms I1219 09:51:32.722681 31905 hierarchical.cpp:521] Slave bded856d-1c7f-4fad-a8bc-3629ba8c59d3-S0 (60ab6e727501) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):2048) I1219 09:51:32.722713 31909 master.cpp:5252] Sending 1 offers to framework bded856d-...
Story points: 1