Project: mesos
Issue MESOS-4318: PersistentVolumeTest.BadACLNoPrincipal is flaky
https://builds.apache.org/job/Mesos/1457/COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=centos:7,label_exp=docker%7C%7CHadoop/consoleFull {noformat} [ RUN ] PersistentVolumeTest.BadACLNoPrincipal I0108 01:13:16.117883 1325 leveldb.cpp:174] Opened db in 2.614722ms I0108 01:13:16.118650 1325 leveldb.cpp:181] Compacted db in 706567ns I0108 01:13:16.118702 1325 leveldb.cpp:196] Created db iterator in 24489ns I0108 01:13:16.118723 1325 leveldb.cpp:202] Seeked to beginning of db in 2436ns I0108 01:13:16.118738 1325 leveldb.cpp:271] Iterated through 0 keys in the db in 397ns I0108 01:13:16.118793 1325 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0108 01:13:16.119627 1348 recover.cpp:447] Starting replica recovery I0108 01:13:16.120352 1348 recover.cpp:473] Replica is in EMPTY status I0108 01:13:16.121750 1357 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (7084)@172.17.0.2:32801 I0108 01:13:16.122297 1353 recover.cpp:193] Received a recover response from a replica in EMPTY status I0108 01:13:16.122747 1350 recover.cpp:564] Updating replica status to STARTING I0108 01:13:16.123625 1354 master.cpp:365] Master 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2 (d9632dd1c41e) started on 172.17.0.2:32801 I0108 01:13:16.123946 1347 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 728242ns I0108 01:13:16.123999 1347 replica.cpp:320] Persisted replica status to STARTING I0108 01:13:16.123708 1354 master.cpp:367] Flags at startup: --acls="create_volumes { principals { values: "test-principal" } volume_types { type: ANY } } create_volumes { principals { type: ANY } volume_types { type: NONE } } " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/f2rA75/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.27.0/_inst/share/mesos/webui" --work_dir="/tmp/f2rA75/master" --zk_session_timeout="10secs" I0108 01:13:16.124219 1354 master.cpp:414] Master allowing unauthenticated frameworks to register I0108 01:13:16.124236 1354 master.cpp:417] Master only allowing authenticated slaves to register I0108 01:13:16.124248 1354 credentials.hpp:35] Loading credentials for authentication from '/tmp/f2rA75/credentials' I0108 01:13:16.124294 1358 recover.cpp:473] Replica is in STARTING status I0108 01:13:16.124644 1354 master.cpp:456] Using default 'crammd5' authenticator I0108 01:13:16.124820 1354 master.cpp:493] Authorization enabled W0108 01:13:16.124843 1354 master.cpp:553] The '--roles' flag is deprecated. This flag will be removed in the future. See the Mesos 0.27 upgrade notes for more information I0108 01:13:16.125154 1348 hierarchical.cpp:147] Initialized hierarchical allocator process I0108 01:13:16.125334 1345 whitelist_watcher.cpp:77] No whitelist given I0108 01:13:16.126065 1346 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (7085)@172.17.0.2:32801 I0108 01:13:16.126806 1348 recover.cpp:193] Received a recover response from a replica in STARTING status I0108 01:13:16.128237 1354 recover.cpp:564] Updating replica status to VOTING I0108 01:13:16.128402 1359 master.cpp:1629] The newly elected leader is master@172.17.0.2:32801 with id 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2 I0108 01:13:16.128489 1359 master.cpp:1642] Elected as the leading master! I0108 01:13:16.128523 1359 master.cpp:1387] Recovering from registrar I0108 01:13:16.128756 1355 registrar.cpp:307] Recovering registrar I0108 01:13:16.129259 1344 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 531437ns I0108 01:13:16.129292 1344 replica.cpp:320] Persisted replica status to VOTING I0108 01:13:16.129425 1358 recover.cpp:578] Successfully joined the Paxos group I0108 01:13:16.129680 1358 recover.cpp:462] Recover process terminated I0108 01:13:16.130187 1358 log.cpp:659] Attempting to start the writer I0108 01:13:16.131613 1352 replica.cpp:493] Replica received implicit promise request from (7086)@172.17.0.2:32801 with proposal 1 I0108 01:13:16.131983 1352 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 333646ns I0108 01:13:16.132004 1352 replica.cpp:342] Persisted promised to 1 I0108 01:13:16.132627 1348 coordinator.cpp:238] Coordinator attempting to fill missing positions I0108 01:13:16.133896 1349 replica.cpp:388] Replica received explicit promise request from (7087)@172.17.0.2:32801 for position 0 with proposal 2 I0108 01:13:16.134289 1349 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 349652ns I0108 01:13:16.134317 1349 replica.cpp:712] Persisted action at 0 I0108 01:13:16.135470 1351 replica.cpp:537] Replica received write request for position 0 from (7088)@172.17.0.2:32801 I0108 01:13:16.135537 1351 leveldb.cpp:436] Reading position from leveldb took 36181ns I0108 01:13:16.135901 1351 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 308752ns I0108 01:13:16.135924 1351 replica.cpp:712] Persisted action at 0 I0108 01:13:16.136529 1347 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0108 01:13:16.136889 1347 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 327106ns I0108 01:13:16.136916 1347 replica.cpp:712] Persisted action at 0 I0108 01:13:16.136943 1347 replica.cpp:697] Replica learned NOP action at position 0 I0108 01:13:16.137707 1359 log.cpp:675] Writer started with ending position 0 I0108 01:13:16.138844 1348 leveldb.cpp:436] Reading position from leveldb took 31371ns I0108 01:13:16.139878 1356 registrar.cpp:340] Successfully fetched the registry (0B) in 0ns I0108 01:13:16.140012 1356 registrar.cpp:439] Applied 1 operations in 42063ns; attempting to update the 'registry' I0108 01:13:16.140797 1355 log.cpp:683] Attempting to append 170 bytes to the log I0108 01:13:16.140974 1345 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0108 01:13:16.141744 1354 replica.cpp:537] Replica received write request for position 1 from (7089)@172.17.0.2:32801 I0108 01:13:16.142226 1354 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 441971ns I0108 01:13:16.142251 1354 replica.cpp:712] Persisted action at 1 I0108 01:13:16.142860 1351 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0108 01:13:16.143198 1351 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 305928ns I0108 01:13:16.143223 1351 replica.cpp:712] Persisted action at 1 I0108 01:13:16.143241 1351 replica.cpp:697] Replica learned APPEND action at position 1 I0108 01:13:16.144271 1354 registrar.cpp:484] Successfully updated the 'registry' in 0ns I0108 01:13:16.144435 1354 registrar.cpp:370] Successfully recovered registrar I0108 01:13:16.144567 1359 log.cpp:702] Attempting to truncate the log to 1 I0108 01:13:16.144780 1359 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0108 01:13:16.144989 1348 hierarchical.cpp:165] Skipping recovery of hierarchical allocator: nothing to recover I0108 01:13:16.144928 1354 master.cpp:1439] Recovered 0 slaves from the Registry (131B) ; allowing 10mins for slaves to re-register I0108 01:13:16.145690 1357 replica.cpp:537] Replica received write request for position 2 from (7090)@172.17.0.2:32801 I0108 01:13:16.146072 1357 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 345113ns I0108 01:13:16.146097 1357 replica.cpp:712] Persisted action at 2 I0108 01:13:16.146667 1358 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0108 01:13:16.147060 1358 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 283648ns I0108 01:13:16.147116 1358 leveldb.cpp:399] Deleting ~1 keys from leveldb took 32174ns I0108 01:13:16.147135 1358 replica.cpp:712] Persisted action at 2 I0108 01:13:16.147153 1358 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0108 01:13:16.166832 1325 containerizer.cpp:139] Using isolation: posix/cpu,posix/mem,filesystem/posix W0108 01:13:16.167556 1325 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I0108 01:13:16.170526 1349 slave.cpp:191] Slave started on 231)@172.17.0.2:32801 I0108 01:13:16.170718 1349 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="https://auth.docker.io" --docker_kill_orphans="true" --docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.27.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):2048" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY" I0108 01:13:16.171269 1349 credentials.hpp:83] Loading credential for authentication from '/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY/credential' I0108 01:13:16.171505 1349 slave.cpp:322] Slave using credential for: test-principal I0108 01:13:16.171747 1349 resources.cpp:481] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):2048 Trying semicolon-delimited string format instead I0108 01:13:16.172266 1349 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] I0108 01:13:16.172327 1349 slave.cpp:400] Slave attributes: [ ] I0108 01:13:16.172340 1349 slave.cpp:405] Slave hostname: d9632dd1c41e I0108 01:13:16.172353 1349 slave.cpp:410] Slave checkpoint: true I0108 01:13:16.173418 1353 state.cpp:58] Recovering state from '/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY/meta' I0108 01:13:16.173521 1325 sched.cpp:164] Version: 0.27.0 I0108 01:13:16.174054 1345 status_update_manager.cpp:200] Recovering status update manager I0108 01:13:16.174289 1353 containerizer.cpp:387] Recovering containerizer I0108 01:13:16.174295 1356 sched.cpp:268] New master detected at master@172.17.0.2:32801 I0108 01:13:16.174387 1356 sched.cpp:278] No credentials provided. Attempting to register without authentication I0108 01:13:16.174409 1356 sched.cpp:722] Sending SUBSCRIBE call to master@172.17.0.2:32801 I0108 01:13:16.174515 1356 sched.cpp:755] Will retry registration in 1.699889272secs if necessary I0108 01:13:16.174653 1349 master.cpp:2197] Received SUBSCRIBE call for framework 'no-principal' at scheduler-bf0ed267-b4c4-412d-9fb0-84c85cd2fbce@172.17.0.2:32801 I0108 01:13:16.174823 1349 master.cpp:1668] Authorizing framework principal '' to receive offers for role 'role1' I0108 01:13:16.175250 1347 master.cpp:2268] Subscribing framework no-principal with checkpointing disabled and capabilities [ ] I0108 01:13:16.175359 1353 slave.cpp:4429] Finished recovery I0108 01:13:16.175715 1345 hierarchical.cpp:260] Added framework 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-0000 I0108 01:13:16.175734 1351 sched.cpp:649] Framework registered with 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-0000 I0108 01:13:16.175792 1345 hierarchical.cpp:1329] No resources available to allocate! I0108 01:13:16.175833 1345 hierarchical.cpp:1423] No inverse offers to send out! I0108 01:13:16.175853 1353 slave.cpp:4601] Querying resource estimator for oversubscribable resources I0108 01:13:16.175869 1345 hierarchical.cpp:1079] Performed allocation for 0 slaves in 127881ns I0108 01:13:16.175923 1351 sched.cpp:663] Scheduler::registered took 27956ns I0108 01:13:16.176110 1353 slave.cpp:729] New master detected at master@172.17.0.2:32801 I0108 01:13:16.176187 1353 slave.cpp:792] Authenticating with master master@172.17.0.2:32801 I0108 01:13:16.176216 1353 slave.cpp:797] Using default CRAM-MD5 authenticatee I0108 01:13:16.176398 1357 status_update_manager.cpp:174] Pausing sending status updates I0108 01:13:16.176404 1353 slave.cpp:765] Detecting new master I0108 01:13:16.176463 1358 authenticatee.cpp:121] Creating new client SASL connection I0108 01:13:16.176553 1353 slave.cpp:4615] Received oversubscribable resources from the resource estimator I0108 01:13:16.176709 1353 master.cpp:5445] Authenticating slave(231)@172.17.0.2:32801 I0108 01:13:16.176823 1359 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(516)@172.17.0.2:32801 I0108 01:13:16.177135 1348 authenticator.cpp:98] Creating new server SASL connection I0108 01:13:16.177373 1356 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0108 01:13:16.177399 1356 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0108 01:13:16.177502 1344 authenticator.cpp:203] Received SASL authentication start I0108 01:13:16.177563 1344 authenticator.cpp:325] Authentication requires more steps I0108 01:13:16.177680 1346 authenticatee.cpp:258] Received SASL authentication step I0108 01:13:16.177848 1354 authenticator.cpp:231] Received SASL authentication step I0108 01:13:16.177883 1354 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'd9632dd1c41e' server FQDN: 'd9632dd1c41e' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0108 01:13:16.177894 1354 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0108 01:13:16.177944 1354 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0108 01:13:16.177994 1354 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'd9632dd1c41e' server FQDN: 'd9632dd1c41e' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0108 01:13:16.178014 1354 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0108 01:13:16.178040 1354 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0108 01:13:16.178066 1354 authenticator.cpp:317] Authentication success I0108 01:13:16.178256 1355 authenticatee.cpp:298] Authentication success I0108 01:13:16.178315 1354 master.cpp:5475] Successfully authenticated principal 'test-principal' at slave(231)@172.17.0.2:32801 I0108 01:13:16.178356 1355 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(516)@172.17.0.2:32801 I0108 01:13:16.178710 1354 slave.cpp:860] Successfully authenticated with master master@172.17.0.2:32801 I0108 01:13:16.178865 1354 slave.cpp:1254] Will retry registration in 13.009431ms if necessary I0108 01:13:16.179138 1350 master.cpp:4154] Registering slave at slave(231)@172.17.0.2:32801 (d9632dd1c41e) with id 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 I0108 01:13:16.179628 1345 registrar.cpp:439] Applied 1 operations in 71663ns; attempting to update the 'registry' I0108 01:13:16.180505 1356 log.cpp:683] Attempting to append 343 bytes to the log I0108 01:13:16.180711 1352 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0108 01:13:16.181499 1350 replica.cpp:537] Replica received write request for position 3 from (7103)@172.17.0.2:32801 I0108 01:13:16.182080 1350 leveldb.cpp:341] Persisting action (362 bytes) to leveldb took 537757ns I0108 01:13:16.182112 1350 replica.cpp:712] Persisted action at 3 I0108 01:13:16.182749 1351 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0108 01:13:16.183120 1351 leveldb.cpp:341] Persisting action (364 bytes) to leveldb took 340999ns I0108 01:13:16.183151 1351 replica.cpp:712] Persisted action at 3 I0108 01:13:16.183177 1351 replica.cpp:697] Replica learned APPEND action at position 3 I0108 01:13:16.184787 1348 registrar.cpp:484] Successfully updated the 'registry' in 0ns I0108 01:13:16.185287 1348 log.cpp:702] Attempting to truncate the log to 3 I0108 01:13:16.185484 1349 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0108 01:13:16.186043 1353 slave.cpp:3371] Received ping from slave-observer(230)@172.17.0.2:32801 I0108 01:13:16.186074 1345 master.cpp:4222] Registered slave 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 at slave(231)@172.17.0.2:32801 (d9632dd1c41e) with cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] I0108 01:13:16.186224 1353 slave.cpp:904] Registered with master master@172.17.0.2:32801; given slave ID 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 I0108 01:13:16.186441 1353 fetcher.cpp:81] Clearing fetcher cache I0108 01:13:16.186486 1349 hierarchical.cpp:465] Added slave 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 (d9632dd1c41e) with cpus(*):2; mem(*):1024; disk(role1):2048; ports(*):[31000-32000] (allocated: ) I0108 01:13:16.186658 1346 status_update_manager.cpp:181] Resuming sending status updates I0108 01:13:16.186885 1353 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_BadACLNoPrincipal_yqJjLY/meta/slaves/773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0/slave.info' I0108 01:13:16.186905 1350 replica.cpp:537] Replica received write request for position 4 from (7104)@172.17.0.2:32801 I0108 01:13:16.187595 1350 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 645704ns I0108 01:13:16.187628 1350 replica.cpp:712] Persisted action at 4 I0108 01:13:16.188347 1349 hierarchical.cpp:1423] No inverse offers to send out! I0108 01:13:16.188475 1349 hierarchical.cpp:1101] Performed allocation for slave 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 in 1.861833ms I0108 01:13:16.188560 1348 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0108 01:13:16.188385 1353 slave.cpp:963] Forwarding total oversubscribed resources I0108 01:13:16.189275 1344 master.cpp:5274] Sending 1 offers to framework 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-0000 (no-principal) at scheduler-bf0ed267-b4c4-412d-9fb0-84c85cd2fbce@172.17.0.2:32801 I0108 01:13:16.189792 1344 master.cpp:4564] Received update of slave 773d31e8-383d-4e4b-aa68-f9a3fb9f1fc2-S0 at slave(231)@1...
Story points: 1