Project: mesos
Issue MESOS-4542: MasterQuotaTest.AvailableResourcesAfterRescinding is flaky.
Can be reproduced by running {{GLOG_v=1 GTEST_FILTER="MasterQuotaTest.AvailableResourcesAfterRescinding" ./bin/mesos-tests.sh --gtest_shuffle --gtest_break_on_failure --gtest_repeat=1000 --verbose}}. h5. Verbose log from a bad run: {code} [ RUN ] MasterQuotaTest.AvailableResourcesAfterRescinding I0128 12:20:27.568657 2080858880 resources.cpp:564] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0128 12:20:27.570142 2080858880 resources.cpp:564] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0128 12:20:27.583225 2080858880 leveldb.cpp:174] Opened db in 6241us I0128 12:20:27.584353 2080858880 leveldb.cpp:181] Compacted db in 1026us I0128 12:20:27.584429 2080858880 leveldb.cpp:196] Created db iterator in 12us I0128 12:20:27.584442 2080858880 leveldb.cpp:202] Seeked to beginning of db in 7us I0128 12:20:27.584453 2080858880 leveldb.cpp:271] Iterated through 0 keys in the db in 6us I0128 12:20:27.584475 2080858880 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0128 12:20:27.584918 300445696 recover.cpp:447] Starting replica recovery I0128 12:20:27.585113 300445696 recover.cpp:473] Replica is in EMPTY status I0128 12:20:27.585916 297226240 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (18274)@192.168.178.24:51278 I0128 12:20:27.586086 297762816 recover.cpp:193] Received a recover response from a replica in EMPTY status I0128 12:20:27.586449 297226240 recover.cpp:564] Updating replica status to STARTING I0128 12:20:27.587204 300445696 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 624us I0128 12:20:27.587242 300445696 replica.cpp:320] Persisted replica status to STARTING I0128 12:20:27.587376 299372544 recover.cpp:473] Replica is in STARTING status I0128 12:20:27.588050 300982272 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (18275)@192.168.178.24:51278 I0128 12:20:27.588235 300445696 recover.cpp:193] Received a recover response from a replica in STARTING status I0128 12:20:27.588572 297762816 recover.cpp:564] Updating replica status to VOTING I0128 12:20:27.588850 297226240 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 140us I0128 12:20:27.588879 297226240 replica.cpp:320] Persisted replica status to VOTING I0128 12:20:27.588975 299909120 recover.cpp:578] Successfully joined the Paxos group I0128 12:20:27.589154 299909120 recover.cpp:462] Recover process terminated I0128 12:20:27.599486 298835968 master.cpp:374] Master 531344bd-56f4-4e4f-8f6f-a6a9d36058c7 (alexr.fritz.box) started on 192.168.178.24:51278 I0128 12:20:27.599520 298835968 master.cpp:376] Flags at startup: --acls="" --allocation_interval="50ms" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/private/tmp/NlzPSo/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1,role2" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/private/tmp/NlzPSo/master" --zk_session_timeout="10secs" I0128 12:20:27.599753 298835968 master.cpp:421] Master only allowing authenticated frameworks to register I0128 12:20:27.599769 298835968 master.cpp:426] Master only allowing authenticated slaves to register I0128 12:20:27.599781 298835968 credentials.hpp:35] Loading credentials for authentication from '/private/tmp/NlzPSo/credentials' I0128 12:20:27.600082 298835968 master.cpp:466] Using default 'crammd5' authenticator I0128 12:20:27.600163 298835968 master.cpp:535] Using default 'basic' HTTP authenticator I0128 12:20:27.600327 298835968 master.cpp:569] Authorization enabled W0128 12:20:27.600345 298835968 master.cpp:629] The '--roles' flag is deprecated. This flag will be removed in the future. See the Mesos 0.27 upgrade notes for more information I0128 12:20:27.600497 297762816 whitelist_watcher.cpp:77] No whitelist given I0128 12:20:27.600503 297226240 hierarchical.cpp:144] Initialized hierarchical allocator process I0128 12:20:27.601965 297226240 master.cpp:1710] The newly elected leader is master@192.168.178.24:51278 with id 531344bd-56f4-4e4f-8f6f-a6a9d36058c7 I0128 12:20:27.601995 297226240 master.cpp:1723] Elected as the leading master! I0128 12:20:27.602007 297226240 master.cpp:1468] Recovering from registrar I0128 12:20:27.602083 300445696 registrar.cpp:307] Recovering registrar I0128 12:20:27.602460 297226240 log.cpp:659] Attempting to start the writer I0128 12:20:27.603514 299909120 replica.cpp:493] Replica received implicit promise request from (18277)@192.168.178.24:51278 with proposal 1 I0128 12:20:27.603734 299909120 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 205us I0128 12:20:27.603768 299909120 replica.cpp:342] Persisted promised to 1 I0128 12:20:27.604194 299909120 coordinator.cpp:238] Coordinator attempting to fill missing positions I0128 12:20:27.605311 299372544 replica.cpp:388] Replica received explicit promise request from (18278)@192.168.178.24:51278 for position 0 with proposal 2 I0128 12:20:27.605468 299372544 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 133us I0128 12:20:27.605494 299372544 replica.cpp:712] Persisted action at 0 I0128 12:20:27.606441 298835968 replica.cpp:537] Replica received write request for position 0 from (18279)@192.168.178.24:51278 I0128 12:20:27.606492 298835968 leveldb.cpp:436] Reading position from leveldb took 29us I0128 12:20:27.606665 298835968 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 151us I0128 12:20:27.606688 298835968 replica.cpp:712] Persisted action at 0 I0128 12:20:27.607244 297226240 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0128 12:20:27.607409 297226240 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 152us I0128 12:20:27.607441 297226240 replica.cpp:712] Persisted action at 0 I0128 12:20:27.607457 297226240 replica.cpp:697] Replica learned NOP action at position 0 I0128 12:20:27.607853 297226240 log.cpp:675] Writer started with ending position 0 I0128 12:20:27.608649 299372544 leveldb.cpp:436] Reading position from leveldb took 158us I0128 12:20:27.609539 298835968 registrar.cpp:340] Successfully fetched the registry (0B) in 7.426816ms I0128 12:20:27.609763 298835968 registrar.cpp:439] Applied 1 operations in 54us; attempting to update the 'registry' I0128 12:20:27.610216 300982272 log.cpp:683] Attempting to append 186 bytes to the log I0128 12:20:27.610297 298835968 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0128 12:20:27.611016 299909120 replica.cpp:537] Replica received write request for position 1 from (18280)@192.168.178.24:51278 I0128 12:20:27.611188 299909120 leveldb.cpp:341] Persisting action (205 bytes) to leveldb took 153us I0128 12:20:27.611222 299909120 replica.cpp:712] Persisted action at 1 I0128 12:20:27.611843 299909120 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0128 12:20:27.612004 299909120 leveldb.cpp:341] Persisting action (207 bytes) to leveldb took 147us I0128 12:20:27.612035 299909120 replica.cpp:712] Persisted action at 1 I0128 12:20:27.612052 299909120 replica.cpp:697] Replica learned APPEND action at position 1 I0128 12:20:27.612742 300982272 registrar.cpp:484] Successfully updated the 'registry' in 2.924032ms I0128 12:20:27.612846 300982272 registrar.cpp:370] Successfully recovered registrar I0128 12:20:27.612936 298835968 log.cpp:702] Attempting to truncate the log to 1 I0128 12:20:27.613005 297762816 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0128 12:20:27.613323 298299392 master.cpp:1520] Recovered 0 slaves from the Registry (147B) ; allowing 10mins for slaves to re-register I0128 12:20:27.613364 298835968 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover I0128 12:20:27.613966 300445696 replica.cpp:537] Replica received write request for position 2 from (18281)@192.168.178.24:51278 I0128 12:20:27.614131 300445696 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 151us I0128 12:20:27.614166 300445696 replica.cpp:712] Persisted action at 2 I0128 12:20:27.614660 299372544 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0128 12:20:27.614828 299372544 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 158us I0128 12:20:27.614876 299372544 leveldb.cpp:399] Deleting ~1 keys from leveldb took 28us I0128 12:20:27.614898 299372544 replica.cpp:712] Persisted action at 2 I0128 12:20:27.614915 299372544 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0128 12:20:27.625591 2080858880 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix I0128 12:20:27.629758 298299392 slave.cpp:192] Slave started on 871)@192.168.178.24:51278 I0128 12:20:27.629791 298299392 slave.cpp:193] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="https://auth.docker.io" --docker_kill_orphans="true" --docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/alex/Projects/mesos/build/default/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf" I0128 12:20:27.630067 298299392 credentials.hpp:83] Loading credential for authentication from '/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf/credential' I0128 12:20:27.630223 298299392 slave.cpp:323] Slave using credential for: test-principal I0128 12:20:27.630360 298299392 resources.cpp:564] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0128 12:20:27.630818 298299392 slave.cpp:463] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0128 12:20:27.630869 298299392 slave.cpp:471] Slave attributes: [ ] I0128 12:20:27.630882 298299392 slave.cpp:476] Slave hostname: alexr.fritz.box I0128 12:20:27.631352 300982272 state.cpp:58] Recovering state from '/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf/meta' I0128 12:20:27.631515 299909120 status_update_manager.cpp:200] Recovering status update manager I0128 12:20:27.631702 298835968 containerizer.cpp:390] Recovering containerizer I0128 12:20:27.632589 297226240 provisioner.cpp:245] Provisioner recovery complete I0128 12:20:27.632807 298835968 slave.cpp:4495] Finished recovery I0128 12:20:27.633539 298835968 slave.cpp:4667] Querying resource estimator for oversubscribable resources I0128 12:20:27.633752 300445696 status_update_manager.cpp:174] Pausing sending status updates I0128 12:20:27.633754 298835968 slave.cpp:795] New master detected at master@192.168.178.24:51278 I0128 12:20:27.633806 298835968 slave.cpp:858] Authenticating with master master@192.168.178.24:51278 I0128 12:20:27.633824 298835968 slave.cpp:863] Using default CRAM-MD5 authenticatee I0128 12:20:27.633903 298835968 slave.cpp:831] Detecting new master I0128 12:20:27.633913 299372544 authenticatee.cpp:121] Creating new client SASL connection I0128 12:20:27.634016 298835968 slave.cpp:4681] Received oversubscribable resources from the resource estimator I0128 12:20:27.634076 297226240 master.cpp:5521] Authenticating slave(871)@192.168.178.24:51278 I0128 12:20:27.634130 299372544 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(1741)@192.168.178.24:51278 I0128 12:20:27.634255 297226240 authenticator.cpp:98] Creating new server SASL connection I0128 12:20:27.634348 300982272 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0128 12:20:27.634367 300982272 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0128 12:20:27.634454 298835968 authenticator.cpp:203] Received SASL authentication start I0128 12:20:27.634515 298835968 authenticator.cpp:325] Authentication requires more steps I0128 12:20:27.634572 298835968 authenticatee.cpp:258] Received SASL authentication step I0128 12:20:27.634706 297226240 authenticator.cpp:231] Received SASL authentication step I0128 12:20:27.634757 297226240 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'alexr.fritz.box' server FQDN: 'alexr.fritz.box' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0128 12:20:27.634771 297226240 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0128 12:20:27.634793 297226240 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0128 12:20:27.634809 297226240 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'alexr.fritz.box' server FQDN: 'alexr.fritz.box' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0128 12:20:27.634819 297226240 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0128 12:20:27.634827 297226240 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0128 12:20:27.634893 297226240 authenticator.cpp:317] Authentication success I0128 12:20:27.634958 298835968 authenticatee.cpp:298] Authentication success I0128 12:20:27.635030 298299392 master.cpp:5551] Successfully authenticated principal 'test-principal' at slave(871)@192.168.178.24:51278 I0128 12:20:27.635079 300445696 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(1741)@192.168.178.24:51278 I0128 12:20:27.635195 299372544 slave.cpp:926] Successfully authenticated with master master@192.168.178.24:51278 I0128 12:20:27.635273 299372544 slave.cpp:1320] Will retry registration in 5.823453ms if necessary I0128 12:20:27.635365 299909120 master.cpp:4235] Registering slave at slave(871)@192.168.178.24:51278 (alexr.fritz.box) with id 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 I0128 12:20:27.635542 297762816 registrar.cpp:439] Applied 1 operations in 41us; attempting to update the 'registry' I0128 12:20:27.635889 299372544 log.cpp:683] Attempting to append 358 bytes to the log I0128 12:20:27.636011 298299392 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0128 12:20:27.636693 300982272 replica.cpp:537] Replica received write request for position 3 from (18295)@192.168.178.24:51278 I0128 12:20:27.636860 300982272 leveldb.cpp:341] Persisting action (377 bytes) to leveldb took 139us I0128 12:20:27.636885 300982272 replica.cpp:712] Persisted action at 3 I0128 12:20:27.637380 299909120 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0128 12:20:27.637547 299909120 leveldb.cpp:341] Persisting action (379 bytes) to leveldb took 132us I0128 12:20:27.637573 299909120 replica.cpp:712] Persisted action at 3 I0128 12:20:27.637589 299909120 replica.cpp:697] Replica learned APPEND action at position 3 I0128 12:20:27.638362 298835968 registrar.cpp:484] Successfully updated the 'registry' in 2.77504ms I0128 12:20:27.638589 300445696 log.cpp:702] Attempting to truncate the log to 3 I0128 12:20:27.638684 298299392 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0128 12:20:27.638825 300445696 slave.cpp:3435] Received ping from slave-observer(871)@192.168.178.24:51278 I0128 12:20:27.639081 300982272 hierarchical.cpp:473] Added slave 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 (alexr.fritz.box) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0128 12:20:27.639117 299909120 master.cpp:4303] Registered slave 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 at slave(871)@192.168.178.24:51278 (alexr.fritz.box) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0128 12:20:27.639165 300982272 hierarchical.cpp:1403] No resources available to allocate! I0128 12:20:27.639168 297226240 slave.cpp:970] Registered with master master@192.168.178.24:51278; given slave ID 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 I0128 12:20:27.639189 297226240 fetcher.cpp:81] Clearing fetcher cache I0128 12:20:27.639183 300982272 hierarchical.cpp:1116] Performed allocation for slave 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 in 77us I0128 12:20:27.639348 297762816 status_update_manager.cpp:181] Resuming sending status updates I0128 12:20:27.639519 298835968 replica.cpp:537] Replica received write request for position 4 from (18296)@192.168.178.24:51278 I0128 12:20:27.639678 298835968 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 142us I0128 12:20:27.639708 298835968 replica.cpp:712] Persisted action at 4 I0128 12:20:27.640115 300982272 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0128 12:20:27.640276 300982272 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 137us I0128 12:20:27.640312 300982272 leveldb.cpp:399] Deleting ~2 keys from leveldb took 21us I0128 12:20:27.640326 300982272 replica.cpp:712] Persisted action at 4 I0128 12:20:27.640336 300982272 replica.cpp:697] Replica learned TRUNCATE action at position 4 I0128 12:20:27.642145 297226240 slave.cpp:993] Checkpointing SlaveInfo to '/tmp/MasterQuotaTest_AvailableResourcesAfterRescinding_gS9Qcf/meta/slaves/531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0/slave.info' I0128 12:20:27.643354 297226240 slave.cpp:1029] Forwarding total oversubscribed resources I0128 12:20:27.643458 300445696 master.cpp:4644] Received update of slave 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 at slave(871)@192.168.178.24:51278 (alexr.fritz.box) with total oversubscribed resources I0128 12:20:27.643710 298299392 hierarchical.cpp:531] Slave 531344bd-56f4-4e4f-8f6f-a6a9d36058c7-S0 (alexr.fritz.box) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) I0128 12:20:27.643769 298299392 hierarchical.cpp:1403] No resources available to allocate! I0128 12:20:27.643805 2...
Story points: 3