Project: mesos
Issue MESOS-4768: MasterMaintenanceTest.InverseOffers is flaky
[MESOS-4169] significantly sped up this test, but also surfaced some more flakiness. This can be fixed in the same way as [MESOS-4059]. Verbose logs from ASF Centos7 build: {code} [ RUN ] MasterMaintenanceTest.InverseOffers I0224 22:35:53.714018 1948 leveldb.cpp:174] Opened db in 2.034387ms I0224 22:35:53.714663 1948 leveldb.cpp:181] Compacted db in 608839ns I0224 22:35:53.714709 1948 leveldb.cpp:196] Created db iterator in 19043ns I0224 22:35:53.714844 1948 leveldb.cpp:202] Seeked to beginning of db in 2330ns I0224 22:35:53.714956 1948 leveldb.cpp:271] Iterated through 0 keys in the db in 518ns I0224 22:35:53.715092 1948 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0224 22:35:53.715646 1968 recover.cpp:447] Starting replica recovery I0224 22:35:53.715915 1981 recover.cpp:473] Replica is in EMPTY status I0224 22:35:53.717067 1972 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (4533)@172.17.0.1:36678 I0224 22:35:53.717445 1981 recover.cpp:193] Received a recover response from a replica in EMPTY status I0224 22:35:53.717888 1978 recover.cpp:564] Updating replica status to STARTING I0224 22:35:53.718585 1979 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 525061ns I0224 22:35:53.718618 1979 replica.cpp:320] Persisted replica status to STARTING I0224 22:35:53.718827 1982 recover.cpp:473] Replica is in STARTING status I0224 22:35:53.719728 1969 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (4534)@172.17.0.1:36678 I0224 22:35:53.719974 1971 recover.cpp:193] Received a recover response from a replica in STARTING status I0224 22:35:53.720369 1970 recover.cpp:564] Updating replica status to VOTING I0224 22:35:53.720789 1982 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 322308ns I0224 22:35:53.720823 1982 replica.cpp:320] Persisted replica status to VOTING I0224 22:35:53.720968 1982 recover.cpp:578] Successfully joined the Paxos group I0224 22:35:53.721101 1982 recover.cpp:462] Recover process terminated I0224 22:35:53.721698 1982 master.cpp:376] Master aab18b61-7811-4c43-a672-d1a63818c880 (4db5fa128d2d) started on 172.17.0.1:36678 I0224 22:35:53.721719 1982 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/MjbcWP/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.28.0/_inst/share/mesos/webui" --work_dir="/tmp/MjbcWP/master" --zk_session_timeout="10secs" I0224 22:35:53.722039 1982 master.cpp:425] Master allowing unauthenticated frameworks to register I0224 22:35:53.722053 1982 master.cpp:428] Master only allowing authenticated slaves to register I0224 22:35:53.722061 1982 credentials.hpp:35] Loading credentials for authentication from '/tmp/MjbcWP/credentials' I0224 22:35:53.722394 1982 master.cpp:468] Using default 'crammd5' authenticator I0224 22:35:53.722525 1982 master.cpp:537] Using default 'basic' HTTP authenticator I0224 22:35:53.722661 1982 master.cpp:571] Authorization enabled I0224 22:35:53.722813 1968 hierarchical.cpp:144] Initialized hierarchical allocator process I0224 22:35:53.722846 1980 whitelist_watcher.cpp:77] No whitelist given I0224 22:35:53.724957 1977 master.cpp:1712] The newly elected leader is master@172.17.0.1:36678 with id aab18b61-7811-4c43-a672-d1a63818c880 I0224 22:35:53.725000 1977 master.cpp:1725] Elected as the leading master! I0224 22:35:53.725023 1977 master.cpp:1470] Recovering from registrar I0224 22:35:53.725306 1967 registrar.cpp:307] Recovering registrar I0224 22:35:53.725808 1977 log.cpp:659] Attempting to start the writer I0224 22:35:53.727145 1973 replica.cpp:493] Replica received implicit promise request from (4536)@172.17.0.1:36678 with proposal 1 I0224 22:35:53.727728 1973 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 424560ns I0224 22:35:53.727828 1973 replica.cpp:342] Persisted promised to 1 I0224 22:35:53.729080 1973 coordinator.cpp:238] Coordinator attempting to fill missing positions I0224 22:35:53.731009 1979 replica.cpp:388] Replica received explicit promise request from (4537)@172.17.0.1:36678 for position 0 with proposal 2 I0224 22:35:53.731580 1979 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 478479ns I0224 22:35:53.731613 1979 replica.cpp:712] Persisted action at 0 I0224 22:35:53.734354 1979 replica.cpp:537] Replica received write request for position 0 from (4538)@172.17.0.1:36678 I0224 22:35:53.734485 1979 leveldb.cpp:436] Reading position from leveldb took 60879ns I0224 22:35:53.735877 1979 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 1.324061ms I0224 22:35:53.735930 1979 replica.cpp:712] Persisted action at 0 I0224 22:35:53.737061 1970 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0224 22:35:53.738881 1970 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.772814ms I0224 22:35:53.738939 1970 replica.cpp:712] Persisted action at 0 I0224 22:35:53.738975 1970 replica.cpp:697] Replica learned NOP action at position 0 I0224 22:35:53.740136 1976 log.cpp:675] Writer started with ending position 0 I0224 22:35:53.741750 1976 leveldb.cpp:436] Reading position from leveldb took 74863ns I0224 22:35:53.743479 1976 registrar.cpp:340] Successfully fetched the registry (0B) in 18.11968ms I0224 22:35:53.743755 1976 registrar.cpp:439] Applied 1 operations in 56670ns; attempting to update the 'registry' I0224 22:35:53.745604 1978 log.cpp:683] Attempting to append 170 bytes to the log I0224 22:35:53.745905 1977 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0224 22:35:53.746968 1981 replica.cpp:537] Replica received write request for position 1 from (4539)@172.17.0.1:36678 I0224 22:35:53.747480 1981 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 456947ns I0224 22:35:53.747609 1981 replica.cpp:712] Persisted action at 1 I0224 22:35:53.750448 1981 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0224 22:35:53.751158 1981 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 535163ns I0224 22:35:53.751258 1981 replica.cpp:712] Persisted action at 1 I0224 22:35:53.751389 1981 replica.cpp:697] Replica learned APPEND action at position 1 I0224 22:35:53.753149 1979 registrar.cpp:484] Successfully updated the 'registry' in 9.228032ms I0224 22:35:53.753324 1979 registrar.cpp:370] Successfully recovered registrar I0224 22:35:53.753593 1979 log.cpp:702] Attempting to truncate the log to 1 I0224 22:35:53.753805 1979 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0224 22:35:53.754055 1981 master.cpp:1522] Recovered 0 slaves from the Registry (131B) ; allowing 10mins for slaves to re-register I0224 22:35:53.754349 1979 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover I0224 22:35:53.755764 1977 replica.cpp:537] Replica received write request for position 2 from (4540)@172.17.0.1:36678 I0224 22:35:53.756459 1977 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 488559ns I0224 22:35:53.756561 1977 replica.cpp:712] Persisted action at 2 I0224 22:35:53.757932 1972 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0224 22:35:53.758400 1972 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 343827ns I0224 22:35:53.758539 1972 leveldb.cpp:399] Deleting ~1 keys from leveldb took 34231ns I0224 22:35:53.758658 1972 replica.cpp:712] Persisted action at 2 I0224 22:35:53.758782 1972 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0224 22:35:53.778059 1978 slave.cpp:193] Slave started on 115)@172.17.0.1:36678 I0224 22:35:53.778105 1978 slave.cpp:194] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="maintenance-host" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.28.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF" I0224 22:35:53.778609 1978 credentials.hpp:83] Loading credential for authentication from '/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF/credential' I0224 22:35:53.779175 1978 slave.cpp:324] Slave using credential for: test-principal I0224 22:35:53.779520 1978 resources.cpp:576] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0224 22:35:53.780192 1978 slave.cpp:464] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0224 22:35:53.780362 1978 slave.cpp:472] Slave attributes: [ ] I0224 22:35:53.780483 1978 slave.cpp:477] Slave hostname: maintenance-host I0224 22:35:53.782126 1967 state.cpp:58] Recovering state from '/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF/meta' I0224 22:35:53.782892 1969 status_update_manager.cpp:200] Recovering status update manager I0224 22:35:53.783242 1969 slave.cpp:4565] Finished recovery I0224 22:35:53.784001 1969 slave.cpp:4737] Querying resource estimator for oversubscribable resources I0224 22:35:53.784678 1969 slave.cpp:796] New master detected at master@172.17.0.1:36678 I0224 22:35:53.784874 1967 status_update_manager.cpp:174] Pausing sending status updates I0224 22:35:53.784808 1969 slave.cpp:859] Authenticating with master master@172.17.0.1:36678 I0224 22:35:53.784945 1969 slave.cpp:864] Using default CRAM-MD5 authenticatee I0224 22:35:53.785181 1969 slave.cpp:832] Detecting new master I0224 22:35:53.785326 1969 slave.cpp:4751] Received oversubscribable resources from the resource estimator I0224 22:35:53.785557 1969 authenticatee.cpp:121] Creating new client SASL connection I0224 22:35:53.786227 1969 master.cpp:5526] Authenticating slave(115)@172.17.0.1:36678 I0224 22:35:53.786492 1969 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(298)@172.17.0.1:36678 I0224 22:35:53.786962 1969 authenticator.cpp:98] Creating new server SASL connection I0224 22:35:53.787274 1969 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0224 22:35:53.787308 1969 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0224 22:35:53.787400 1969 authenticator.cpp:203] Received SASL authentication start I0224 22:35:53.787470 1969 authenticator.cpp:325] Authentication requires more steps I0224 22:35:53.787884 1972 authenticatee.cpp:258] Received SASL authentication step I0224 22:35:53.787992 1972 authenticator.cpp:231] Received SASL authentication step I0224 22:35:53.788027 1972 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '4db5fa128d2d' server FQDN: '4db5fa128d2d' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0224 22:35:53.788040 1972 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0224 22:35:53.788090 1972 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0224 22:35:53.788122 1972 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '4db5fa128d2d' server FQDN: '4db5fa128d2d' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0224 22:35:53.788136 1972 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0224 22:35:53.788146 1972 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0224 22:35:53.788164 1972 authenticator.cpp:317] Authentication success I0224 22:35:53.788331 1972 authenticatee.cpp:298] Authentication success I0224 22:35:53.788439 1972 master.cpp:5556] Successfully authenticated principal 'test-principal' at slave(115)@172.17.0.1:36678 I0224 22:35:53.788529 1972 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(298)@172.17.0.1:36678 I0224 22:35:53.788988 1972 slave.cpp:927] Successfully authenticated with master master@172.17.0.1:36678 I0224 22:35:53.789139 1972 slave.cpp:1321] Will retry registration in 1.535786ms if necessary I0224 22:35:53.789515 1972 master.cpp:4240] Registering slave at slave(115)@172.17.0.1:36678 (maintenance-host) with id aab18b61-7811-4c43-a672-d1a63818c880-S0 I0224 22:35:53.790577 1972 registrar.cpp:439] Applied 1 operations in 78745ns; attempting to update the 'registry' I0224 22:35:53.791128 1971 process.cpp:3141] Handling HTTP event for process 'master' with path: '/master/maintenance/schedule' I0224 22:35:53.791877 1971 http.cpp:501] HTTP POST for /master/maintenance/schedule from 172.17.0.1:45095 I0224 22:35:53.793313 1972 log.cpp:683] Attempting to append 343 bytes to the log I0224 22:35:53.793586 1972 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0224 22:35:53.794533 1971 replica.cpp:537] Replica received write request for position 3 from (4547)@172.17.0.1:36678 I0224 22:35:53.794862 1971 leveldb.cpp:341] Persisting action (362 bytes) to leveldb took 283614ns I0224 22:35:53.794893 1971 replica.cpp:712] Persisted action at 3 I0224 22:35:53.796646 1979 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0224 22:35:53.797102 1972 slave.cpp:1321] Will retry registration in 17.198963ms if necessary I0224 22:35:53.797186 1979 leveldb.cpp:341] Persisting action (364 bytes) to leveldb took 498502ns I0224 22:35:53.797230 1979 replica.cpp:712] Persisted action at 3 I0224 22:35:53.797260 1979 replica.cpp:697] Replica learned APPEND action at position 3 I0224 22:35:53.797417 1972 master.cpp:4228] Ignoring register slave message from slave(115)@172.17.0.1:36678 (maintenance-host) as admission is already in progress I0224 22:35:53.799119 1978 registrar.cpp:484] Successfully updated the 'registry' in 8.45824ms I0224 22:35:53.799613 1978 registrar.cpp:439] Applied 1 operations in 176193ns; attempting to update the 'registry' I0224 22:35:53.800472 1972 master.cpp:4308] Registered slave aab18b61-7811-4c43-a672-d1a63818c880-S0 at slave(115)@172.17.0.1:36678 (maintenance-host) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0224 22:35:53.800623 1978 log.cpp:702] Attempting to truncate the log to 3 I0224 22:35:53.801255 1969 hierarchical.cpp:473] Added slave aab18b61-7811-4c43-a672-d1a63818c880-S0 (maintenance-host) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0224 22:35:53.801301 1978 slave.cpp:971] Registered with master master@172.17.0.1:36678; given slave ID aab18b61-7811-4c43-a672-d1a63818c880-S0 I0224 22:35:53.801331 1978 fetcher.cpp:81] Clearing fetcher cache I0224 22:35:53.801431 1969 hierarchical.cpp:1434] No resources available to allocate! I0224 22:35:53.801466 1969 hierarchical.cpp:1147] Performed allocation for slave aab18b61-7811-4c43-a672-d1a63818c880-S0 in 162751ns I0224 22:35:53.801532 1969 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0224 22:35:53.801867 1978 slave.cpp:994] Checkpointing SlaveInfo to '/tmp/MasterMaintenanceTest_InverseOffers_ywqvFF/meta/slaves/aab18b61-7811-4c43-a672-d1a63818c880-S0/slave.info' I0224 22:35:53.801877 1969 status_update_manager.cpp:181] Resuming sending status updates I0224 22:35:53.802898 1977 replica.cpp:537] Replica received write request for position 4 from (4548)@172.17.0.1:36678 I0224 22:35:53.803252 1978 slave.cpp:1030] Forwarding total oversubscribed resources I0224 22:35:53.803640 1970 master.cpp:4649] Received update of slave aab18b61-7811-4c43-a672-d1a63818c880-S0 at slave(115)@172.17.0.1:36678 (maintenance-host) with total oversubscribed resources I0224 22:35:53.803858 1977 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 912626ns I0224 22:35:53.803889 1977 replica.cpp:712] Persisted action at 4 I0224 22:35:53.804144 1978 slave.cpp:3482] Received ping from slave-observer(117)@172.17.0.1:36678 I0224 22:35:53.804535 1971 hierarchical.cpp:531] Slave aab18b61-7811-4c43-a672-d1a63818c880-S0 (maintenance-host) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) I0224 22:35:53.804684 1971 hierarchical.cpp:1434] No resources available to allocate! I0224 22:35:53.804714 1971 hierarchical.cpp:1147] Performed allocation for slave aab18b61-7811-4c43-a672-d1a63818c880-S0 in 131453ns I0224 22:35:53.805541 1967 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0224 22:35:53.805941 1967 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 366444ns I0224 22:35:53.806015 1967 leveldb.cpp:399] Deleting ~2 keys from leveldb took 42808ns I0224 22:35:53.806041 1967 replica.cpp:712] Persisted action at 4 I0224 22:35:53.806066 1967 replica.cpp:697] Replica learned TRUNCATE action at position 4 I0224 22:35:53.807355 1978 log.cpp:683] Attempting to append 465 bytes to the log I0224 22:35:53.807551 1978 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 5 I0224 22:35:53.809638 1979 replica.cpp:537] Replica received write request for position 5 from (4549)@172.17.0.1:36678 I0224 22:35:53.810858 1979 leveldb.cpp:341] Persisting action (484 bytes) to leveldb took 1.167663ms I0224 22:35:53.810904 1979 replica.cpp:712] Persisted action at 5 I0224 22:35:53.811997 1979 replica.cpp:691] Replica received learned notice for position 5 from @0.0.0.0:0 I0224 22:35:53...
Story points: 1