Project: mesos
Issue MESOS-5146: MasterAllocatorTest/1.RebalancedForUpdatedWeights is flaky.
Observed on the ASF CI: {code} [ RUN ] MasterAllocatorTest/1.RebalancedForUpdatedWeights I0407 22:34:10.330394 29278 cluster.cpp:149] Creating default 'local' authorizer I0407 22:34:10.466182 29278 leveldb.cpp:174] Opened db in 135.608207ms I0407 22:34:10.516398 29278 leveldb.cpp:181] Compacted db in 50.159558ms I0407 22:34:10.516464 29278 leveldb.cpp:196] Created db iterator in 34959ns I0407 22:34:10.516484 29278 leveldb.cpp:202] Seeked to beginning of db in 10195ns I0407 22:34:10.516496 29278 leveldb.cpp:271] Iterated through 0 keys in the db in 7324ns I0407 22:34:10.516547 29278 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0407 22:34:10.517277 29298 recover.cpp:447] Starting replica recovery I0407 22:34:10.517693 29300 recover.cpp:473] Replica is in EMPTY status I0407 22:34:10.520251 29310 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (4775)@172.17.0.3:35855 I0407 22:34:10.520611 29311 recover.cpp:193] Received a recover response from a replica in EMPTY status I0407 22:34:10.521164 29299 recover.cpp:564] Updating replica status to STARTING I0407 22:34:10.523435 29298 master.cpp:382] Master f59f9057-a5c7-43e1-b129-96862e640a12 (129e11060069) started on 172.17.0.3:35855 I0407 22:34:10.523473 29298 master.cpp:384] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/3rZY8C/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.29.0/_inst/share/mesos/webui" --work_dir="/tmp/3rZY8C/master" --zk_session_timeout="10secs" I0407 22:34:10.523885 29298 master.cpp:433] Master only allowing authenticated frameworks to register I0407 22:34:10.523901 29298 master.cpp:438] Master only allowing authenticated agents to register I0407 22:34:10.523913 29298 credentials.hpp:37] Loading credentials for authentication from '/tmp/3rZY8C/credentials' I0407 22:34:10.524298 29298 master.cpp:480] Using default 'crammd5' authenticator I0407 22:34:10.524441 29298 master.cpp:551] Using default 'basic' HTTP authenticator I0407 22:34:10.524564 29298 master.cpp:589] Authorization enabled I0407 22:34:10.525269 29305 hierarchical.cpp:145] Initialized hierarchical allocator process I0407 22:34:10.525333 29305 whitelist_watcher.cpp:77] No whitelist given I0407 22:34:10.527331 29298 master.cpp:1832] The newly elected leader is master@172.17.0.3:35855 with id f59f9057-a5c7-43e1-b129-96862e640a12 I0407 22:34:10.527441 29298 master.cpp:1845] Elected as the leading master! I0407 22:34:10.527545 29298 master.cpp:1532] Recovering from registrar I0407 22:34:10.527889 29298 registrar.cpp:331] Recovering registrar I0407 22:34:10.549734 29299 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 28.25177ms I0407 22:34:10.549782 29299 replica.cpp:320] Persisted replica status to STARTING I0407 22:34:10.550010 29299 recover.cpp:473] Replica is in STARTING status I0407 22:34:10.551352 29299 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (4777)@172.17.0.3:35855 I0407 22:34:10.551676 29299 recover.cpp:193] Received a recover response from a replica in STARTING status I0407 22:34:10.552315 29308 recover.cpp:564] Updating replica status to VOTING I0407 22:34:10.574865 29308 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 22.413614ms I0407 22:34:10.574928 29308 replica.cpp:320] Persisted replica status to VOTING I0407 22:34:10.575103 29308 recover.cpp:578] Successfully joined the Paxos group I0407 22:34:10.575346 29308 recover.cpp:462] Recover process terminated I0407 22:34:10.575913 29308 log.cpp:659] Attempting to start the writer I0407 22:34:10.577512 29308 replica.cpp:493] Replica received implicit promise request from (4778)@172.17.0.3:35855 with proposal 1 I0407 22:34:10.599984 29308 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 22.453613ms I0407 22:34:10.600026 29308 replica.cpp:342] Persisted promised to 1 I0407 22:34:10.601773 29304 coordinator.cpp:238] Coordinator attempting to fill missing positions I0407 22:34:10.603757 29307 replica.cpp:388] Replica received explicit promise request from (4779)@172.17.0.3:35855 for position 0 with proposal 2 I0407 22:34:10.634392 29307 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 30.269987ms I0407 22:34:10.634829 29307 replica.cpp:712] Persisted action at 0 I0407 22:34:10.637017 29297 replica.cpp:537] Replica received write request for position 0 from (4780)@172.17.0.3:35855 I0407 22:34:10.637099 29297 leveldb.cpp:436] Reading position from leveldb took 52948ns I0407 22:34:10.676170 29297 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 38.917487ms I0407 22:34:10.676352 29297 replica.cpp:712] Persisted action at 0 I0407 22:34:10.677564 29306 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0407 22:34:10.717959 29306 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 40.306229ms I0407 22:34:10.718202 29306 replica.cpp:712] Persisted action at 0 I0407 22:34:10.718399 29306 replica.cpp:697] Replica learned NOP action at position 0 I0407 22:34:10.719883 29306 log.cpp:675] Writer started with ending position 0 I0407 22:34:10.721688 29305 leveldb.cpp:436] Reading position from leveldb took 75934ns I0407 22:34:10.723640 29306 registrar.cpp:364] Successfully fetched the registry (0B) in 195648us I0407 22:34:10.723999 29306 registrar.cpp:463] Applied 1 operations in 108099ns; attempting to update the 'registry' I0407 22:34:10.725077 29311 log.cpp:683] Attempting to append 170 bytes to the log I0407 22:34:10.725328 29308 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0407 22:34:10.726552 29299 replica.cpp:537] Replica received write request for position 1 from (4781)@172.17.0.3:35855 I0407 22:34:10.759747 29299 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 33.089719ms I0407 22:34:10.759976 29299 replica.cpp:712] Persisted action at 1 I0407 22:34:10.761739 29299 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0407 22:34:10.801522 29299 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 39.694064ms I0407 22:34:10.801602 29299 replica.cpp:712] Persisted action at 1 I0407 22:34:10.801638 29299 replica.cpp:697] Replica learned APPEND action at position 1 I0407 22:34:10.803371 29311 registrar.cpp:508] Successfully updated the 'registry' in 79.163904ms I0407 22:34:10.803829 29311 registrar.cpp:394] Successfully recovered registrar I0407 22:34:10.804585 29311 master.cpp:1640] Recovered 0 agents from the Registry (131B) ; allowing 10mins for agents to re-register I0407 22:34:10.805269 29308 log.cpp:702] Attempting to truncate the log to 1 I0407 22:34:10.805721 29310 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0407 22:34:10.805276 29296 hierarchical.cpp:172] Skipping recovery of hierarchical allocator: nothing to recover I0407 22:34:10.806529 29307 replica.cpp:537] Replica received write request for position 2 from (4782)@172.17.0.3:35855 I0407 22:34:10.843320 29307 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 36.77593ms I0407 22:34:10.843531 29307 replica.cpp:712] Persisted action at 2 I0407 22:34:10.845369 29311 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0407 22:34:10.885098 29311 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 39.641102ms I0407 22:34:10.885401 29311 leveldb.cpp:399] Deleting ~1 keys from leveldb took 88701ns I0407 22:34:10.885745 29311 replica.cpp:712] Persisted action at 2 I0407 22:34:10.885862 29311 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0407 22:34:10.900660 29278 containerizer.cpp:155] Using isolation: posix/cpu,posix/mem,filesystem/posix W0407 22:34:10.901793 29278 backend.cpp:66] Failed to create 'bind' backend: BindBackend requires root privileges I0407 22:34:10.905488 29302 slave.cpp:201] Agent started on 111)@172.17.0.3:35855 I0407 22:34:10.905553 29302 slave.cpp:202] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_credentials="/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.29.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:4096;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa" I0407 22:34:10.906365 29302 credentials.hpp:86] Loading credential for authentication from '/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/credential' I0407 22:34:10.906787 29302 slave.cpp:339] Agent using credential for: test-principal I0407 22:34:10.907202 29302 credentials.hpp:37] Loading credentials for authentication from '/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/http_credentials' I0407 22:34:10.907713 29302 slave.cpp:391] Using default 'basic' HTTP authenticator I0407 22:34:10.908499 29302 resources.cpp:572] Parsing resources as JSON failed: cpus:2;mem:1024;disk:4096;ports:[31000-32000] Trying semicolon-delimited string format instead I0407 22:34:10.910189 29302 slave.cpp:590] Agent resources: cpus(*):2; mem(*):1024; disk(*):4096; ports(*):[31000-32000] I0407 22:34:10.910362 29302 slave.cpp:598] Agent attributes: [ ] I0407 22:34:10.910465 29302 slave.cpp:603] Agent hostname: 129e11060069 I0407 22:34:10.913280 29303 state.cpp:57] Recovering state from '/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/meta' I0407 22:34:10.914621 29303 status_update_manager.cpp:200] Recovering status update manager I0407 22:34:10.915226 29303 containerizer.cpp:416] Recovering containerizer I0407 22:34:10.917246 29301 provisioner.cpp:245] Provisioner recovery complete I0407 22:34:10.917733 29301 slave.cpp:4784] Finished recovery I0407 22:34:10.918226 29301 slave.cpp:4956] Querying resource estimator for oversubscribable resources I0407 22:34:10.918529 29301 slave.cpp:4970] Received oversubscribable resources from the resource estimator I0407 22:34:10.918908 29304 slave.cpp:939] New master detected at master@172.17.0.3:35855 I0407 22:34:10.918988 29304 slave.cpp:1002] Authenticating with master master@172.17.0.3:35855 I0407 22:34:10.919098 29301 status_update_manager.cpp:174] Pausing sending status updates I0407 22:34:10.919309 29304 slave.cpp:1007] Using default CRAM-MD5 authenticatee I0407 22:34:10.919535 29304 slave.cpp:975] Detecting new master I0407 22:34:10.919747 29308 authenticatee.cpp:121] Creating new client SASL connection I0407 22:34:10.920413 29308 master.cpp:5695] Authenticating slave(111)@172.17.0.3:35855 I0407 22:34:10.920650 29308 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(278)@172.17.0.3:35855 I0407 22:34:10.921020 29308 authenticator.cpp:98] Creating new server SASL connection I0407 22:34:10.921308 29308 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0407 22:34:10.921424 29308 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0407 22:34:10.921596 29308 authenticator.cpp:203] Received SASL authentication start I0407 22:34:10.921752 29308 authenticator.cpp:325] Authentication requires more steps I0407 22:34:10.921957 29307 authenticatee.cpp:258] Received SASL authentication step I0407 22:34:10.922178 29308 authenticator.cpp:231] Received SASL authentication step I0407 22:34:10.922214 29308 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '129e11060069' server FQDN: '129e11060069' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0407 22:34:10.922229 29308 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0407 22:34:10.922281 29308 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0407 22:34:10.922309 29308 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '129e11060069' server FQDN: '129e11060069' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0407 22:34:10.922322 29308 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0407 22:34:10.922332 29308 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0407 22:34:10.922353 29308 authenticator.cpp:317] Authentication success I0407 22:34:10.922436 29307 authenticatee.cpp:298] Authentication success I0407 22:34:10.922587 29308 master.cpp:5725] Successfully authenticated principal 'test-principal' at slave(111)@172.17.0.3:35855 I0407 22:34:10.922668 29299 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(278)@172.17.0.3:35855 I0407 22:34:10.923256 29307 slave.cpp:1072] Successfully authenticated with master master@172.17.0.3:35855 I0407 22:34:10.923429 29307 slave.cpp:1468] Will retry registration in 3.220345ms if necessary I0407 22:34:10.923707 29302 master.cpp:4406] Registering agent at slave(111)@172.17.0.3:35855 (129e11060069) with id f59f9057-a5c7-43e1-b129-96862e640a12-S0 I0407 22:34:10.924239 29309 registrar.cpp:463] Applied 1 operations in 105794ns; attempting to update the 'registry' I0407 22:34:10.925787 29309 log.cpp:683] Attempting to append 339 bytes to the log I0407 22:34:10.926028 29309 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0407 22:34:10.927139 29309 replica.cpp:537] Replica received write request for position 3 from (4797)@172.17.0.3:35855 I0407 22:34:10.929083 29305 slave.cpp:1468] Will retry registration in 39.293556ms if necessary I0407 22:34:10.929363 29305 master.cpp:4394] Ignoring register agent message from slave(111)@172.17.0.3:35855 (129e11060069) as admission is already in progress I0407 22:34:10.968843 29309 leveldb.cpp:341] Persisting action (358 bytes) to leveldb took 41.68025ms I0407 22:34:10.969005 29309 replica.cpp:712] Persisted action at 3 I0407 22:34:10.969741 29309 slave.cpp:1468] Will retry registration in 54.852242ms if necessary I0407 22:34:10.970118 29309 master.cpp:4394] Ignoring register agent message from slave(111)@172.17.0.3:35855 (129e11060069) as admission is already in progress I0407 22:34:10.970852 29306 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0407 22:34:11.010634 29306 leveldb.cpp:341] Persisting action (360 bytes) to leveldb took 39.680272ms I0407 22:34:11.010840 29306 replica.cpp:712] Persisted action at 3 I0407 22:34:11.011014 29306 replica.cpp:697] Replica learned APPEND action at position 3 I0407 22:34:11.014020 29306 registrar.cpp:508] Successfully updated the 'registry' in 89.684224ms I0407 22:34:11.014181 29296 log.cpp:702] Attempting to truncate the log to 3 I0407 22:34:11.014606 29296 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0407 22:34:11.015836 29298 replica.cpp:537] Replica received write request for position 4 from (4798)@172.17.0.3:35855 I0407 22:34:11.016973 29296 master.cpp:4474] Registered agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 at slave(111)@172.17.0.3:35855 (129e11060069) with cpus(*):2; mem(*):1024; disk(*):4096; ports(*):[31000-32000] I0407 22:34:11.017518 29304 hierarchical.cpp:476] Added agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 (129e11060069) with cpus(*):2; mem(*):1024; disk(*):4096; ports(*):[31000-32000] (allocated: ) I0407 22:34:11.017763 29311 slave.cpp:1116] Registered with master master@172.17.0.3:35855; given agent ID f59f9057-a5c7-43e1-b129-96862e640a12-S0 I0407 22:34:11.018362 29311 fetcher.cpp:81] Clearing fetcher cache I0407 22:34:11.018870 29311 slave.cpp:1139] Checkpointing SlaveInfo to '/tmp/MasterAllocatorTest_1_RebalancedForUpdatedWeights_9aCAYa/meta/slaves/f59f9057-a5c7-43e1-b129-96862e640a12-S0/slave.info' I0407 22:34:11.018890 29307 status_update_manager.cpp:181] Resuming sending status updates I0407 22:34:11.019182 29304 hierarchical.cpp:1491] No resources available to allocate! I0407 22:34:11.019304 29304 hierarchical.cpp:1165] Performed allocation for agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 in 1.077349ms I0407 22:34:11.019493 29311 slave.cpp:1176] Forwarding total oversubscribed resources I0407 22:34:11.019726 29311 slave.cpp:3675] Received ping from slave-observer(112)@172.17.0.3:35855 I0407 22:34:11.019878 29299 master.cpp:4818] Received update of agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 at slave(111)@172.17.0.3:35855 (129e11060069) with total oversubscribed resources I0407 22:34:11.020845 29305 hierarchical.cpp:534] Agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 (129e11060069) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):4096; ports(*):[31000-32000], allocated: ) I0407 22:34:11.021005 29305 hierarchical.cpp:1491] No resources available to allocate! I0407 22:34:11.021065 29305 hierarchical.cpp:1165] Performed allocation for agent f59f9057-a5c7-43e1-b129-96862e640a12-S0 in 173907ns I0407 22:34:11.022289 29278 containerizer.cpp:155] Using isolation: posix/cpu,posix/mem,filesystem/posix W0407 22:34:11.023422 29278 backend.cpp:66] Failed to create 'bind' backend: BindBackend requires root privileges I0407 22:34:11.026309 29309 slave.cpp:201] Agent started on 112)@172.17.0.3:35855 I0407 22:34:11.026410 29309 slave.cpp:202] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --cgroups_cpu_e...
Story points: 1