Project: mesos
Issue MESOS-4002: ReservationEndpointsTest.UnreserveAvailableAndOfferedResources is flaky
Showed up on ASF CI: ( test kept looping on and on and ultimately failing the build after 300 minutes ) https://builds.apache.org/job/Mesos/COMPILER=gcc,CONFIGURATION=--verbose,OS=ubuntu%3A14.04,label_exp=docker%7C%7CHadoop/1269/changes {code} [ RUN ] ReservationEndpointsTest.UnreserveAvailableAndOfferedResources I1124 01:07:20.050729 30260 leveldb.cpp:174] Opened db in 107.434842ms I1124 01:07:20.099630 30260 leveldb.cpp:181] Compacted db in 48.82312ms I1124 01:07:20.099722 30260 leveldb.cpp:196] Created db iterator in 29905ns I1124 01:07:20.099738 30260 leveldb.cpp:202] Seeked to beginning of db in 3145ns I1124 01:07:20.099750 30260 leveldb.cpp:271] Iterated through 0 keys in the db in 279ns I1124 01:07:20.099804 30260 replica.cpp:778] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1124 01:07:20.100637 30292 recover.cpp:447] Starting replica recovery I1124 01:07:20.100934 30292 recover.cpp:473] Replica is in EMPTY status I1124 01:07:20.103240 30288 replica.cpp:674] Replica in EMPTY status received a broadcasted recover request from (6305)@172.17.18.107:37993 I1124 01:07:20.103672 30292 recover.cpp:193] Received a recover response from a replica in EMPTY status I1124 01:07:20.104142 30292 recover.cpp:564] Updating replica status to STARTING I1124 01:07:20.114534 30284 master.cpp:365] Master ad27bc60-16d1-4239-9a65-235a991f9600 (9f2f81738d5e) started on 172.17.18.107:37993 I1124 01:07:20.114558 30284 master.cpp:367] Flags at startup: --acls="" --allocation_interval="1000secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/I60I5f/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/I60I5f/master" --zk_session_timeout="10secs" I1124 01:07:20.114809 30284 master.cpp:412] Master only allowing authenticated frameworks to register I1124 01:07:20.114820 30284 master.cpp:417] Master only allowing authenticated slaves to register I1124 01:07:20.114825 30284 credentials.hpp:35] Loading credentials for authentication from '/tmp/I60I5f/credentials' I1124 01:07:20.115067 30284 master.cpp:456] Using default 'crammd5' authenticator I1124 01:07:20.115320 30284 master.cpp:493] Authorization enabled I1124 01:07:20.115792 30285 hierarchical.cpp:162] Initialized hierarchical allocator process I1124 01:07:20.115855 30285 whitelist_watcher.cpp:77] No whitelist given I1124 01:07:20.118755 30285 master.cpp:1625] The newly elected leader is master@172.17.18.107:37993 with id ad27bc60-16d1-4239-9a65-235a991f9600 I1124 01:07:20.118788 30285 master.cpp:1638] Elected as the leading master! I1124 01:07:20.118809 30285 master.cpp:1383] Recovering from registrar I1124 01:07:20.119078 30285 registrar.cpp:307] Recovering registrar I1124 01:07:20.143256 30292 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 38.787419ms I1124 01:07:20.143347 30292 replica.cpp:321] Persisted replica status to STARTING I1124 01:07:20.143717 30292 recover.cpp:473] Replica is in STARTING status I1124 01:07:20.145454 30286 replica.cpp:674] Replica in STARTING status received a broadcasted recover request from (6307)@172.17.18.107:37993 I1124 01:07:20.145979 30292 recover.cpp:193] Received a recover response from a replica in STARTING status I1124 01:07:20.146654 30292 recover.cpp:564] Updating replica status to VOTING I1124 01:07:20.182672 30286 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 35.422256ms I1124 01:07:20.182747 30286 replica.cpp:321] Persisted replica status to VOTING I1124 01:07:20.182929 30286 recover.cpp:578] Successfully joined the Paxos group I1124 01:07:20.183115 30286 recover.cpp:462] Recover process terminated I1124 01:07:20.183831 30286 log.cpp:659] Attempting to start the writer I1124 01:07:20.185907 30285 replica.cpp:494] Replica received implicit promise request from (6308)@172.17.18.107:37993 with proposal 1 I1124 01:07:20.225256 30285 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 39.291288ms I1124 01:07:20.225344 30285 replica.cpp:343] Persisted promised to 1 I1124 01:07:20.226850 30286 coordinator.cpp:238] Coordinator attempting to fill missing positions I1124 01:07:20.228394 30293 replica.cpp:389] Replica received explicit promise request from (6309)@172.17.18.107:37993 for position 0 with proposal 2 I1124 01:07:20.266371 30293 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 37.874181ms I1124 01:07:20.266456 30293 replica.cpp:713] Persisted action at 0 I1124 01:07:20.267927 30293 replica.cpp:538] Replica received write request for position 0 from (6310)@172.17.18.107:37993 I1124 01:07:20.268002 30293 leveldb.cpp:436] Reading position from leveldb took 37139ns I1124 01:07:20.308117 30293 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 39.961976ms I1124 01:07:20.308205 30293 replica.cpp:713] Persisted action at 0 I1124 01:07:20.309033 30290 replica.cpp:692] Replica received learned notice for position 0 from @0.0.0.0:0 I1124 01:07:20.343257 30290 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 34.175337ms I1124 01:07:20.343343 30290 replica.cpp:713] Persisted action at 0 I1124 01:07:20.343377 30290 replica.cpp:698] Replica learned NOP action at position 0 I1124 01:07:20.344446 30282 log.cpp:675] Writer started with ending position 0 I1124 01:07:20.346143 30291 leveldb.cpp:436] Reading position from leveldb took 56896ns I1124 01:07:20.347618 30291 registrar.cpp:340] Successfully fetched the registry (0B) in 228.495104ms I1124 01:07:20.347862 30291 registrar.cpp:439] Applied 1 operations in 41164ns; attempting to update the 'registry' I1124 01:07:20.348794 30279 log.cpp:683] Attempting to append 178 bytes to the log I1124 01:07:20.349081 30279 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I1124 01:07:20.350244 30294 replica.cpp:538] Replica received write request for position 1 from (6311)@172.17.18.107:37993 I1124 01:07:20.385246 30294 leveldb.cpp:341] Persisting action (197 bytes) to leveldb took 34.872508ms I1124 01:07:20.385323 30294 replica.cpp:713] Persisted action at 1 I1124 01:07:20.386814 30294 replica.cpp:692] Replica received learned notice for position 1 from @0.0.0.0:0 I1124 01:07:20.425163 30294 leveldb.cpp:341] Persisting action (199 bytes) to leveldb took 38.282493ms I1124 01:07:20.425262 30294 replica.cpp:713] Persisted action at 1 I1124 01:07:20.425298 30294 replica.cpp:698] Replica learned APPEND action at position 1 I1124 01:07:20.427994 30287 registrar.cpp:484] Successfully updated the 'registry' in 79.949056ms I1124 01:07:20.428141 30283 log.cpp:702] Attempting to truncate the log to 1 I1124 01:07:20.428738 30287 registrar.cpp:370] Successfully recovered registrar I1124 01:07:20.429306 30290 master.cpp:1435] Recovered 0 slaves from the Registry (139B) ; allowing 10mins for slaves to re-register I1124 01:07:20.429592 30290 hierarchical.cpp:174] Allocator recovery is not supported yet I1124 01:07:20.430083 30294 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I1124 01:07:20.431411 30294 replica.cpp:538] Replica received write request for position 2 from (6312)@172.17.18.107:37993 I1124 01:07:20.467258 30294 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 35.661978ms I1124 01:07:20.467342 30294 replica.cpp:713] Persisted action at 2 I1124 01:07:20.468842 30290 replica.cpp:692] Replica received learned notice for position 2 from @0.0.0.0:0 I1124 01:07:20.502264 30290 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 33.367074ms I1124 01:07:20.502426 30290 leveldb.cpp:399] Deleting ~1 keys from leveldb took 80765ns I1124 01:07:20.502452 30290 replica.cpp:713] Persisted action at 2 I1124 01:07:20.502488 30290 replica.cpp:698] Replica learned TRUNCATE action at position 2 I1124 01:07:20.510509 30260 containerizer.cpp:141] Using isolation: posix/cpu,posix/mem,filesystem/posix W1124 01:07:20.511119 30260 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I1124 01:07:20.516801 30288 slave.cpp:189] Slave started on 219)@172.17.18.107:37993 I1124 01:07:20.516839 30288 slave.cpp:190] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr" I1124 01:07:20.517670 30288 credentials.hpp:83] Loading credential for authentication from '/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr/credential' I1124 01:07:20.517982 30288 slave.cpp:320] Slave using credential for: test-principal I1124 01:07:20.518334 30288 resources.cpp:472] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I1124 01:07:20.518815 30260 resources.cpp:472] Parsing resources as JSON failed: cpus:1;mem:128 Trying semicolon-delimited string format instead I1124 01:07:20.518975 30288 slave.cpp:390] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I1124 01:07:20.519104 30288 slave.cpp:398] Slave attributes: [ ] I1124 01:07:20.519124 30288 slave.cpp:403] Slave hostname: 9f2f81738d5e I1124 01:07:20.519136 30288 slave.cpp:408] Slave checkpoint: true I1124 01:07:20.519407 30260 resources.cpp:472] Parsing resources as JSON failed: mem:384 Trying semicolon-delimited string format instead I1124 01:07:20.522702 30288 state.cpp:52] Recovering state from '/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr/meta' I1124 01:07:20.523265 30288 status_update_manager.cpp:200] Recovering status update manager I1124 01:07:20.523531 30288 containerizer.cpp:383] Recovering containerizer I1124 01:07:20.524998 30288 slave.cpp:4258] Finished recovery I1124 01:07:20.525802 30288 slave.cpp:4430] Querying resource estimator for oversubscribable resources I1124 01:07:20.526753 30288 slave.cpp:727] New master detected at master@172.17.18.107:37993 I1124 01:07:20.527292 30288 slave.cpp:790] Authenticating with master master@172.17.18.107:37993 I1124 01:07:20.528240 30288 slave.cpp:795] Using default CRAM-MD5 authenticatee I1124 01:07:20.527003 30286 status_update_manager.cpp:174] Pausing sending status updates I1124 01:07:20.528955 30285 authenticatee.cpp:121] Creating new client SASL connection I1124 01:07:20.529469 30285 master.cpp:5169] Authenticating slave(219)@172.17.18.107:37993 I1124 01:07:20.529729 30283 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(515)@172.17.18.107:37993 I1124 01:07:20.530287 30283 authenticator.cpp:98] Creating new server SASL connection I1124 01:07:20.530764 30285 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I1124 01:07:20.530903 30285 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I1124 01:07:20.531096 30285 authenticator.cpp:203] Received SASL authentication start I1124 01:07:20.531241 30285 authenticator.cpp:325] Authentication requires more steps I1124 01:07:20.531388 30285 authenticatee.cpp:258] Received SASL authentication step I1124 01:07:20.531616 30285 authenticator.cpp:231] Received SASL authentication step I1124 01:07:20.531668 30285 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '9f2f81738d5e' server FQDN: '9f2f81738d5e' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1124 01:07:20.531690 30285 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I1124 01:07:20.531774 30285 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1124 01:07:20.531834 30285 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: '9f2f81738d5e' server FQDN: '9f2f81738d5e' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1124 01:07:20.531855 30285 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1124 01:07:20.531867 30285 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1124 01:07:20.531903 30285 authenticator.cpp:317] Authentication success I1124 01:07:20.532016 30283 authenticatee.cpp:298] Authentication success I1124 01:07:20.532331 30281 master.cpp:5199] Successfully authenticated principal 'test-principal' at slave(219)@172.17.18.107:37993 I1124 01:07:20.532652 30291 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(515)@172.17.18.107:37993 I1124 01:07:20.533113 30288 slave.cpp:763] Detecting new master I1124 01:07:20.533628 30288 slave.cpp:4444] Received oversubscribable resources from the resource estimator I1124 01:07:20.546396 30288 slave.cpp:858] Successfully authenticated with master master@172.17.18.107:37993 I1124 01:07:20.547111 30287 master.cpp:3878] Registering slave at slave(219)@172.17.18.107:37993 (9f2f81738d5e) with id ad27bc60-16d1-4239-9a65-235a991f9600-S0 I1124 01:07:20.547886 30287 registrar.cpp:439] Applied 1 operations in 91121ns; attempting to update the 'registry' I1124 01:07:20.550647 30287 log.cpp:683] Attempting to append 347 bytes to the log I1124 01:07:20.550935 30279 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I1124 01:07:20.551534 30288 slave.cpp:1252] Will retry registration in 3.399312ms if necessary I1124 01:07:20.551868 30291 replica.cpp:538] Replica received write request for position 3 from (6324)@172.17.18.107:37993 I1124 01:07:20.557605 30281 slave.cpp:1252] Will retry registration in 16.296866ms if necessary I1124 01:07:20.557891 30293 master.cpp:3866] Ignoring register slave message from slave(219)@172.17.18.107:37993 (9f2f81738d5e) as admission is already in progress I1124 01:07:20.574681 30279 slave.cpp:1252] Will retry registration in 73.52632ms if necessary I1124 01:07:20.575078 30293 master.cpp:3866] Ignoring register slave message from slave(219)@172.17.18.107:37993 (9f2f81738d5e) as admission is already in progress I1124 01:07:20.586236 30291 leveldb.cpp:341] Persisting action (366 bytes) to leveldb took 34.301173ms I1124 01:07:20.586287 30291 replica.cpp:713] Persisted action at 3 I1124 01:07:20.587509 30289 replica.cpp:692] Replica received learned notice for position 3 from @0.0.0.0:0 I1124 01:07:20.611263 30289 leveldb.cpp:341] Persisting action (368 bytes) to leveldb took 23.677211ms I1124 01:07:20.611352 30289 replica.cpp:713] Persisted action at 3 I1124 01:07:20.611387 30289 replica.cpp:698] Replica learned APPEND action at position 3 I1124 01:07:20.613580 30279 registrar.cpp:484] Successfully updated the 'registry' in 65.490944ms I1124 01:07:20.613802 30288 log.cpp:702] Attempting to truncate the log to 3 I1124 01:07:20.613993 30288 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I1124 01:07:20.615281 30289 replica.cpp:538] Replica received write request for position 4 from (6325)@172.17.18.107:37993 I1124 01:07:20.615883 30279 master.cpp:3946] Registered slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 at slave(219)@172.17.18.107:37993 (9f2f81738d5e) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I1124 01:07:20.616261 30282 slave.cpp:902] Registered with master master@172.17.18.107:37993; given slave ID ad27bc60-16d1-4239-9a65-235a991f9600-S0 I1124 01:07:20.616883 30282 fetcher.cpp:79] Clearing fetcher cache I1124 01:07:20.617261 30280 status_update_manager.cpp:181] Resuming sending status updates I1124 01:07:20.617766 30282 slave.cpp:925] Checkpointing SlaveInfo to '/tmp/ReservationEndpointsTest_UnreserveAvailableAndOfferedResources_CSzecr/meta/slaves/ad27bc60-16d1-4239-9a65-235a991f9600-S0/slave.info' I1124 01:07:20.616550 30284 hierarchical.cpp:380] Added slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 (9f2f81738d5e) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I1124 01:07:20.618670 30282 slave.cpp:961] Forwarding total oversubscribed resources I1124 01:07:20.618932 30282 slave.cpp:3197] Received ping from slave-observer(216)@172.17.18.107:37993 I1124 01:07:20.619288 30285 master.cpp:4288] Received update of slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 at slave(219)@172.17.18.107:37993 (9f2f81738d5e) with total oversubscribed resources I1124 01:07:20.619446 30284 hierarchical.cpp:1066] No resources available to allocate! I1124 01:07:20.619526 30284 hierarchical.cpp:1159] No inverse offers to send out! I1124 01:07:20.619568 30284 hierarchical.cpp:977] Performed allocation for slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 in 1.108641ms I1124 01:07:20.620057 30284 hierarchical.cpp:436] Slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 (9f2f81738d5e) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) I1124 01:07:20.620393 30284 hierarchical.cpp:1066] No resources available to allocate! I1124 01:07:20.620462 30284 hierarchical.cpp:1159] No inverse offers to send out! I1124 01:07:20.620507 30284 hierarchical.cpp:977] Performed allocation for slave ad27bc60-16d1-4239-9a65-235a991f9600-S0 in 395959ns I1124 01:07:20.624356 30285 process.cpp:3067] Handling HTTP event for process 'master' with path: '/master/reserve' I1124 01:07:20.624418 30285 http.cpp:336] HTTP POST for /master/reserve from 172.17.18.107:48995 I1124 01:07:20.626936 30285 master.cpp:6224] Sending checkpointed resources cpus(role, te...
Story points: 1