Project: mesos
Issue MESOS-3273: EventCall Test Framework is flaky
Observed this on ASF CI. h/t [~haosdent@gmail.com] Looks like the HTTP scheduler never sent a SUBSCRIBE request to the master. {code} [ RUN ] ExamplesTest.EventCallFramework Using temporary directory '/tmp/ExamplesTest_EventCallFramework_k4vXkx' I0813 19:55:15.643579 26085 exec.cpp:443] Ignoring exited event because the driver is aborted! Shutting down Sending SIGTERM to process tree at pid 26061 Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 26062 Shutting down Killing the following process trees: [ ] Sending SIGTERM to process tree at pid 26063 Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 26098 Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 26099 Killing the following process trees: [ ] WARNING: Logging before InitGoogleLogging() is written to STDERR I0813 19:55:17.161726 26100 process.cpp:1012] libprocess is initialized on 172.17.2.10:60249 for 16 cpus I0813 19:55:17.161888 26100 logging.cpp:177] Logging to STDERR I0813 19:55:17.163625 26100 scheduler.cpp:157] Version: 0.24.0 I0813 19:55:17.175302 26100 leveldb.cpp:176] Opened db in 3.167446ms I0813 19:55:17.176393 26100 leveldb.cpp:183] Compacted db in 1.047996ms I0813 19:55:17.176496 26100 leveldb.cpp:198] Created db iterator in 77155ns I0813 19:55:17.176518 26100 leveldb.cpp:204] Seeked to beginning of db in 8429ns I0813 19:55:17.176527 26100 leveldb.cpp:273] Iterated through 0 keys in the db in 4219ns I0813 19:55:17.176708 26100 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0813 19:55:17.178951 26136 recover.cpp:449] Starting replica recovery I0813 19:55:17.179934 26136 recover.cpp:475] Replica is in EMPTY status I0813 19:55:17.181970 26126 master.cpp:378] Master 20150813-195517-167907756-60249-26100 (297daca2d01a) started on 172.17.2.10:60249 I0813 19:55:17.182317 26126 master.cpp:380] Flags at startup: --acls="permissive: false register_frameworks { principals { type: SOME values: "test-principal" } roles { type: SOME values: "*" } } run_tasks { principals { type: SOME values: "test-principal" } users { type: SOME values: "mesos" } } " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --credentials="/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials" --framework_sorter="drf" --help="false" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.24.0/src/webui" --work_dir="/tmp/mesos-II8Gua" --zk_session_timeout="10secs" I0813 19:55:17.183475 26126 master.cpp:427] Master allowing unauthenticated frameworks to register I0813 19:55:17.183536 26126 master.cpp:432] Master allowing unauthenticated slaves to register I0813 19:55:17.183615 26126 credentials.hpp:37] Loading credentials for authentication from '/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials' W0813 19:55:17.183859 26126 credentials.hpp:52] Permissions on credentials file '/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials' are too open. It is recommended that your credentials file is NOT accessible by others. I0813 19:55:17.183969 26123 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0813 19:55:17.184306 26126 master.cpp:469] Using default 'crammd5' authenticator I0813 19:55:17.184661 26126 authenticator.cpp:512] Initializing server SASL I0813 19:55:17.185104 26138 recover.cpp:195] Received a recover response from a replica in EMPTY status I0813 19:55:17.185972 26100 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix I0813 19:55:17.186058 26135 recover.cpp:566] Updating replica status to STARTING I0813 19:55:17.187001 26138 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 654586ns I0813 19:55:17.187037 26138 replica.cpp:323] Persisted replica status to STARTING I0813 19:55:17.187499 26134 recover.cpp:475] Replica is in STARTING status I0813 19:55:17.187605 26126 auxprop.cpp:66] Initialized in-memory auxiliary property plugin I0813 19:55:17.187710 26126 master.cpp:506] Authorization enabled I0813 19:55:17.188657 26138 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0813 19:55:17.188853 26131 hierarchical.hpp:346] Initialized hierarchical allocator process I0813 19:55:17.189252 26132 whitelist_watcher.cpp:79] No whitelist given I0813 19:55:17.189321 26134 recover.cpp:195] Received a recover response from a replica in STARTING status I0813 19:55:17.190001 26125 recover.cpp:566] Updating replica status to VOTING I0813 19:55:17.190696 26124 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 357331ns I0813 19:55:17.190775 26124 replica.cpp:323] Persisted replica status to VOTING I0813 19:55:17.190970 26133 recover.cpp:580] Successfully joined the Paxos group I0813 19:55:17.192183 26129 recover.cpp:464] Recover process terminated I0813 19:55:17.192699 26123 slave.cpp:190] Slave started on 1)@172.17.2.10:60249 I0813 19:55:17.192741 26123 slave.cpp:191] Flags at startup: --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-II8Gua/0" I0813 19:55:17.194514 26100 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix I0813 19:55:17.194658 26123 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0813 19:55:17.194854 26123 slave.cpp:384] Slave hostname: 297daca2d01a I0813 19:55:17.194877 26123 slave.cpp:389] Slave checkpoint: true I0813 19:55:17.196751 26132 master.cpp:1524] The newly elected leader is master@172.17.2.10:60249 with id 20150813-195517-167907756-60249-26100 I0813 19:55:17.196797 26132 master.cpp:1537] Elected as the leading master! I0813 19:55:17.196815 26132 master.cpp:1307] Recovering from registrar I0813 19:55:17.197032 26138 registrar.cpp:311] Recovering registrar I0813 19:55:17.197845 26132 slave.cpp:190] Slave started on 2)@172.17.2.10:60249 I0813 19:55:17.198420 26125 log.cpp:661] Attempting to start the writer I0813 19:55:17.197948 26132 slave.cpp:191] Flags at startup: --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-II8Gua/1" I0813 19:55:17.199121 26132 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0813 19:55:17.199235 26138 state.cpp:54] Recovering state from '/tmp/mesos-II8Gua/0/meta' I0813 19:55:17.199322 26132 slave.cpp:384] Slave hostname: 297daca2d01a I0813 19:55:17.199345 26132 slave.cpp:389] Slave checkpoint: true I0813 19:55:17.199676 26100 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix I0813 19:55:17.200085 26135 state.cpp:54] Recovering state from '/tmp/mesos-II8Gua/1/meta' I0813 19:55:17.200317 26132 status_update_manager.cpp:202] Recovering status update manager I0813 19:55:17.200371 26129 status_update_manager.cpp:202] Recovering status update manager I0813 19:55:17.202003 26129 replica.cpp:477] Replica received implicit promise request with proposal 1 I0813 19:55:17.202585 26131 slave.cpp:190] Slave started on 3)@172.17.2.10:60249 I0813 19:55:17.202596 26129 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 523191ns I0813 19:55:17.202756 26129 replica.cpp:345] Persisted promised to 1 I0813 19:55:17.202770 26132 containerizer.cpp:379] Recovering containerizer I0813 19:55:17.203061 26135 containerizer.cpp:379] Recovering containerizer I0813 19:55:17.202663 26131 slave.cpp:191] Flags at startup: --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-II8Gua/2" I0813 19:55:17.203819 26131 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0813 19:55:17.203930 26131 slave.cpp:384] Slave hostname: 297daca2d01a I0813 19:55:17.203948 26131 slave.cpp:389] Slave checkpoint: true I0813 19:55:17.204674 26137 state.cpp:54] Recovering state from '/tmp/mesos-II8Gua/2/meta' I0813 19:55:17.205178 26135 status_update_manager.cpp:202] Recovering status update manager I0813 19:55:17.205323 26135 containerizer.cpp:379] Recovering containerizer I0813 19:55:17.205521 26136 slave.cpp:4069] Finished recovery I0813 19:55:17.206074 26136 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0813 19:55:17.206424 26128 slave.cpp:4069] Finished recovery I0813 19:55:17.206722 26137 status_update_manager.cpp:176] Pausing sending status updates I0813 19:55:17.206858 26136 slave.cpp:684] New master detected at master@172.17.2.10:60249 I0813 19:55:17.206902 26138 slave.cpp:4069] Finished recovery I0813 19:55:17.206962 26128 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0813 19:55:17.208312 26134 scheduler.cpp:272] New master detected at master@172.17.2.10:60249 I0813 19:55:17.208364 26136 slave.cpp:709] No credentials provided. Attempting to register without authentication I0813 19:55:17.208608 26136 slave.cpp:720] Detecting new master I0813 19:55:17.208839 26138 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0813 19:55:17.209216 26123 coordinator.cpp:231] Coordinator attemping to fill missing position I0813 19:55:17.209247 26127 status_update_manager.cpp:176] Pausing sending status updates I0813 19:55:17.209259 26128 slave.cpp:684] New master detected at master@172.17.2.10:60249 I0813 19:55:17.209322 26127 status_update_manager.cpp:176] Pausing sending status updates I0813 19:55:17.209364 26128 slave.cpp:709] No credentials provided. Attempting to register without authentication I0813 19:55:17.209344 26138 slave.cpp:684] New master detected at master@172.17.2.10:60249 I0813 19:55:17.209455 26128 slave.cpp:720] Detecting new master I0813 19:55:17.209492 26138 slave.cpp:709] No credentials provided. Attempting to register without authentication I0813 19:55:17.209573 26128 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0813 19:55:17.209601 26138 slave.cpp:720] Detecting new master I0813 19:55:17.209730 26138 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0813 19:55:17.209883 26136 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0813 19:55:17.211266 26136 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0813 19:55:17.211771 26136 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 462128ns I0813 19:55:17.211797 26136 replica.cpp:679] Persisted action at 0 I0813 19:55:17.212980 26130 replica.cpp:511] Replica received write request for position 0 I0813 19:55:17.213124 26130 leveldb.cpp:438] Reading position from leveldb took 67075ns I0813 19:55:17.213580 26130 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 301649ns I0813 19:55:17.213603 26130 replica.cpp:679] Persisted action at 0 I0813 19:55:17.214284 26123 replica.cpp:658] Replica received learned notice for position 0 I0813 19:55:17.214622 26123 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 284547ns I0813 19:55:17.214648 26123 replica.cpp:679] Persisted action at 0 I0813 19:55:17.214675 26123 replica.cpp:664] Replica learned NOP action at position 0 I0813 19:55:17.215420 26136 log.cpp:677] Writer started with ending position 0 I0813 19:55:17.217463 26133 leveldb.cpp:438] Reading position from leveldb took 47943ns I0813 19:55:17.220762 26125 registrar.cpp:344] Successfully fetched the registry (0B) in 23.649024ms I0813 19:55:17.221081 26125 registrar.cpp:443] Applied 1 operations in 136902ns; attempting to update the 'registry' I0813 19:55:17.223667 26133 log.cpp:685] Attempting to append 174 bytes to the log I0813 19:55:17.223778 26125 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 1 I0813 19:55:17.224516 26127 replica.cpp:511] Replica received write request for position 1 I0813 19:55:17.225009 26127 leveldb.cpp:343] Persisting action (193 bytes) to leveldb took 466230ns I0813 19:55:17.225042 26127 replica.cpp:679] Persisted action at 1 I0813 19:55:17.225653 26126 replica.cpp:658] Replica received learned notice for position 1 I0813 19:55:17.225953 26126 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 286966ns I0813 19:55:17.225975 26126 replica.cpp:679] Persisted action at 1 I0813 19:55:17.226013 26126 replica.cpp:664] Replica learned APPEND action at position 1 I0813 19:55:17.227545 26137 registrar.cpp:488] Successfully updated the 'registry' in 6.328064ms I0813 19:55:17.227722 26137 registrar.cpp:374] Successfully recovered registrar I0813 19:55:17.227918 26124 log.cpp:704] Attempting to truncate the log to 1 I0813 19:55:17.228024 26133 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 2 I0813 19:55:17.228193 26131 master.cpp:1334] Recovered 0 slaves from the Registry (135B) ; allowing 10mins for slaves to re-register I0813 19:55:17.228659 26127 replica.cpp:511] Replica received write request for position 2 I0813 19:55:17.228972 26127 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 297903ns I0813 19:55:17.229004 26127 replica.cpp:679] Persisted action at 2 I0813 19:55:17.229565 26127 replica.cpp:658] Replica received learned notice for position 2 I0813 19:55:17.229837 26127 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 260326ns I0813 19:55:17.229899 26127 leveldb.cpp:401] Deleting ~1 keys from leveldb took 48697ns I0813 19:55:17.229923 26127 replica.cpp:679] Persisted action at 2 I0813 19:55:17.229956 26127 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0813 19:55:17.325634 26138 slave.cpp:1209] Will retry registration in 445.955946ms if necessary I0813 19:55:17.326088 26124 master.cpp:3635] Registering slave at slave(2)@172.17.2.10:60249 (297daca2d01a) with id 20150813-195517-167907756-60249-26100-S0 I0813 19:55:17.327446 26124 registrar.cpp:443] Applied 1 operations in 231072ns; attempting to update the 'registry' I0813 19:55:17.330252 26136 log.cpp:685] Attempting to append 344 bytes to the log I0813 19:55:17.330407 26132 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 3 I0813 19:55:17.331418 26128 replica.cpp:511] Replica received write request for position 3 I0813 19:55:17.331753 26128 leveldb.cpp:343] Persisting action (363 bytes) to leveldb took 264140ns I0813 19:55:17.331778 26128 replica.cpp:679] Persisted action at 3 I0813 19:55:17.332324 26133 replica.cpp:658] Replica received learned notice for position 3 I0813 19:55:17.332809 26133 leveldb.cpp:343] Persisting action (365 bytes) to leveldb took 313064ns I0813 19:55:17.332834 26133 replica.cpp:679] Persisted action at 3 I0813 19:55:17.332865 26133 replica.cpp:664] Replica learned APPEND action at position 3 I0813 19:55:17.334211 26132 registrar.cpp:488] Successfully updated the 'registry' in 6.668032ms I0813 19:55:17.334430 26127 log.cpp:704] Attempting to truncate the log to 3 I0813 19:55:17.334566 26132 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 4 I0813 19:55:17.335283 26129 replica.cpp:511] Replica received write request for position 4 I0813 19:55:17.335615 26127 slave.cpp:3058] Received ping from slave-observer(1)@172.17.2.10:60249 I0813 19:55:17.335816 26129 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 458268ns I0813 19:55:17.335908 26137 master.cpp:3698] Registered slave 20150813-195517-167907756-60249-26100-S0 at slave(2)@172.17.2.10:60249 (297daca2d01a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+0...
Story points: 5