Project: mesos
Issue MESOS-4257: ExamplesTest.NoExecutorFramework runs forever.
{noformat: title=Good Run} [ RUN ] ExamplesTest.NoExecutorFramework I1221 23:10:02.721617 32528 exec.cpp:444] Ignoring exited event because the driver is aborted! Using temporary directory '/tmp/ExamplesTest_NoExecutorFramework_fCmFLn' I1221 23:10:02.721675 32539 exec.cpp:444] Ignoring exited event because the driver is aborted! I1221 23:10:02.722024 32554 exec.cpp:444] Ignoring exited event because the driver is aborted! WARNING: Logging before InitGoogleLogging() is written to STDERR I1221 23:10:05.179466 32569 resources.cpp:478] Parsing resources as JSON failed: cpus:0.1;mem:32;disk:32 Trying semicolon-delimited string format instead I1221 23:10:05.180269 32569 logging.cpp:172] Logging to STDERR I1221 23:10:05.185768 32569 process.cpp:998] libprocess is initialized on 172.17.0.2:40874 for 16 cpus I1221 23:10:05.200728 32569 leveldb.cpp:174] Opened db in 4.184362ms I1221 23:10:05.202234 32569 leveldb.cpp:181] Compacted db in 1.459268ms I1221 23:10:05.202353 32569 leveldb.cpp:196] Created db iterator in 73761ns I1221 23:10:05.202383 32569 leveldb.cpp:202] Seeked to beginning of db in 3382ns I1221 23:10:05.202405 32569 leveldb.cpp:271] Iterated through 0 keys in the db in 633ns I1221 23:10:05.202674 32569 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1221 23:10:05.205301 32604 recover.cpp:447] Starting replica recovery I1221 23:10:05.206414 32569 local.cpp:239] Using 'local' authorizer I1221 23:10:05.206405 32604 recover.cpp:473] Replica is in EMPTY status I1221 23:10:05.209595 32594 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (4)@172.17.0.2:40874 I1221 23:10:05.210916 32596 recover.cpp:193] Received a recover response from a replica in EMPTY status I1221 23:10:05.211515 32597 master.cpp:365] Master 3931c1a8-1cd6-49eb-94c8-d01b33bb008e (6ccf2ee56b13) started on 172.17.0.2:40874 I1221 23:10:05.211699 32605 recover.cpp:564] Updating replica status to STARTING I1221 23:10:05.211539 32597 master.cpp:367] Flags at startup: --acls="permissive: false register_frameworks { principals { type: SOME values: "test-principal" } roles { type: SOME values: "*" } } run_tasks { principals { type: SOME values: "test-principal" } users { type: SOME values: "mesos" } } " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/ExamplesTest_NoExecutorFramework_fCmFLn/credentials" --framework_sorter="drf" --help="true" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.27.0/src/webui" --work_dir="/tmp/mesos-otpdch" --zk_session_timeout="10secs" I1221 23:10:05.212323 32597 master.cpp:412] Master only allowing authenticated frameworks to register I1221 23:10:05.212337 32597 master.cpp:419] Master allowing unauthenticated slaves to register I1221 23:10:05.212347 32597 credentials.hpp:35] Loading credentials for authentication from '/tmp/ExamplesTest_NoExecutorFramework_fCmFLn/credentials' W1221 23:10:05.212442 32597 credentials.hpp:50] Permissions on credentials file '/tmp/ExamplesTest_NoExecutorFramework_fCmFLn/credentials' are too open. It is recommended that your credentials file is NOT accessible by others. I1221 23:10:05.212606 32600 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 656857ns I1221 23:10:05.212620 32597 master.cpp:456] Using default 'crammd5' authenticator I1221 23:10:05.212631 32600 replica.cpp:320] Persisted replica status to STARTING I1221 23:10:05.212893 32597 authenticator.cpp:518] Initializing server SASL I1221 23:10:05.213091 32608 recover.cpp:473] Replica is in STARTING status I1221 23:10:05.213958 32595 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (5)@172.17.0.2:40874 I1221 23:10:05.214323 32594 recover.cpp:193] Received a recover response from a replica in STARTING status I1221 23:10:05.214689 32595 recover.cpp:564] Updating replica status to VOTING I1221 23:10:05.215353 32596 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 487419ns I1221 23:10:05.215384 32596 replica.cpp:320] Persisted replica status to VOTING I1221 23:10:05.215481 32605 recover.cpp:578] Successfully joined the Paxos group I1221 23:10:05.215867 32605 recover.cpp:462] Recover process terminated I1221 23:10:05.216111 32569 containerizer.cpp:141] Using isolation: filesystem/posix,posix/cpu,posix/mem W1221 23:10:05.217021 32569 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I1221 23:10:05.221482 32608 slave.cpp:191] Slave started on 1)@172.17.0.2:40874 I1221 23:10:05.221521 32608 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/mesos-0.27.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/mesos-otpdch/0" I1221 23:10:05.222578 32608 resources.cpp:478] Parsing resources as JSON failed: cpus:2;mem:10240 Trying semicolon-delimited string format instead I1221 23:10:05.223465 32608 slave.cpp:392] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I1221 23:10:05.223621 32569 containerizer.cpp:141] Using isolation: filesystem/posix,posix/cpu,posix/mem I1221 23:10:05.223610 32608 slave.cpp:400] Slave attributes: [ ] I1221 23:10:05.223677 32608 slave.cpp:405] Slave hostname: 6ccf2ee56b13 I1221 23:10:05.223697 32608 slave.cpp:410] Slave checkpoint: true W1221 23:10:05.224143 32569 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I1221 23:10:05.226668 32604 slave.cpp:191] Slave started on 2)@172.17.0.2:40874 I1221 23:10:05.226692 32604 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/mesos-0.27.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/mesos-otpdch/1" I1221 23:10:05.227520 32604 resources.cpp:478] Parsing resources as JSON failed: cpus:2;mem:10240 Trying semicolon-delimited string format instead I1221 23:10:05.228037 32604 slave.cpp:392] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I1221 23:10:05.228148 32604 slave.cpp:400] Slave attributes: [ ] I1221 23:10:05.228169 32604 slave.cpp:405] Slave hostname: 6ccf2ee56b13 I1221 23:10:05.228184 32604 slave.cpp:410] Slave checkpoint: true I1221 23:10:05.229123 32569 containerizer.cpp:141] Using isolation: filesystem/posix,posix/cpu,posix/mem I1221 23:10:05.229641 32605 state.cpp:58] Recovering state from '/tmp/mesos-otpdch/0/meta' W1221 23:10:05.229645 32569 backend.cpp:48] Failed to create 'bind' backend: BindBackend requires root privileges I1221 23:10:05.229636 32595 state.cpp:58] Recovering state from '/tmp/mesos-otpdch/1/meta' I1221 23:10:05.230242 32605 status_update_manager.cpp:200] Recovering status update manager I1221 23:10:05.230254 32598 status_update_manager.cpp:200] Recovering status update manager I1221 23:10:05.230515 32601 containerizer.cpp:383] Recovering containerizer I1221 23:10:05.230562 32602 containerizer.cpp:383] Recovering containerizer I1221 23:10:05.232681 32597 auxprop.cpp:71] Initialized in-memory auxiliary property plugin I1221 23:10:05.232803 32597 master.cpp:493] Authorization enabled I1221 23:10:05.232867 32600 slave.cpp:4427] Finished recovery I1221 23:10:05.232980 32598 slave.cpp:191] Slave started on 3)@172.17.0.2:40874 I1221 23:10:05.233039 32594 slave.cpp:4427] Finished recovery I1221 23:10:05.233376 32599 whitelist_watcher.cpp:77] No whitelist given I1221 23:10:05.233428 32601 hierarchical.cpp:147] Initialized hierarchical allocator process I1221 23:10:05.233003 32598 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/mesos-0.27.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/mesos-otpdch/2" I1221 23:10:05.233744 32600 slave.cpp:4599] Querying resource estimator for oversubscribable resources I1221 23:10:05.233749 32598 resources.cpp:478] Parsing resources as JSON failed: cpus:2;mem:10240 Trying semicolon-delimited string format instead I1221 23:10:05.234222 32598 slave.cpp:392] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I1221 23:10:05.234284 32598 slave.cpp:400] Slave attributes: [ ] I1221 23:10:05.234299 32598 slave.cpp:405] Slave hostname: 6ccf2ee56b13 I1221 23:10:05.234311 32598 slave.cpp:410] Slave checkpoint: true I1221 23:10:05.234338 32600 slave.cpp:729] New master detected at master@172.17.0.2:40874 I1221 23:10:05.234376 32604 status_update_manager.cpp:174] Pausing sending status updates I1221 23:10:05.234424 32600 slave.cpp:754] No credentials provided. Attempting to register without authentication I1221 23:10:05.234522 32600 slave.cpp:765] Detecting new master I1221 23:10:05.234616 32569 sched.cpp:164] Version: 0.27.0 I1221 23:10:05.234658 32600 slave.cpp:4613] Received oversubscribable resources from the resource estimator I1221 23:10:05.234671 32594 slave.cpp:4599] Querying resource estimator for oversubscribable resources I1221 23:10:05.234884 32606 slave.cpp:4613] Received oversubscribable resources from the resource estimator I1221 23:10:05.235038 32595 status_update_manager.cpp:174] Pausing sending status updates I1221 23:10:05.235043 32606 slave.cpp:729] New master detected at master@172.17.0.2:40874 I1221 23:10:05.235111 32606 slave.cpp:754] No credentials provided. Attempting to register without authentication I1221 23:10:05.235147 32606 slave.cpp:765] Detecting new master I1221 23:10:05.235240 32594 state.cpp:58] Recovering state from '/tmp/mesos-otpdch/2/meta' I1221 23:10:05.235443 32608 status_update_manager.cpp:200] Recovering status update manager I1221 23:10:05.235625 32594 containerizer.cpp:383] Recovering containerizer I1221 23:10:05.236549 32599 slave.cpp:4427] Finished recovery I1221 23:10:05.236984 32593 sched.cpp:262] New master detected at master@172.17.0.2:40874 I1221 23:10:05.237004 32599 slave.cpp:4599] Querying resource estimator for oversubscribable resources I1221 23:10:05.237221 32593 sched.cpp:318] Authenticating with master master@172.17.0.2:40874 I1221 23:10:05.237277 32593 sched.cpp:325] Using default CRAM-MD5 authenticatee I1221 23:10:05.237285 32604 status_update_manager.cpp:174] Pausing sending status updates I1221 23:10:05.237288 32599 slave.cpp:729] New master detected at master@172.17.0.2:40874 I1221 23:10:05.237361 32599 slave.cpp:754] No credentials provided. Attempting to register without authentication I1221 23:10:05.237433 32599 slave.cpp:765] Detecting new master I1221 23:10:05.237565 32599 slave.cpp:4613] Received oversubscribable resources from the resource estimator I1221 23:10:05.238154 32605 authenticatee.cpp:97] Initializing client SASL I1221 23:10:05.238315 32605 authenticatee.cpp:121] Creating new client SASL connection I1221 23:10:05.239640 32597 master.cpp:1200] Dropping 'mesos.internal.AuthenticateMessage' message since not elected yet I1221 23:10:05.239765 32597 master.cpp:1629] The newly elected leader is master@172.17.0.2:40874 with id 3931c1a8-1cd6-49eb-94c8-d01b33bb008e I1221 23:10:05.239794 32597 master.cpp:1642] Elected as the leading master! I1221 23:10:05.239843 32597 master.cpp:1387] Recovering from registrar I1221 23:10:05.240056 32600 registrar.cpp:307] Recovering registrar I1221 23:10:05.241477 32608 log.cpp:659] Attempting to start the writer I1221 23:10:05.244540 32600 replica.cpp:493] Replica received implicit promise request from (39)@172.17.0.2:40874 with proposal 1 I1221 23:10:05.245358 32600 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 776937ns I1221 23:10:05.245393 32600 replica.cpp:342] Persisted promised to 1 I1221 23:10:05.246625 32601 coordinator.cpp:238] Coordinator attempting to fill missing positions I1221 23:10:05.248757 32605 replica.cpp:388] Replica received explicit promise request from (40)@172.17.0.2:40874 for position 0 with proposal 2 I1221 23:10:05.249214 32605 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 366567ns I1221 23:10:05.249246 32605 replica.cpp:712] Persisted action at 0 I1221 23:10:05.250998 32599 replica.cpp:537] Replica received write request for position 0 from (41)@172.17.0.2:40874 I1221 23:10:05.251111 32599 leveldb.cpp:436] Reading position from leveldb took 66773ns I1221 23:10:05.251734 32599 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 379612ns I1221 23:10:05.251759 32599 replica.cpp:712] Persisted action at 0 I1221 23:10:05.252555 32601 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I1221 23:10:05.253010 32601 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 381858ns I1221 23:10:05.253036 32601 replica.cpp:712] Persisted action at 0 I1221 23:10:05.253068 32601 replica.cpp:697] Replica learned NOP action at position 0 I1221 23:10:05.254043 32595 log.cpp:675] Writer started with ending position 0 I1221 23:10:05.256741 32595 leveldb.cpp:436] Reading position from leveldb took 48607ns I1221 23:10:05.260617 32601 registrar.cpp:340] Successfully fetched the registry (0B) in 20.47616ms I1221 23:10:05.260988 32601 registrar.cpp:439] Applied 1 operations in 103123ns; attempting to update the 'registry' I1221 23:10:05.264700 32604 log.cpp:683] Attempting to append 170 bytes to the log I1221 23:10:05.265138 32601 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I1221 23:10:05.266208 32603 replica.cpp:537] Replica received write request for position 1 from (42)@172.17.0.2:40874 I1221 23:10:05.266829 32603 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 551087ns I1221 23:10:05.266861 32603 replica.cpp:712] Persisted action at 1 I1221 23:10:05.267918 32605 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I1221 23:10:05.268442 32605 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 453416ns I1221 23:10:05.268470 32605 replica.cpp:712] Persisted action at 1 I1221 23:10:05.268506 32605 replica.cpp:697] Replica learned APPEND action at position 1 I1221 23:10:05.270512 32606 registrar.cpp:484] Successfully updated the 'registry' in 9.375232ms I1221 23:10:05.270705 32606 registrar.cpp:370] Successfully recovered registrar I1221 23:10:05.271045 32602 log.cpp:702] Attempting to truncate the log to 1 I1221 23:10:05.271178 32603 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I1221 23:10:05.271695 32605...
Story points: 3