Project: mesos
Issue MESOS-5669: CNI isolator should not return failure if /etc/hostname does not exist on host.
/etc/hostname may not necessarily exist on every system (e.g., CentOS 6). Currently CNI isolator just return a failure if it does not exist on host, because the isolator need to mount it into the container. This is fine for /etc/host and /etc/resolv.conf, but we should make an exception for /etc/hostname, because hostname may still be accessible even if /etc/hostname doesn't exist. This issue relates to 3 failure tests: {noformat} [22:45:21] :	[Step 10/10] [ RUN ] CniIsolatorTest.ROOT_INTERNET_CURL_LaunchCommandTask [22:45:21]W:	[Step 10/10] I0619 22:45:21.647611 24647 cluster.cpp:155] Creating default 'local' authorizer [22:45:21]W:	[Step 10/10] I0619 22:45:21.655230 24647 leveldb.cpp:174] Opened db in 7.510408ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.657680 24647 leveldb.cpp:181] Compacted db in 2.427309ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.657702 24647 leveldb.cpp:196] Created db iterator in 6209ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.657709 24647 leveldb.cpp:202] Seeked to beginning of db in 692ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.657713 24647 leveldb.cpp:271] Iterated through 0 keys in the db in 431ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.657727 24647 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [22:45:21]W:	[Step 10/10] I0619 22:45:21.657888 24662 recover.cpp:451] Starting replica recovery [22:45:21]W:	[Step 10/10] I0619 22:45:21.658051 24668 recover.cpp:477] Replica is in EMPTY status [22:45:21]W:	[Step 10/10] I0619 22:45:21.658495 24664 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (18401)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.658583 24662 recover.cpp:197] Received a recover response from a replica in EMPTY status [22:45:21]W:	[Step 10/10] I0619 22:45:21.658687 24664 recover.cpp:568] Updating replica status to STARTING [22:45:21]W:	[Step 10/10] I0619 22:45:21.659111 24664 master.cpp:382] Master 9a4a353b-91c5-43b9-8c37-19245c37758c (ip-172-30-2-247.mesosphere.io) started on 172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.659126 24664 master.cpp:384] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http="true" --authenticate_http_frameworks="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/l8346Z/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/l8346Z/master" --zk_session_timeout="10secs" [22:45:21]W:	[Step 10/10] I0619 22:45:21.659267 24664 master.cpp:434] Master only allowing authenticated frameworks to register [22:45:21]W:	[Step 10/10] I0619 22:45:21.659276 24664 master.cpp:448] Master only allowing authenticated agents to register [22:45:21]W:	[Step 10/10] I0619 22:45:21.659278 24664 master.cpp:461] Master only allowing authenticated HTTP frameworks to register [22:45:21]W:	[Step 10/10] I0619 22:45:21.659282 24664 credentials.hpp:37] Loading credentials for authentication from '/tmp/l8346Z/credentials' [22:45:21]W:	[Step 10/10] I0619 22:45:21.659375 24664 master.cpp:506] Using default 'crammd5' authenticator [22:45:21]W:	[Step 10/10] I0619 22:45:21.659415 24664 master.cpp:578] Using default 'basic' HTTP authenticator [22:45:21]W:	[Step 10/10] I0619 22:45:21.659495 24664 master.cpp:658] Using default 'basic' HTTP framework authenticator [22:45:21]W:	[Step 10/10] I0619 22:45:21.659569 24664 master.cpp:705] Authorization enabled [22:45:21]W:	[Step 10/10] I0619 22:45:21.659684 24666 hierarchical.cpp:142] Initialized hierarchical allocator process [22:45:21]W:	[Step 10/10] I0619 22:45:21.659696 24665 whitelist_watcher.cpp:77] No whitelist given [22:45:21]W:	[Step 10/10] I0619 22:45:21.660269 24666 master.cpp:1969] The newly elected leader is master@172.30.2.247:42024 with id 9a4a353b-91c5-43b9-8c37-19245c37758c [22:45:21]W:	[Step 10/10] I0619 22:45:21.660281 24666 master.cpp:1982] Elected as the leading master! [22:45:21]W:	[Step 10/10] I0619 22:45:21.660290 24666 master.cpp:1669] Recovering from registrar [22:45:21]W:	[Step 10/10] I0619 22:45:21.660342 24662 registrar.cpp:332] Recovering registrar [22:45:21]W:	[Step 10/10] I0619 22:45:21.661232 24669 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.48585ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.661254 24669 replica.cpp:320] Persisted replica status to STARTING [22:45:21]W:	[Step 10/10] I0619 22:45:21.661326 24669 recover.cpp:477] Replica is in STARTING status [22:45:21]W:	[Step 10/10] I0619 22:45:21.661667 24668 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (18404)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.661758 24665 recover.cpp:197] Received a recover response from a replica in STARTING status [22:45:21]W:	[Step 10/10] I0619 22:45:21.661893 24664 recover.cpp:568] Updating replica status to VOTING [22:45:21]W:	[Step 10/10] I0619 22:45:21.663851 24664 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.915617ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.663866 24664 replica.cpp:320] Persisted replica status to VOTING [22:45:21]W:	[Step 10/10] I0619 22:45:21.663899 24664 recover.cpp:582] Successfully joined the Paxos group [22:45:21]W:	[Step 10/10] I0619 22:45:21.663944 24664 recover.cpp:466] Recover process terminated [22:45:21]W:	[Step 10/10] I0619 22:45:21.664088 24668 log.cpp:553] Attempting to start the writer [22:45:21]W:	[Step 10/10] I0619 22:45:21.664556 24668 replica.cpp:493] Replica received implicit promise request from (18405)@172.30.2.247:42024 with proposal 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.666551 24668 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.971938ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.666566 24668 replica.cpp:342] Persisted promised to 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.666767 24667 coordinator.cpp:238] Coordinator attempting to fill missing positions [22:45:21]W:	[Step 10/10] I0619 22:45:21.667230 24668 replica.cpp:388] Replica received explicit promise request from (18406)@172.30.2.247:42024 for position 0 with proposal 2 [22:45:21]W:	[Step 10/10] I0619 22:45:21.669271 24668 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 2.02399ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.669287 24668 replica.cpp:712] Persisted action at 0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.669656 24669 replica.cpp:537] Replica received write request for position 0 from (18407)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.669680 24669 leveldb.cpp:436] Reading position from leveldb took 10808ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.671674 24669 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 1.977316ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.671689 24669 replica.cpp:712] Persisted action at 0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.671907 24665 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.673920 24665 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.991274ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.673935 24665 replica.cpp:712] Persisted action at 0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.673941 24665 replica.cpp:697] Replica learned NOP action at position 0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.674190 24665 log.cpp:569] Writer started with ending position 0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.674489 24663 leveldb.cpp:436] Reading position from leveldb took 9059ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.674718 24663 registrar.cpp:365] Successfully fetched the registry (0B) in 14.355968ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.674747 24663 registrar.cpp:464] Applied 1 operations in 3070ns; attempting to update the 'registry' [22:45:21]W:	[Step 10/10] I0619 22:45:21.674935 24665 log.cpp:577] Attempting to append 209 bytes to the log [22:45:21]W:	[Step 10/10] I0619 22:45:21.674978 24665 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.675242 24666 replica.cpp:537] Replica received write request for position 1 from (18408)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.677088 24666 leveldb.cpp:341] Persisting action (228 bytes) to leveldb took 1.823904ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.677103 24666 replica.cpp:712] Persisted action at 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.677299 24667 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.679270 24667 leveldb.cpp:341] Persisting action (230 bytes) to leveldb took 1.952303ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.679286 24667 replica.cpp:712] Persisted action at 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.679291 24667 replica.cpp:697] Replica learned APPEND action at position 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.679481 24663 registrar.cpp:509] Successfully updated the 'registry' in 4.715264ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.679503 24666 log.cpp:596] Attempting to truncate the log to 1 [22:45:21]W:	[Step 10/10] I0619 22:45:21.679560 24667 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 [22:45:21]W:	[Step 10/10] I0619 22:45:21.679581 24663 registrar.cpp:395] Successfully recovered registrar [22:45:21]W:	[Step 10/10] I0619 22:45:21.679745 24664 master.cpp:1777] Recovered 0 agents from the Registry (170B) ; allowing 10mins for agents to re-register [22:45:21]W:	[Step 10/10] I0619 22:45:21.679774 24662 hierarchical.cpp:169] Skipping recovery of hierarchical allocator: nothing to recover [22:45:21]W:	[Step 10/10] I0619 22:45:21.679986 24662 replica.cpp:537] Replica received write request for position 2 from (18409)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.681895 24662 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.891877ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.681910 24662 replica.cpp:712] Persisted action at 2 [22:45:21]W:	[Step 10/10] I0619 22:45:21.682160 24666 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 [22:45:21]W:	[Step 10/10] I0619 22:45:21.684331 24666 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 2.153217ms [22:45:21]W:	[Step 10/10] I0619 22:45:21.684375 24666 leveldb.cpp:399] Deleting ~1 keys from leveldb took 26973ns [22:45:21]W:	[Step 10/10] I0619 22:45:21.684383 24666 replica.cpp:712] Persisted action at 2 [22:45:21]W:	[Step 10/10] I0619 22:45:21.684389 24666 replica.cpp:697] Replica learned TRUNCATE action at position 2 [22:45:21]W:	[Step 10/10] I0619 22:45:21.691529 24647 containerizer.cpp:201] Using isolation: docker/runtime,filesystem/linux,network/cni [22:45:21]W:	[Step 10/10] I0619 22:45:21.694491 24647 linux_launcher.cpp:101] Using /cgroup/freezer as the freezer hierarchy for the Linux launcher [22:45:21]W:	[Step 10/10] E0619 22:45:21.699741 24647 shell.hpp:106] Command 'hadoop version 2>&1' failed; this is the output: [22:45:21]W:	[Step 10/10] sh: hadoop: command not found [22:45:21]W:	[Step 10/10] I0619 22:45:21.699769 24647 fetcher.cpp:62] Skipping URI fetcher plugin 'hadoop' as it could not be created: Failed to create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was either not found or exited with a non-zero exit status: 127 [22:45:21]W:	[Step 10/10] I0619 22:45:21.699823 24647 registry_puller.cpp:111] Creating registry puller with docker registry 'https://registry-1.docker.io' [22:45:21]W:	[Step 10/10] I0619 22:45:21.700865 24647 linux.cpp:146] Bind mounting '/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG' and making it a shared mount [22:45:21]W:	[Step 10/10] I0619 22:45:21.707801 24647 cni.cpp:286] Bind mounting '/var/run/mesos/isolators/network/cni' and making it a shared mount [22:45:21]W:	[Step 10/10] I0619 22:45:21.714337 24647 cluster.cpp:432] Creating default 'local' authorizer [22:45:21]W:	[Step 10/10] I0619 22:45:21.714825 24668 slave.cpp:203] Agent started on 468)@172.30.2.247:42024 [22:45:21]W:	[Step 10/10] I0619 22:45:21.714839 24668 slave.cpp:204] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/l8346Z/store" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG/http_credentials" --image_providers="docker" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="docker/runtime,filesystem/linux,network/cni" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --network_cni_config_dir="/tmp/l8346Z/configs" --network_cni_plugins_dir="/tmp/l8346Z/plugins" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG" [22:45:21]W:	[Step 10/10] I0619 22:45:21.715116 24668 credentials.hpp:86] Loading credential for authentication from '/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG/credential' [22:45:21]W:	[Step 10/10] I0619 22:45:21.715195 24668 slave.cpp:341] Agent using credential for: test-principal [22:45:21]W:	[Step 10/10] I0619 22:45:21.715214 24668 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/CniIsolatorTest_ROOT_INTERNET_CURL_LaunchCommandTask_CVAWpG/http_credentials' [22:45:21]W:	[Step 10/10] I0619 22:45:21.715296 24668 slave.cpp:393] Using default 'basic' HTTP authenticator [22:45:21]W:	[Step 10/10] I0619 22:45:21.715400 24668 resources.cpp:572] Parsing resources as JSON failed: cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000] {noformat} {noformat} [22:45:38] :	[Step 10/10] [ RUN ] CniIsolatorTest.ROOT_VerifyCheckpointedInfo [22:45:38]W:	[Step 10/10] I0619 22:45:38.459836 24647 cluster.cpp:155] Creating default 'local' authorizer [22:45:38]W:	[Step 10/10] I0619 22:45:38.470319 24647 leveldb.cpp:174] Opened db in 10.34226ms [22:45:38]W:	[Step 10/10] I0619 22:45:38.472771 24647 leveldb.cpp:181] Compacted db in 2.403554ms [22:45:38]W:	[Step 10/10] I0619 22:45:38.472795 24647 leveldb.cpp:196] Created db iterator in 4446ns [22:45:38]W:	[Step 10/10] I0619 22:45:38.472801 24647 leveldb.cpp:202] Seeked to beginning of db in 810ns [22:45:38]W:	[Step 10/10] I0619 22:45:38.472806 24647 leveldb.cpp:271] Iterated through 0 keys in the db in 393ns [22:45:38]W:	[Step 10/10] I0619 22:45:38.472822 24647 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [22:45:38]W:	[Step 10/10] I0619 22:45:38.473093 24665 recover.cpp:451] Starting replica recovery [22:45:38]W:	[Step 10/10] I0619 22:45:38.473260 24663 recover.cpp:477] Replica is in EMPTY status [22:45:38]W:	[Step 10/10] I0619 22:45:38.473647 24663 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (18464)@172.30.2.247:42024 [22:45:38]W:	[Step 10/10] I0619 22:45:38.473752 24665 recover.cpp:197] Received a recover response from a replica in EMPTY status [22:45:38]W:	[Step 10/10] I0619 22:45:38.473896 24667 recover.cpp:568] Updating replica status to STARTING [22:45:38]W:	[Step 10/10] I0619 22:45:38.474319 24663 master.cpp:382] Master 64f1f7ac-e810-4fb1-b549-6e29fc62622b (ip-172-30-2-247.mesosphere.io) started on 172.30.2.247:42024 [22:45:38]W:	[Step 10/10] I0619 22:45:38.474329 24663 master.cpp:384] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http="true" --authenticate_http_frameworks="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/qJWqSY/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/qJWqSY/master" --zk_session_timeout="10secs" [22:45:38]W:	[Step 10/10] I0619 22:45:38.474452 24663 master.cpp:434] Master only allowing authenticated frameworks to register [22:45:38]W:	[Step 10/10] I0619 22:45:38.474457 24663 master.cpp:448] Master only allowing authenticated agents to register [22:45:38]W:	[Step 10/10] I0619 22:45:38.474459 24663 master.cpp:461] Master only allowing authenticated HTTP frameworks to register [22:45:38]W:	[Step 10/10] I0619 22:45:38.474463 24663 credentials.hpp:37] Loading credentials for authentication from '/tmp/qJWqSY/credentials' [22:45:38]W:	[Step 10/10] I0619 22:45:38.474551 24663 master.cpp:506] Using default 'crammd5' authenticator [22:45:38]W:	[Step 10/10] I0619 22:45:38.474598 24663 master.cpp:578] Using default 'basic' HTTP authenticator [22:45:38]W:	[Step 10/10] I0619 22:45:38.474643 24663 master.cpp:658] Using default 'basic' HTTP framework authenticator [22:45:38]W:	[Step 10/10] I0619 22:45:38.474674 2466...
Story points: 3