Change:
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? Ingress, hosts and /etc/hosts Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request There have already been a few issues regarding modifying : , But both have a different use-case. First some information before I explain my use-case. I have a pod which run a web app (nginx container + php-fpm container) and it available over multiple sub domains. I want request to the exposed sub-/domains (ex. ) to go directly to the local webserver instant of going the long way (Cloudflare -Ingress Controller -Node -Container). There is 2 reason behind this: is faster/reduce overhead. of our app code call the exposed domains, and need to hit the same server, as the request oriented from (the code write to some tmp file, and other PHP code need to read that tmp file, when it get a request. Bad design: properly). cc Kubernetes version (use ): Environment: DigitalOcean, CoreOS.
I'm not sure I understand the use case, especially the 2nd. This is the case of a pod talking to itself via the public hostname? Why not just use localhost?
Public hostname == ingress . Yes, without going through the load balancer, some of the PHP also require "sticky" sessions and then it is just easier if the exposed domains just point to localhost. That require extra logic in the app code and when the app code expose multiple sub-/domains, then it is just easier adding the domain to /etc/hosts.
Do you know all the hostnames at pod startup time, or can they possibly be later? I guess I am not against this idea in principle, I just have a nagging suspicion it will blow in all the corner cases. I guess we could look at a community implementation, if someone did one. I don't think that we have time on our side right now to drive it.
I currently have a script like this: , which "extract" the hostnames from the nginx config at container startup (and "patch" the location of the file). It also configure to forward some request internally (we don't expose all domains externally).
-- I'm not sure about your particular use case as the host name extraction logic seems to be quite complex (parsing of the nginx config). The two things I do see that are useful: Ability for a ConfigMap to be written out as a single file into the container instead of injecting it as a directory of files. This would solve a lot of problems where a piece of the container configuration needs to be trivially changed. I'm fairly certain this is already filed, I can't seem to find the issue. Inheritance of host entries from the outer node.
I think the use-case beyond service is if the hosts need to be loopback or needs to be configured per pod. I tried to understand how is controlled by kubelet, and it seems like this feature would not be hard to implement. I'm willing to spend some time on this. in terms of implementation: from my reading, host mount is done by inside , and the content of the hosts file is controlled by the methods that call them all have access to the pod object. it seems like we can add a field to PodSpec (can be done as an Pod-level alpha annotation to start): or if we want something more explicit: that can be read into and , which then mounts as to the Pod.
Let me know if ^ makes sense. If so, I can make a formal proposal in or just a PR, whichever is appropriate.
I don't think this warrants a proposal or feature. I would argue for the more expliciti form of the API, but hostnames has to be plural, and IP has to be the mergeKey. or - any objections? Consider from a CRI point of view?
For me, this solves my problems.
so the struct would be something like where the keys are IP addresses, and values are hostnames. I presume this field would live under PodSpec. is the key we want to surface? ? the content would be: which after some merging would look like: (the go struct represented in yaml) and the output could either look like: or I think the latter is better since it's one line per entry, and seems to follow existing convention in .
Your proposal can definitely work for my use-case, although I would love if I could configure it from a init container, but this could work. Keep up the good work :)
I wouldn't try to massage it into a map, just: , Kristian Klausen
So you'd be passing in into . That makes sense. This might be a stretch, but what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP.
This is currently handled completely by kubelet, so it wouldn't be an issue for CRI. kubelet doesn't mount if the pod is using the host network, so this would either need to be changed, or validation should reject the pod if this is not allowed. If you need to dynamically change the file after the container starts, I think suggestion of trying to make ConfigMap work would be better.
I think validation to reject the option if the pod is using the host network makes sense. is still a static file though. And if you inject a file that replaces , what are you going to do with existing dynamic hostnames that are by kubelet?
I don't have any concern from CRI perspective, and the proposal look fine to me from the high level. what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP. But can I request not stretching the feature to support dynamic changes? Pod can be deleted, and re-created, thus the pod's IP might be changed. Currently we only allow container image being updated in PodSpec. Unless we allow updating the new field, let's call it etcHost for now, in PodSpec, it won't work. Even we allow that field being updated, I think that is overkill.
With regards to the dynamicness, I'm just trying to design the API in such a way that's flexible. The field in PodSpec won't be dynamic. I imagine something like this: (The idea is borrowed from ) validation ensures that is only present if type is . then IP is gathered at Pod creation time. In kubelet's :
Why do you need pod IP? Why not use 127.0.0.1 ? , Ricky Pai <:
The use-case that i'm more interested in is support for nodeIP, but I'm using podIP as an example because it's easier to illustrate with current code. But yes I agree with you that pod IP might not make sense since you have access to localhost already. I can imagine nodeIP being used to talk to daemons that are shared across pods. is a proper fix for the use-case, but this could be one way to address it.
sorry for the scope-creep. I'm just afraid using IP as a string field makes it hard to add support for other IP sources should that ever be an option. but it's probably premature right now. for now, I'll just proceed with:
Given that we have a way to get node IP through downward API, why do we need that here? On Apr 13, 2017 6:09 PM, "Ricky Pai" <:
Ideally, if the applications running inside the containers have flexible ways to configure dependent hostnames like through environment variables, then downward API would be sufficient. We can probably extend the argument that the whole support for is moot if applications supported flexible injection of dependent hostnames. The problem is many applications don't provide this level of flexibility. Hostnames are often hard-coded inside config files and the client libraries. Outside of Kubernetes, I might use a script to manage and expect nothing else to change the file; while it is possible that I use an init container to do the same modification to , since the mounting of happens whenever a container is restarted, I risk the file being changed and my application not being able to connect to its dependent services. This creates much headache migrating applications from a more lenient environment to Kubernetes, which is what we've been facing. I think support is the right direction and is why I'm willing to commit time to make it happen.
We have another issue open for "node-local Services" which sounds like what you want. I am not a big fan of the expansion of host mappings like this. I'd rather go all the way and give nodes a static link-local address or else solve node-local services. , Ricky Pai <:
Thanks for all the feedback I'm well aware of node-local services, but it seems like progress on the proposal is stalled. I believe node-local services would be the proper way to address the use case I described, especially with the "locality" concept you described in that can be expanded beyond the node. However, in lieu of node-local services, with references to the node IP might be a quick way to address the use case. Other tickets describing the need for node-level sidecars also proposed either using node-local services or : I also understand this complicates the interface a little bit more, but it would really benefit cluster operators. If you still think this is inappropriate, then I can proceed with this interface next week:
I don't have a strong opinion on this topic but it seems that it should be addressed as part of what is the status of the proposal?
I am not familiar with the node-local services and its progress, but if this can satisfy the use cases, I'd suggest that not adding new fields to the pod API for a quick workaround.
I feel the original proposal got digressed by my zealous talk of node IP support. Allow me to clarify: serves several use cases: external hostname with a specific IP. i.e. if I want to say mock out an external hostname like a hostname with localhost. i.e. if I want to override to point to we allow dynamic injection of nodeIPs, it can be an easy way for pods to talk to nodes in absence of node-local services. only the third use-case is relevant to node-local services. I think it's still useful to support adding entries to even with static IPs
I understand the third case, but I'd like to consider other options to solve that, first. I think the rest is uncontentious. , Ricky Pai <:
Thanks. I'll get the changes up this week. There'll likely be three changes: tests for existing hosts file content (there's no tests currently) change for on to use in Is there any possibility to cherry-pick those back in 1.6? Or would this have to wait for 1.7?
AFAIK, we don't cherrypick features.
has been created for the API change
just to clarify, are you in favor of adding HostMappings to the API?
I am neutral on it. It's come up often enough that i believe the problem is real. The API doesn't seem harmful to me, what do you think? You're the guardian of CRI... , Yu-Ju Hong <:
Also the name is terrible - help? HostAliases? DNSAliases? , Tim Hockin <: solve that, first. I think the rest is uncontentious. , just to clarify, are you in favor of adding HostMappings to the API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I'm up for . might accidentally imply that it's actually interfacing with kube-dns.
I am neutral but just wanted to make sure there are no obvious, alternative ways to achieve this. BTW, the proposed solution changes the k8s API, but not CRI :) On the other hand, if we support this on the k8s API level, we should discourage users from modifying this file directly. For example, reported a bug where users modify in the init containers, and later found that kubelet overwrites it.
We should definitely discourage modifying that, but we should perhaps respect it if a user provided a volume mount. Thoughts on name? , Yu-Ju Hong <:
I am terrible at naming... I agree that having DNS in the name can be misleading. sounds ok. might be more direct?
I'd go for brevity when possible. HostAliases ? On Apr 27, 2017 5:27 PM, "Yu-Ju Hong" <:
I'm updating the PR to use . thanks for the suggestion
Just posted the last PR to enable this feature:
could you add a node e2e test for this feature?
Do we need e2e? Every e2e is a liability, and slows down test runs - e2e is needed if we can't cover in unit or integration. , Yu-Ju Hong <:
Unit tests are great to ensure the individual functions work correctly, but so far it's not sufficient to prevent regression in our codebase from my experiences. Since this is a new addition/feature to the API, I'd prefer having a corresponding node e2e test. Integration test can be sufficient, but I've not had any pleasant experience debugging integration tests. Maybe that has changed already?
no, integrtaion tests are under-serviced tech-wise. , Yu-Ju Hong <:
does a simple node e2e test sound good to you?
SGTM , Yu-Ju Hong <:
I saw that the HostAlias's comment: I'm wondering why it couldn't work to hostNetwork pods? And how does a hostNework pod initialize its /etc/hosts file?
Are we still missing a test for this? I'd hate to have to kick it out of the 1.7 release for lack of tests..
yes we are still missing a test. what's the deadline for 1.7 release? sorry i was on PTO and dropped the ball
hostNetwork pods uses the host's file, while only non-hostNetwork pods have managed by kubelet. because kubelet does not manage the file when using hostNetwork, kubelet can't write additional entries from hostAliases.
Just chatted on slack, for containers using the host network, docker actually makes a copy of "/etc/hosts" on the host for each of them. I think kubelet can potentially support HostAliases for hostnetwork pods by copying and mounting the file. Not sure if there are be other concerns though.
if you make the change to make kublet copy and manage for hostNetwork pods, I can make the change for HostAliases. one concern: currently, the logic includes , including from and . Should this be allowed? if not, there should probably be validation disallowing those field when hostNetwork is used. there currently isn't any unless I read the code wrong.
node e2e test:
THANKS! , Ricky Pai <:
That would be a great feature! Currently we have pods using hostNetwork and using init containers to restore some host mappings -- basically . With HostAliases feature it would be fantastic fit!
We are using and for hostNetwork pods.
This is something I don't understand in the code: It seems that it is still mounting "/etc/hosts" for hostnetwork pods if . Does that mean kubelet is still managing it?
One of our current assumptions is that kubelet will mount a shared /etc/hosts (which is not node's) for all containers within a pod in hostNetwork mode. Which seems to be the codepath in the code linked above. Is this holding true?
I checked the git history and that was just an artifact caused by switching from internal to v1 API objects. The original code is: In the internal API, is a field in , so you can see why had to be non-nil. In the v1 API, is a field in the pod spec: . When switching all code to use the versioned (v1) objects, this got lost in translation. I believe the apiserver sets the default pod SecurityContext, so it should never be nil. I'll send a PR to remove the redundant check.
Thanks for looking into that. Currently even with hostNetwork pods, it will have mounts for . Checking within the pod: If the check is removed, what would happen on the shared mount of /etc/hosts ? My concern and question is about the guarantee here. Is there still a copy of which is shared and mounted for all containers within a pod?
Removing the check itself won't affect anything since securitycontext should always be non-nil. That mount for hostnetowrk pods was by docker (not by kubelet) so it should be one per container, and not shared by containers within a pod. I didn't try it myself, but it should be easy to verify.
Tried and verified that was shared between containers within a pod for k8s 1.6.4 .
Also verified that modifying for a hostNetwork pod did not change the file on the node.
Then docker did the right thing and use the same file for containers sharing the same network namespace :-)
That makes sense.
Will HostAlias support for HostNetwork pod happen in 1.7 ?
I'm trying to get in so it can make the 1.7 release deadline. help reviewing the PR would be great.
in a multi-container hostNetwork pod, what happens when you change the file, then restart one of the containers? in non-hostNetwork pod, it would revert the changes to
anyways, i'm also in favor of kubelet managing the file even for hostNetwork pods. that way, we can use features like hostAliases, and also then we don't depend on the container runtime to dictate the behavior of the hosts file. it's not clear whether running rktnetes with hostNetwork pod would yield the same result
Is this a dup of ? How are they different?
HostAlias is useful to provide local overrides for a particular Pod, but it seems like would want the alternative DNS name of a service accessible for all Pods.
re: documentation, i'm thinking either adding notes about HostAliases in or as a separate page under "Services, Load Balancing, and Networking"
Is there any issue created for supporting HostAlias in hostnetwork mode?
I can write up what I think will be needed to support HostAlias in hostNetwork pods
This is now in 1.7:
filed issues to follow up with HostNetwork support:
Thank you for this feature. Our use case is different. On an nginx/php-fpm set up where our local dev is via docker-compose yet we're deploying into kubernetes, we can use the exact same nginx config on both environments. We're currently working around this via HOSTALIASES (alias or or whatever the php container is into , which satisfies both deployment types), but this is hacky and doesn't work in Alpine.
the discrepancy between dev/prod would probably cause problems beyond just difference in networking setup. have you considered using to have parity between the two environments?
We have, and found it way too cumbersome for local development. It's way heavier in resources as well - obviously there's a lot more than just docker needed for minikube to run, and most of our developers use Mac, where all this is virtualised. In contrast, docker-compose is a lot easier, more concise, and we have yet to find any environment-y issues caused by difference between these two, except for this one issue we fortunately have a workaround for (HOSTALIASES). We use the exact same base container images for development as well as builds. From the point of view of the services themselves, there's no difference at all being run on one environment or the other - they talk to each other via HTTP or queues.
that's unfortunate. I guess minikube does have overhead since you have to run it inside a VM on your mac machines. But since all components are written in go and cross-compiled in darwin, I wonder if you could setup your own etcd (homebrew has this), then your own kubelet. Docker you can run Docker on Mac, then just have kubelet talk to the local docker socket. This sounds like it would work with lower resource overhead than minikube since you don't to run in a VM. Either way, I'm glad HostAlias is able to address your problem. I just wish you didn't have to deal with this discrepancy in the first place.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close

Discussion:
}) } if mountEtcHostsFile { <del> hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain) </del> <ins> hostAliases := pod.Spec.HostAliases hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain, hostAliases) </ins> if err != nil { return nil, err }

// makeHostsMount makes the mountpoint for the hosts file that the containers // in a pod are injected with. <del> func makeHostsMount(podDir, podIP, hostName, hostDomainName string) (*kubecontainer.Mount, error) { </del> <ins> func makeHostsMount(podDir, podIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) (*kubecontainer.Mount, error) { </ins> hostsFilePath := path.Join(podDir, "etc-hosts") <del> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName); err != nil { </del> <ins> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName, hostAliases); err != nil { </ins> return nil, err } return &kubecontainer.Mount{

// ensureHostsFile ensures that the given host file has an up-to-date ip, host // name, and domain name. <del> func ensureHostsFile(fileName, hostIP, hostName, hostDomainName string) error { </del> <ins> func ensureHostsFile(fileName, hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) error { </ins> if _, err := os.Stat(fileName); os.IsExist(err) { glog.V(4).Infof("kubernetes-managed etc-hosts file exits. Will not be recreated: %q", fileName) return nil } <del> content := hostsFileContent(hostIP, hostName, hostDomainName) </del> <ins> content := hostsFileContent(hostIP, hostName, hostDomainName, hostAliases) </ins> return ioutil.WriteFile(fileName, content, 0644) } // hostsFileContent is the content of the managed etc hosts <del> func hostsFileContent(hostIP, hostName, hostDomainName string) []byte { </del> <ins> func hostsFileContent(hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) []byte { </ins> var buffer bytes.Buffer buffer.WriteString("# Kubernetes-managed hosts file.n") buffer.WriteString("127.0.0.1tlocalhostn")                      // ipv4 localhost

} else { buffer.WriteString(fmt.Sprintf("%st%sn", hostIP, hostName)) } <ins> // write each IP/hostname pair as an entry into hosts file for _, hostAlias := range hostAliases { for _, hostname := range hostAlias.Hostnames { buffer.WriteString(fmt.Sprintf("%st%sn", hostAlias.IP, hostname)) } } </ins> return buffer.Bytes() }

hostIP          string hostName        string hostDomainName  string <ins> hostAliases     []v1.HostAlias </ins> expectedContent string }{ { "123.45.67.89", "podFoo", "", <ins> []v1.HostAlias{}, </ins> `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback

"203.0.113.1", "podFoo", "domainFoo", <ins> []v1.HostAlias{}, </ins> `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback

203.0.113.1	podFoo.domainFoo	podFoo `, }, <ins> { "203.0.113.1", "podFoo", "domainFoo", []v1.HostAlias{ {IP: "123.45.67.89", Hostnames: []string{"foo", "bar", "baz"}}, }, `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback fe00::0	ip6-localnet fe00::0	ip6-mcastprefix fe00::1	ip6-allnodes fe00::2	ip6-allrouters 203.0.113.1	podFoo.domainFoo	podFoo 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz `, }, { "203.0.113.1", "podFoo", "domainFoo", []v1.HostAlias{ {IP: "123.45.67.89", Hostnames: []string{"foo", "bar", "baz"}}, {IP: "456.78.90.123", Hostnames: []string{"park", "doo", "boo"}}, }, `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback fe00::0	ip6-localnet fe00::0	ip6-mcastprefix fe00::1	ip6-allnodes fe00::2	ip6-allrouters 203.0.113.1	podFoo.domainFoo	podFoo 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz 456.78.90.123	park 456.78.90.123	doo 456.78.90.123	boo `, }, </ins> } for _, testCase := range testCases { <del> actualContent := string(hostsFileContent(testCase.hostIP, testCase.hostName, testCase.hostDomainName)) </del> <ins> actualContent := string(hostsFileContent(testCase.hostIP, testCase.hostName, testCase.hostDomainName, testCase.hostAliases)) </ins> assert.Equal(t, testCase.expectedContent, actualContent, "hosts file content not expected") } }