Change:
What happened: The is taken from without regard for the main family of the cluster. No warning or error is given if gets the wrong family. What you expected to happen: That is taken from the "main" ip family, or at the very least the POD is not started and an error is given if it's not. How to reproduce it (as minimally and precisely as possible): Use the CNI-plugin and ipam. Use a single node cluster to avoid that assign the same address to PODs in different nodes. This may be CRI-plugin dependent, I use . The PODs are assigned dual addresses which has been supported since v1.9. Note the order; IPv6 first. In a dual-stack or a single-stack cluster with main family IPv4 will get an IPv6 address even if the main family is IPv4; Anything else we need to know?: First, note that this is not a dual-stack problem. may get the wrong family in a single-stack cluster if the PODs has dual addresses. The bug may however have been introduced with the dual-stack support, before K8s may have selected the correct family, I have not checked that. This is reported and discussed in There it is considered as a CNI-plugin problem, the CNI-plugin must present addresses in the order K8s wants them. Problem is that CNI-plugins doesn't know this. The addresses takes the path; The problem can be addressed in either of these places. The current situation is that the full responsibility lies on the , it must send the addresses in the "correct order". This is not the best place. The CNI-plugins are not a K8s-only thing and to require that a "prefered-family" or something similar must be supported by CNI-plugins to be "Kubernetes compliant" should be avoided. The CRI-plugin is a K8s-only thing and a "prefered-family" can be introduced so the CRI-plugin can sort the address array before sending it to . This is however undesirable since it adds a configuration complexity, it put a responsibility on the user or installation tool to configure the "prefered-family" to all CNI-plugins now and in the future. The best is to handle this in K8s itself where the "main" family is known. Is this really necessary? All communication works even if is not of the main family. EndpointSlices seem to get this right regardless of order. Endpoint on the other hand seem to take the (old) podIP. So perhaps; is sufficient? If really must be , then K8s should sort the array so belongs to the main family. The problem exist on all supported K8s versions and on "master". If installed in the default way the situation is; Calico always send IPv4 first Cilum always send IPv6 first I don't think you can control the order, but I have not asked. Environment: Kubernetes versions: v1.17.12, v1.18.9, v1.19.2, v1.20.0-alpha.1, master v1.20.0-alpha.1.257+ Cloud provider or hardware configuration: None OS (e.g: ): xcluster Kernel: linux-5.8.1 Install tools: None Network plugin and version: bridge, host-local v0.8.7 Others: CRI-plugin: cri-o 1.18.3
/sig network
For the record; all communication via services works perfectly fine even if the order in is "faulty" with not belonging to the main family in k8s v1.18.8.
I think this is a duplicate of CNI plugins must report multiple IPs in the order expected by the cluster. The podIPs[0] == podIP requirement is for backwards compatibility with clients that do not read podIPs. Server-side reordering causes problems with clients that continually write/reconcile until the server list exactly matches their local list.
The problem exist in single-stack K8s clusters when PODs has dual addresses (supported since K8s v1.9, i.e way before K8s dual-stack which came in v1.16) And this issue is not a duplicate of . That issue is about a problem when the CNI-plugin uses the "correct" order. This issue is about the fact that K8s puts undocumented requirements on the CNI-plugins. There is no requirement on order in the . And since this requirement is undocumented it is not supported for instance by cilium and calico. The itself should be the requirement is for backwards compatibility with clients that do not read podIPs. Since they doesn't read podIPs the value of should not matter to them, or? I think this should be handled in K8s, but if the responsibility is pushed downhill to the CNI-plugins then two things must be done in K8s; extra requirement on "Kubernetes compliant" CNI-plugins (outside the CNI spec) must be formally documented and communicated to the CNI-plugin providers. a "faulty" CNI-plugin accidently put's an IP-address of the "wrong" family in the first item then K8s must report an error and not start the POD. Please note again that this is not a K8s dual-stack problem. In a single-stack IPv6 K8s cluster the CNI-plugins must provide an IPv6 address first in the list.
The reason podIP must match podIPs[0] is related to updates to an existing object by an old client. See for a related problem when this was not done. An old client updated an existing object, setting only the podIP field (which is the only one it was aware of). The API server needed to detect that and populate the podIPs field correctly for new clients to read. The detection is based on the mismatch between podIP and podIPs[0], and the server responds by populating . That means that new clients (aware of both fields) must set podIP and podIPs[0] to match.
So this is weird, but... dual-stack clusters do not actually have a "main family", because decisions about IP families are made by multiple components, and kubernetes clusters do not have a single source of configuration truth that applies to all components. So it is explicitly allowed to have kube-apiserver default to IPv4-primary IPv6-secondary for services, and have your network plugin default to IPv6-primary IPv4-secondary for pod IPs, and to have some nodes have a single IPv4 address and some have a single IPv6 address and some have dual-stack addresses. It's not super clear why you'd want to do that, but we decided we had to allow it, simply because there's no easy way to prevent it. Of course in scenarios where the entire cluster is being installed/deployed by a single tool, one would expect that tool to configure all of the components in a consistent way. There is no "order K8s wants them". Kubernetes doesn't want the addresses in any order. You want them in a particular order. Therefore you must configure the CNI plugin to return them in that order. Eg, in the example above, you should reverse the two in the config. Kubernetes's interpretation of this is that if Cilium sends the IPv6 address first, then it is telling Kubernetes that Kubernetes ought to prefer the IPv6 one. It appears that the CNI spec doesn't actually say that though... It doesn't seem to say anything about the ordering of multiple IPs... So maybe that should be clarified. If the CNI authors feel that the ordering of returned addresses is not meaningful then perhaps kubelet's behavior could be changed.
Thoughts on this?
CNI doesn't make any statement about ordering of IPs. It definitely doesn't have any notion of a "primary IP". It seems that containerd itself does have some
I mean that's literally what I just said :stuckouttongue:. The question is more if that is intentional and/or changeable. It seems strange that the decision about whether to allocate IPv4 and/or IPv6 IPs is left up to the network plugin, with no input from the runtime, but the decision about what to do with IPv4 and/or IPv6 IPs is then left up to the runtime, with no input from the network plugin.
That is very good an as it should be but... The quote is from the initial commant. Please note that this is a single-stack cluster! And that has a main family even though the PODs have dual addresses.
Right; don't comment on issues just before going to bed. Ooops. I don't think we want to introduce the notion of IP ordering / or preference in to CNI for a few reasons -- It's a bit of a breaking change, depending on how you read the spec It would be awkward to push the "preferred family" from k8s all the way down through to CNI It doesn't match how Linux works - interfaces don't have primary IPs (legacy aside). So fixing this in the apisever is probably the right place? BTW, what, exactly, defines a cluster's address family? ClusterCIDR, but that's optional (as is PodCIDR). ServiceCIDR, certainly. Is there an explicit config flag?
no flag, you configure the cluster with IPv4 or IPv6 addresses, of course if you have pod with IPv4 addresses and service with IPv6 that will not work kubelet set the status.PodIPs In a single-stack cluster you don't have a main family, you only have ONE family If we don't have PodIPs and only PodIP in a single-stack cluster this will be clearly a configuration problem in the CNI,no? I personally, think that this is a configuration problem and that we created it because we pluralised PodIP without a feature gate
Yes, and this is how it looks in an IPv4 single-stack cluster ( NOT set) if the CNI-plugin has been configured for dual addresses on PODs but returns the assigned addresses in the "wrong" order (ipv6 first);
Why is requesting an IPv6 address for pods in an IPv4 cluster not a configuration error? To me it seems a bit like running an ARM image on an AMD64 machine; Kubernetes will go through some motions but will not make it work or tell you what you did wrong.
Dual addresses in PODs has been supported from K8s v1.9, i.e. long before K8s dual-stack so that is not a configuration error. In the CNI specification there is nothing about required order for addresses or "prefered family" so it is not a configuration error for the CNI-plugin either. For K8s functions the order in does not matter. But design of upgrade requires that see Problem is that K8s (kubelet) sets "podIP = podIPs[0]" without checking the family. Here some error should be in place, like "CNI-plugin not compatible with K8s" and the POD should not be started, or better, sort the addresses or remove the requirement. This is a K8s problem. CNI-plugins should not have to care about order of reported addresses at least not without a CNI specification update.
This doesn't appear to answer my question. What is the meaning of an IPv6 pod address in an IPv4 cluster?
A cluster does not have an address family. Usually it seems like it does but that's just because the administrator has configured all of the individual components in a consistent way. But most components' configuration only affects what that component does (though in some cases that will have knock-on effects). kube-controller-manager sets based on its (assuming is set) The network plugin (indirectly) sets (which may be affected by kcm's choice of ) based on its own configuration Either kubelet or an external cloud provider sets : based on , if passed, in the non-cloud-provider case based on cloud-provider-specific configuration (or hardcoded defaults) in the cloud case kubelet sets from kube-apiserver sets based on its kube-controller-manager creates endpoints, mostly based on the IP families chosen by kube-apiserver, except in the case of headless selectorless services, where it lets the user set endpoints of whatever family it wants, because it doesn't usually know whether the cluster is single or dual stack. kube-proxy determines whether rules are actually created for IPv4 and/or IPv6 services, based on the passed in if the dual stack feature gate is disabled, and unconditionally dual stack if the feature gate is enabled the cloud provider determines whether services get IPv4 and/or IPv6 IPs, based on cloud-provider-specific config or hardcoded defaults. (EDIT: well, and probably ) This... is kind of a mess, but there's no real way to fix it centrally given the way that kubernetes works. But we could add a config option to kubelet to unambiguously specify whether it should act like it's in a SSIPv4/SSIPv6/DSIPv4Primary/DSIPv6Primary cluster, and then make assignment make use of that.
Oof. Well. Maybe it makes sense for kubelet to stop setting PodIP, and leave that decision to the apiserver? It could make that decision based on ServiceCIDR, and order the families accordingly?
No, the dual stack KEP even explicitly calls out the fact that that it doesn't work that way: There is no single source of truth for kubernetes cluster config, and thus there is no single truth about whether the cluster is IPv4, IPv6, dual-stack-IPv4-primary, or dual-stack-IPv6-primary. If we want kubelet to impose some policy on PodIPs, then we need to tell it what policy to impose, because it has no entirely-reliable way of determining what policies any other component is imposing.
That is still much better than to require this function from the CNI-plugin which is not even K8s and follows a specification.
Please note that the networking e.g services, external access, works fine whatever order are in and whatever value has. I have not found any communication problem in K8s = v1.19 caused by this. The value of seem more "cosmetic". But it looks wrong to see ipv6 addresses of PODs in an ipv4-only cluster (). The problem seem to be an unfortunate upgrade design that imposed the constraint If that can some way be relaxed this becomes a no-problem as I see it.
/triage accepted /assign
What is the material impact of this behavior?
we can not relax because that will break existing clients. a side note. Using dualstack configured CNI on a none dual stack cluster was pre dualstack hack, if it must remain in use then CNI should only report one IP not two.
what's your reasoning to change CNI rather than kubelet?
We can't just put undocumented requirements on the CNI-plugins, they are not part of K8s, they have a formal specification and they have other users beside K8s. At least K8s must formally document the extra requirements on CNI-plugins that K8s has for the CNI-plugin to be K8s compliant.
As highlighted on the discussion there is no single truth of what clusterIP family is. Also kubelet will have to depend on which is not always used by CNIs. You can have a perfectly functional cluster that does not use PodCIDRs at all (I believe Cilium for example does not use it). Also I am not saying change CNI. A well configured CNI (single family for single stack cluster) will not have this problem.
Is the latter statement a retraction of the former? If not, how can they be reconciled?
He's saying that an administrator deploying a CNI plugin in a single-stack kubernetes cluster ought to configure that CNI plugin to return a single IP address, of the family being used by the kubernetes cluster. A CNI plugin that does not allow you configure what address families it uses may not be compatible with all kubernetes clusters. Eg, a CNI plugin that only supports IPv4 cannot be used as the network plugin in a single-stack IPv6 kubernetes cluster.
So, to sum up: Kubelet will always use the first IP returned by the CNI plugin as , regardless of any other cluster configuration. Some CNI plugins, when configured to do dual stack, do not give you a way to configure which IP comes first, meaning kubelet may pick a that is not the IP family the admin considers the cluster's primary IP family In dual-stack clusters, this does not appear to cause any actual problems, other than that the "default" pod IPs shown in "" may not be the ones the administrator wants Nothing in the current kubelet configuration is supposed to indicate whether kubelet should prefer IPv4 or IPv6 pod IPs. It would be possible to add a new config option indicating this, or we could just declare that should also affect the sorting of pod IPs, because really why wouldn't you want that? In single-stack clusters, it has been suggested that the right answer is "don't do that then"; ie, the administrator should configure the CNI plugin to only return a single IP, and if they can't do that, then that's the CNI plugin's problem, not Kubernetes's
oh, and: if the CNI plugin returns multiple IPs which are not a dual-stack pair (eg, if it returns 2 IPv4 IPs, or 1 IPv4 IP and 2 IPv6 IPs), then kubelet will fail to update the pod at all, because it assumes it can just copy the IPs returned from CNI into , but the apiserver will only accept if it is either a single IP or a dual-stack pair.
That would be excellent! Then it would not be necessary to stop supporting dual-stack pods in a single stack K8s. And Cilium users wanting to upgrade their ipv4-only cluster to dual stack would not suddenly see ipv6 addresses on a .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Just for the record, this is an actual problem when trying to deploy rook-ceph. Rook makes strong assumption about family being the same as family and just fails to create Ceph cluster when they do not match. are configured to listen only on but the advertised addresses are the corresponding service . This result in listening only on pod IPv6 address but being advertised using the service IPv4 address.

Discussion:
"k8s.io/kubernetes/pkg/volume/util/volumepathhandler" volumevalidation "k8s.io/kubernetes/pkg/volume/validation" "k8s.io/kubernetes/third_party/forked/golang/expansion" <ins> utilnet "k8s.io/utils/net" </ins> ) const (

// alter the kubelet state at all. func (kl *Kubelet) convertStatusToAPIStatus(pod *v1.Pod, podStatus *kubecontainer.PodStatus) *v1.PodStatus { var apiPodStatus v1.PodStatus <del> apiPodStatus.PodIPs = make([]v1.PodIP, 0, len(podStatus.IPs)) </del> <ins> // The runtime pod status may have an arbitrary number of IPs, in an arbitrary // order. Pick out the first returned IP of the same IP family as the node IP // first, followed by the first IP of the opposite IP family (if any). podIPs := make([]v1.PodIP, 0, len(podStatus.IPs)) var validPrimaryIP, validSecondaryIP func(ip string) bool if len(kl.nodeIPs) == 0 || utilnet.IsIPv4(kl.nodeIPs[0]) { validPrimaryIP = utilnet.IsIPv4String validSecondaryIP = utilnet.IsIPv6String } else { validPrimaryIP = utilnet.IsIPv6String validSecondaryIP = utilnet.IsIPv4String } </ins> for _, ip := range podStatus.IPs { <del> apiPodStatus.PodIPs = append(apiPodStatus.PodIPs, v1.PodIP{ IP: ip, }) </del> <ins> if validPrimaryIP(ip) { podIPs = append(podIPs, v1.PodIP{IP: ip}) break } </ins> } <del> if len(apiPodStatus.PodIPs) > 0 { apiPodStatus.PodIP = apiPodStatus.PodIPs[0].IP </del> <ins> for _, ip := range podStatus.IPs { if validSecondaryIP(ip) { podIPs = append(podIPs, v1.PodIP{IP: ip}) break } </ins> } <ins> apiPodStatus.PodIPs = podIPs if len(podIPs) > 0 { apiPodStatus.PodIP = podIPs[0].IP } </ins> // set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod)

"errors" "fmt" "io/ioutil" <ins> "net" </ins> "os" "path/filepath" "reflect"

}) } } <ins> func TestGenerateAPIPodStatusPodIPs(t *testing.T) { testcases := []struct { name      string nodeIP    string criPodIPs []string podIPs    []v1.PodIP }{ { name:      "Simple", nodeIP:    "", criPodIPs: []string{"10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "Dual-stack", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with explicit node IP", nodeIP:    "192.168.1.1", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with CRI returning wrong family first", nodeIP:    "", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with explicit node IP with CRI returning wrong family first", nodeIP:    "192.168.1.1", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with IPv6 node IP", nodeIP:    "fd00::5678", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, {IP: "10.0.0.1"}, }, }, { name:      "Dual-stack with IPv6 node IP, other CRI order", nodeIP:    "fd00::5678", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, {IP: "10.0.0.1"}, }, }, { name:      "No Pod IP matching Node IP", nodeIP:    "fd00::5678", criPodIPs: []string{"10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "No Pod IP matching (unspecified) Node IP", nodeIP:    "", criPodIPs: []string{"fd01::1234"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, }, }, { name:      "Multiple IPv4 IPs", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "10.0.0.2", "10.0.0.3"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "Multiple Dual-Stack IPs", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "10.0.0.2", "fd01::1234", "10.0.0.3", "fd01::5678"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kl := testKubelet.kubelet if tc.nodeIP != "" { kl.nodeIPs = []net.IP{net.ParseIP(tc.nodeIP)} } pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, IPs:       tc.criPodIPs, } status := kl.generateAPIPodStatus(pod, criStatus) if !reflect.DeepEqual(status.PodIPs, tc.podIPs) { t.Fatalf("Expected PodIPs %#v, got %#v", tc.podIPs, status.PodIPs) } if status.PodIP != status.PodIPs[0].IP { t.Fatalf("Expected PodIP %q to equal PodIPs[0].IP %q", status.PodIP, status.PodIPs[0].IP) } }) } } </ins>