Change:
See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case. However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.

Discussion:
pod.Spec.NodeName = nodeName pod.ObjectMeta.SelfLink = getSelfLink(pod.Name, pod.Namespace) <ins> if pod.Annotations == nil { pod.Annotations = make(map[string]string) } // The generated UID is the hash of the file. pod.Annotations[kubetypes.ConfigHashAnnotationKey] = string(pod.UID) </ins> return nil }

}, expected: CreatePodUpdate(kubetypes.SET, kubetypes.FileSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> Name:      "test-" + hostname, UID:       "12345", Namespace: "mynamespace", SelfLink:  getSelfLink("test-"+hostname, "mynamespace"), </del> <ins> Name:        "test-" + hostname, UID:         "12345", Namespace:   "mynamespace", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "12345"}, SelfLink:    getSelfLink("test-"+hostname, "mynamespace"), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "mynamespace", SelfLink: getSelfLink("foo-"+hostname, "mynamespace"), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "mynamespace", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, "mynamespace"), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

}, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "222", Name:      "bar" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "222", Name:        "bar" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "222"}, SelfLink:    getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

if pod.Name != mirrorPod.Name || pod.Namespace != mirrorPod.Namespace { return false } <del> return api.Semantic.DeepEqual(&pod.Spec, &mirrorPod.Spec) </del> <ins> hash, ok := getHashFromMirrorPod(mirrorPod) if !ok { return false } return hash == getPodHash(pod) </ins> } func podsMapToPods(UIDMap map[types.UID]*api.Pod) []*api.Pod {

for k, v := range pod.Annotations { copyPod.Annotations[k] = v } <del> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = kubetypes.MirrorType </del> <ins> copyPod.Annotations[kubetypes.ConfigMirrorAnnotationKey] = getPodHash(pod) </ins> _, err := mc.apiserverClient.Pods(copyPod.Namespace).Create(&copyPod) return err

} func IsMirrorPod(pod *api.Pod) bool { <del> if value, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey]; !ok { return false } else { return value == kubetypes.MirrorType } </del> <ins> _, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return ok } func getHashFromMirrorPod(pod *api.Pod) (string, bool) { hash, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return hash, ok } func getPodHash(pod *api.Pod) string { // The annotation exists for all static pods. return pod.Annotations[kubetypes.ConfigHashAnnotationKey] </ins> }

const ConfigSourceAnnotationKey = "kubernetes.io/config.source" const ConfigMirrorAnnotationKey = "kubernetes.io/config.mirror" const ConfigFirstSeenAnnotationKey = "kubernetes.io/config.seen" <ins> const ConfigHashAnnotationKey = "kubernetes.io/config.hash" </ins> // PodOperation defines what changes will be made on a pod configuration. type PodOperation int

// Updates from all sources AllSource = "*" <del> // Used for ConfigMirrorAnnotationKey. MirrorType = "mirror" </del> NamespaceDefault = api.NamespaceDefault )