Change:
Nodes can't recognize the static pods that they create possibly because the defaulting is different. always returns false. Here is a json diff of two pod specs before and after they are Semenatic.DeepEqual'd where before is created pod and after is what kubelet things the created pod should be. ! Thus: p0 for triage
I think the solution is to not apply defaults to mirror pods in the API Server.
Yes,we have to add specific code to API Server to detect the mirror pod, then not apply defaults.
The right fix should be , which we didn't get to before v1.0. I will start on this and at least get it in for v1.1 to prevent future complications. Since we cannot update the node/kubelet, options are: suggests to disable applying defaults to mirror pods as a temporary fix. IIRC, was against treating mirror pod differently at the apiserver. Since this is only a workaround, perhaps we can relax the constraint? a script to update the pod manifest files.
Disabling defaults has a number of problems: are version-specific. Conversion may fail if defaults are not applied. may fail if defaults are not applied. using defaulted fields assumes default values have been applied. defaulting pass is not the only place where values can be set, as mentioned in . For instance, if the user were to change the kube-system LimitRange, that would cause the same problem. This is another case where would have been useful. Which gives me another idea. For mirror pods, we could capture the entire metadata+spec provided by the user as an annotation, as we do in the client for kubectl apply, and then we could return that in GET. PUT may not work. We'd have to resolve the size concerns raised by . How serious of a problem is this delete-recreate loop?
Ok, we are going with approach proposed in . I am closing this issue as dup of
"entire metadata+spec" in plain text? It'd be hard for kubelet to diff using this. It'd be better to let kubelet store hash of its own choosing so that it can easily compare when there is any change.
Even with , we can't fix v1.0 kubelet.
I agree with
Yes, I were confused. The comparison is in Kubelet, and we cannot patch 1.0 kubelet in this case.
There are no serious issues with this bug: static pods are running properly, but corresponding mirror pods coming and gone. cc/ Can we treat this as part of GKE cluster upgrade process? Can we upgrade GKE master nodes, then ask the users to upgrade their fluentd yaml file in their cluster? For OSS kubernetes users, most likely they won't run into this issue. But we should document this as known issue for 1.1.
Or can we update the fluentd yaml files when updating the master?
cc/
This is a cc issue, really. But the fast answer is no, because "then ask the users to upgrade their fluentd yaml file in their cluster?" Historically our users are extremely slow to do maintenance.
Our current plan for this issue is just document as a known issue for 1.1 release.
In a 100 node cluster this is going to cause +100 writes +100 deletes per second. Is that extra load a concern?
This would be +100 writes +100 deletes every 10 seconds in a 100-node cluster. We are still discussing options to let user update their manifest files.
Ok, from the logs in the issue description, it looked more frequent but most of the deletes are failing with key not found
's easy to trigger a sync for all pods in kubelet. Any change in any pod would trigger a sync. In this case, the DELETE/ADD of the mirror pod could also trigger another sync. So, yes, it could be worse than 10s... We are looking at patching this in the apiserver, based on suggestion.
Yeah, my suggestion is fixing defaulting/conversion/validation as needed to make exceptions for mirror pods.
I'm not 100% sure it's needed, but if necessary we can turn the defaulting fix on in apiserver, and leave it off in scheduler, so that it gets the defaults it expects.
Yeah, it is way more frequent than 10s. I pointed this out to earlier. We had a meeting with and looks like: Providing a scripts to convert fluentd yaml file and asking the user to run is not a solution. Our users are very slow on such maintenance work. Piggyback such conversion work on master updating might not a solution for GKE clusters. We do have access to GKE nodes, and we don't do that. To obtain the permission to run script and sync a file requires more work from PM side. This is a API backward compatibility issue anyway. Given today's version management policy, we should handle such compatibility issue. I checked our default code: request diff was introduced by TerminationGracePeriodSeconds was introduced along with graceful termination support. There is no conversion for both fields There is no validation checking both fields today I think it is safe to disable such defaulting for mirror pods shouldn't cause any issue for 1.1 release. Sending PR shortly.
In order to not break the v1.1+ kubelet (which applies newer defaults), we should only skip defaulting for pods from pre-v1.1 kubelets. An easy way to do this is to reuse , which changes the mirror key annotation value from "mirror" to the hash of the pod manifest file. In short, if the key exists, it's a mirror pod. If the value is "mirror", it is sent by a pre-v1.1 kubelet. Otherwise, it's sent by a v1.1+ kubelet. Note that we will also need to maintain the special defaulting case for the future releases until we drop support for v1.0 nodes.
Just this morning, we discussed providing a script/tool to update static pod configs for people with large clusters. Why have we changed that plan? Any hack in apiserver is going to be confusing and fragile. For instance, a LimitRange installed for kube-system would defeat the hack. If we were to add a hack in apiserver, we'd need to remove it in 1.2.
we talked to and the conclusion was that node update may not be feasible as . I agree that any apiserver fix is hacky and fragile, but it seems the most viable option right now. We would need to carry this fix in apiserver as long as GKE supports v1.N master + v1.0 node, unless we can make updating the configs part of the cluster upgrade process.
Ah. These fields are problems because they existed in 1.0, but weren't used nor populated. Entirely new fields are silently dropped during decoding. We should cherrypick the solution into 1.0.7, since it appears there will be one.
That is, we should cherrypick into the 1.0 branch.
Ref.
If the hack is going to remain indefinitely, I'm not comfortable circumventing defaulting, conversion, and validation.
Just not performing the defaulting for these specific fields on just mirror pods seems safer.
In particular, I could easily imagine a new version of the pod API in 1.2 or 1.3, which will require defaulting and conversion to be applied, in general.
We'll be discussing this soon after 1.1 lands, I expect. My proposal would be for v1.N masters to support v1.N-1 and v1.N-2 (and maybe v1.N-3) nodes. That gives users 6-9 months to lag their nodes behind their masters. But it warrants much further discussion. This isn't a viable solution either, because we can't depend on users updating their nodes to the latest 1.0 rev. We expect arbitrary 1.0 nodes to work, including 1.0.6 all the way back to 1.0.1, per our current .
As Ike noted, cherrypicking into 1.0 kubelets is not feasible for solving this, because the whole reason for the problem is that people aren't updating their nodes. The conversion fix in is much preferable to the defaulting fix: it's less brittle, and components that need the defaulting (e.g. scheduler) will see the defaulting (because they run the conversion stack themselves), but on the wire the mirror pods from old kubelets will remain undefaulted.
I didn't say cherrypicking into 1.0 would solve the problem for everything. The behavior in 1.0 is pretty broken, even when not considering upgrade. We have a fix. We should cherrypick it into 1.0.
I don't see how different components would see different values for these fields, assuming they are all updated to the new code. As discussed offline, the scheduler and node controller need to handle cases of empty requests and pod termination for 1.0 kubelets, anyway, so I think the conversion approach is fine. Please put the explanation in code comments, though, not just in issue/PR comments.
Since the conversion stack does the defaulting at the end and runs in every component, and scheduler, node-controller, etc all consume api.Pod objects (instead of v1.Pod objects), they will have a view of the object with the fields defaulted. The conversion based fix essentially retains the defaulting in *api.Pod objects, but undoes it in any serialized representation of the object. Over the wire the defaulting will not be baked in to the object. Older clients that have not been recompiled (including these kubelets) would see a version of the object without the fields being defaulted.
Defaulting is run first:
Oh, you're right, I somehow had that backwards in my head. Sorry, everyone. I think that tilts me back in favor of . This means that defaulting is part of the versioned structs and therefore changing the defaulting is an API breaking change, which we claimed we wouldn't do. Can we change our minds about the defaulting? How about moving this particular defaulting into conversion functions, which is the intended way of doing something like this?
In , we only reset the defaults when converting api.Pod to v1.Pod. Shouldn't all components still see the defaulting values? Agreed that this is an API breaking change. Perhaps we should not change the defaults at all? mentioned in that scheduler already deals with empty requests. It's true in , but seems untrue in . Maybe this should be fixed. As for , we can also ask kubelet to apply a default value if we truly on to have the 30 second dfault.
resources.requests and terminationGracePeriodSeconds are effectively newly fields, so setting defaults is not a breaking change for most clients. If Kubelet were comparing the pre-decoded pods returned by apisever, it would barf on every new field we've . It's an accident of the implementation that it broke on exactly these fields.
Actually, neither priorities nor predicates takes values from limits any longer. That code is rely on the defaulting pass to copy limits to requests in the case that the requests field is absent. We would need to change that, yes, if we were to not set requests for the 1.0 mirror pods.
This Kubelet behavior was known to be broken back in May: What we're trying to do now is figure out a workaround for this broken behavior. My recommended solution is: not set resources.requests and terminationGracePeriodSeconds to non-nil values for just 1.0 mirror pods. All other fields and all other pods, including 1.1 mirror pods, should have default values set normally. Doing this in the defaulting pass makes the most sense, assuming that it's straightforward to do given the structure of that pass. the scheduler fit and priority functions to use limits for pods where requests are entirely absent. that the new code in the node controller to handle termination of pods on 1.0 kubelets deals with unset terminationGracePeriodSeconds appropriately. a kind/workaround-removal issue to remove these hacks once we don't need to worry about 1.0 kubelets.
Just for comparison, an alternative: this default setting from the defaulting function. it to the conversion function instead. mirror pods from 1.0 kubelets (MPF1K), store the original value in an annotation. MPF1K, when converting to v1, restore the original value & remove the annotation. No changes needed anywhere else in the system, maintenance cost is very low, we can carry this as long as needed.
I don't see how that keeps the default values off the wire.
Actually, I think I see. Yes, I think that could work.
The defaults are on the v1-internal transition, and removed on the internal-v1 transition.
(3) is converting v1 to IR, and (4) is converting IR to v1, either for storing to etcd or to communicate to another component, such as Kubelet.
I'm saying stop using the default setting code for setting these "breaking" defaults.
Right.
I don't think we want to do this for pods in general, though, just for MPF1K
So I think we can actually just add a few more changes to PR and it should work. (Just move the defaulting.)
Yes, I agree. Just for MPF1K. We can easily broaden it if needed.
SGTM
We're going to want to do something similar for new fields to alpha API versions that we don't want to expose in the stable, serialized version of the API.
So if this works, we should add it to
Yes. Changing defaults is a breaking change and therefore needs to happen in conversion. Adding new fields with new defaults is fine, just not changing the behavior of existing fields. course I do think kubelet is being a very inflexible client here, but we can't fix the stuff that's already out there.
+1 to both points. We should've patched kubelet sooner to avoid this, but it's out there and kubelet 1.0 remains a stubborn client. Sorry for that. thanks for coming up with the solution! Since both of you agreed on , I will change my PR acoordingly.
So that there is no misunderstanding that leads to future such problems: Whether fields are new or pre-existing, it is an invalid assumption that unset fields may not be changed, whether by defaulting or by some other process, such as admission control.
This issue will be considered fixed for 1.1 once is submitted. (I didn't read this thread carefully enough yet to know whether additional work is necessary before considering this whole issue closed.)
Update: has just been merged. Closing the issue.
Probably we should not close this issue until we do a cherrypick?
The change was cherrypicked, so closing this issue

Discussion:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

"spec.hostPID", }, }, <del> { name: "reseting defaults for pre-v1.1 mirror pods", input: ` { "kind":"Pod", "apiVersion":"v1", "metadata":{ "name":"my-pod-name", "namespace":"my-pod-namespace", "annotations": { "kubernetes.io/config.mirror": "mirror" } }, "spec": { "containers":[{ "name":"a", "image":"my-container-image", "resources": { "limits": { "cpu": "100m" } } }] } } `, absentKeys: []string{ "spec.terminationGracePeriodSeconds", "spec.containers[0].resources.requests", }, }, { name: "preserving defaults for v1.1+ mirror pods", input: ` { "kind":"Pod", "apiVersion":"v1", "metadata":{ "name":"my-pod-name", "namespace":"my-pod-namespace", "annotations": { "kubernetes.io/config.mirror": "cbe924f710c7e26f7693d6a341bcfad0" } }, "spec": { "containers":[{ "name":"a", "image":"my-container-image", "resources": { "limits": { "cpu": "100m" } } }] } } `, expectedKeys: map[string]string{ "spec.terminationGracePeriodSeconds":    "30", "spec.containers[0].resources.requests": "map[cpu:100m]", }, }, </del> } validator := func(obj runtime.Object) field.ErrorList {

"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

out.Annotations[PodInitContainerStatusesBetaAnnotationKey] = string(value) } <del> // We need to reset certain fields for mirror pods from pre-v1.1 kubelet // (#15960). // TODO: Remove this code after we drop support for v1.0 kubelets. if value, ok := in.Annotations[mirrorAnnotationKey]; ok && value == mirrorAnnotationValue_1_0 { // Reset the TerminationGracePeriodSeconds. out.Spec.TerminationGracePeriodSeconds = nil // Reset the resource requests. for i := range out.Spec.Containers { out.Spec.Containers[i].Resources.Requests = nil } } </del> return nil }