Change:
During debugging , I noticed that one of services pods: monitoring and logging pods stuck at pending sometimes forever: I logged into the node, and found the container is running happily on the node: Looks like status is never reported from kubelet. I checked kubelet log, then I found: Why we cannot delete mirror pod? and Created a new one? Because this regression, once we run into such state, the rest of test won't be triggered, and eventually timeout / failed.
checks those service pods in ready state, and prevent the test to run once it is pending. cc/ I think this is the real issue I talked to you earlier about PodStatus.
I spoke to offline. For some reason, kubelet thinks the spec of the mirror pod does not match the static pod locally. It then decides to delete the mirror pod and recreate it, repeatedly. As a result of that, it slows down (due to the QPS limit?) reporting statuses of other pods. This can be reproduced by running the e2e "Shell" test. I am looking into why the specs would suddenly mismatch. Did any recent PR modified the pod spec in apiserver?
The recently introduced default cpu share limit was not there in the static pod, causing the difference between its spec and the mirror pod's.
Seeing this as well. Makes the whole cluster slow and unusable for me. Sent from my iPhone
Is there a desired fix for this?
said she would add default resource limits to the static pod manifest files. If we go for that, I think it's reasonable to fail (not run the static pod) if resource limit is not specified. Also, the resource limits set in the mirror pods are not meaningful because kubelet creates containers based on the spec of the static pod. In general, static pods do not subject to any defaulting values applied at the apiserver.
Pretty sure that this is fixed by . Certainly testing against head since then works well for me. What version are you testing?
was fixing node status, and the problem had been around for a while. This issue is about pod status, which is still unfixed on head. These two are very different, and my cluster did include the fix :) This issue will slow down kubelet at times, but if you get lucky and the pod status (running) get sent out in time, the tests will still pass.
There is no way the issue is fixed by that PR. I am still seeing the issue on HEAD. The system gets worse the longer its up. I agree with the analysis of the issue in source PR Sent from my iPhone
I am on this one, and here is proposed solution in my mind so far (not final yet since I haven't put too much thoughts into it yet): 1) For all our static pods I am going to update them with some initial cpu limit for now. This should quickly stop flakiness caused by this particular issue. 2) Next step is that for static pods, if there is no initial cpu limit, kubelet should refuse to run. Before I do this, I need to send an announcement to our container-vm users about this potential breakage for the in the future. Another potential fix is we can ignore resource changes when comparing if a mirror pod is updated or not. I think is proposed a similiar one earlier. But I had a concern with it: this is defeat the initial purpose we created mirror pods and sync back to master. So if one made some resource changes on the static pods, you have to always ssh-ed into the node to see if the new value applied or not. Also eventually (post 1.0) those mirrored pod's resource limit is going to be used by scheduler when calculating if user pods are fit or not, we need them to be accurated. Another concern is in the code implementation itself to distinuguish mirror pods future from rest of pods. I guess this shouldn't be the big issue here. Any more suggestions here? One puzzle I had is that why master component pods doesn't have issue at all. I manually restart kube-apiserver pod, still doesn't have the issue.
Depending on time we could just remove the default limit from the limit range as a stop gap. Sent from my iPhone
Can you just bypass defaulting at the apiserver when you create mirror pods? I can't think of any other solution that works in general and is easily maintainable. Approach 2 you described in the previous entry works if we are defaulting cpu limit, but breaks as soon as we default anything else. Same problem with ignoring resource differences when comparing mirror pod to static pod -- as long as defaulting only touches resources, that works, but if we change it to touch other stuff, then it breaks.
Commit I had can quickly fix current bleeding issue. Currently we only have master component pods and fluentd pod are file based by default. Master component pods doesn't create any issue now because mirror pods aren't created at the first place (due to recent change made by Also file-based pods will have issue only when its spec doesn't have initial value, and need to update (delete miss matched mirror pod, and recreate a new one). Please note that with my commit, everything is working even I manually update its cpu limit from 100m to 200m, there is no issue at all, and query pod shows accurate result with the new limit too: How badly we request all users using file-based pod feature providing default cpu limit for now? Eventually we want to get rid of file-based pods completely, also encouraging the user provides the initial resource limit is aligning with our direction in a long run anyway.
Just to clarify, what I was suggesting was for a "real fix" not the temporary measure to stop the bleeding (which you already have in your PR). But what do you think of what I suggested as the longer-term fix? My point is that today, we default CPU, but in the future, we may default anything. It seems that the most maintainable fix is to just not do defaulting when creating mirror pod from static pod. Is that possible? I wonder if the reason manual update works (what you described in your last comment) is because we don't go through the defaulting when you do an update, only a create? (Though that sounds like a bug.) IMO we don't need CPU default right now, especially since we already have kubelet flag for limiting pods per machine (though unfortunately it's currently set to 100 because we haven't reconfigured the tests that intentionally run lots of pods per node, to override the flag. we can change the default to 30 and add an override mechanism if this is important.)
Comparing a config file with the live state and expecting them to match is doomed. The recommended approach is described by . Kubelet could store the config as an annotation in the mirror pod and compare with that.
Does the scheduler favor nodes with the fewest pods even if they don't specify resources? LimitRange does cause that to happen.
Storing the config as an annotation sounds good to me. Filed
Although this issues was closed, it still appears in my environment
Same here, this happens from time to time in our environments, is there a way to debug to find the root cause?

Discussion:
containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

containers: - name: fluentd-cloud-logging image: gcr.io/google_containers/fluentd-gcp:1.6 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"