jiebi/CodeConvo huggingface dataset:
Code change:
containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: During debugging , I noticed that one of services pods: monitoring and logging pods stuck at pending sometimes forever: I logged into the node, and found the container is running happily on the node: Looks like status is never reported from kubelet. I checked kubelet log, then I found: Why we cannot delete mirror pod? and Created a new one? Because this regression, once we run into such state, the rest of test won't be triggered, and eventually timeout / failed.
checks those service pods in ready state, and prevent the test to run once it is pending. cc/ I think this is the real issue I talked to you earlier about PodStatus.
I spoke to offline. For some reason, kubelet thinks the spec of the mirror pod does not match the static pod locally. It then decides to delete the mirror pod and recreate it, repeatedly. As a result of that, it slows down (due to the QPS limit?) reporting statuses of other pods. This can be reproduced by running the e2e "Shell" test. I am looking into why the specs would suddenly mismatch. Did any recent PR modified the pod spec in apiserver?
The recently introduced default cpu share limit was not there in the static pod, causing the difference between its spec and the mirror pod's.
Seeing this as well. Makes the whole cluster slow and unusable for me. Sent from my iPhone
Is there a desired fix for this?
said she would add default resource limits to the static pod manifest files. If we go for that, I think it's reasonable to fail (not run the static pod) if resource limit is not specified. Also, the resource limits set in the mirror pods are not meaningful because kubelet creates containers based on the spec of the static pod. In general, static pods do not subject to any defaulting values applied at the apiserver.
Pretty sure that this is fixed by . Certainly testing against head since then works well for me. What version are you testing?
was fixing node status, and the problem had been around for a while. This issue is about pod status, which is still unfixed on head. These two are very different, and my cluster did include the fix :) This issue will slow down kubelet at times, but if you get lucky and the pod status (running) get sent out in time, the tests will still pass.
Code change:
containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: There is no way the issue is fixed by that PR. I am still seeing the issue on HEAD. The system gets worse the longer its up. I agree with the analysis of the issue in source PR Sent from my iPhone
I am on this one, and here is proposed solution in my mind so far (not final yet since I haven't put too much thoughts into it yet): 1) For all our static pods I am going to update them with some initial cpu limit for now. This should quickly stop flakiness caused by this particular issue. 2) Next step is that for static pods, if there is no initial cpu limit, kubelet should refuse to run. Before I do this, I need to send an announcement to our container-vm users about this potential breakage for the in the future. Another potential fix is we can ignore resource changes when comparing if a mirror pod is updated or not. I think is proposed a similiar one earlier. But I had a concern with it: this is defeat the initial purpose we created mirror pods and sync back to master. So if one made some resource changes on the static pods, you have to always ssh-ed into the node to see if the new value applied or not. Also eventually (post 1.0) those mirrored pod's resource limit is going to be used by scheduler when calculating if user pods are fit or not, we need them to be accurated. Another concern is in the code implementation itself to distinuguish mirror pods future from rest of pods. I guess this shouldn't be the big issue here. Any more suggestions here? One puzzle I had is that why master component pods doesn't have issue at all. I manually restart kube-apiserver pod, still doesn't have the issue.
Depending on time we could just remove the default limit from the limit range as a stop gap. Sent from my iPhone
Can you just bypass defaulting at the apiserver when you create mirror pods? I can't think of any other solution that works in general and is easily maintainable. Approach 2 you described in the previous entry works if we are defaulting cpu limit, but breaks as soon as we default anything else.
Code change:
containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: Same problem with ignoring resource differences when comparing mirror pod to static pod -- as long as defaulting only touches resources, that works, but if we change it to touch other stuff, then it breaks.
Commit I had can quickly fix current bleeding issue. Currently we only have master component pods and fluentd pod are file based by default. Master component pods doesn't create any issue now because mirror pods aren't created at the first place (due to recent change made by Also file-based pods will have issue only when its spec doesn't have initial value, and need to update (delete miss matched mirror pod, and recreate a new one). Please note that with my commit, everything is working even I manually update its cpu limit from 100m to 200m, there is no issue at all, and query pod shows accurate result with the new limit too: How badly we request all users using file-based pod feature providing default cpu limit for now? Eventually we want to get rid of file-based pods completely, also encouraging the user provides the initial resource limit is aligning with our direction in a long run anyway.
Just to clarify, what I was suggesting was for a "real fix" not the temporary measure to stop the bleeding (which you already have in your PR). But what do you think of what I suggested as the longer-term fix? My point is that today, we default CPU, but in the future, we may default anything. It seems that the most maintainable fix is to just not do defaulting when creating mirror pod from static pod. Is that possible? I wonder if the reason manual update works (what you described in your last comment) is because we don't go through the defaulting when you do an update, only a create? (Though that sounds like a bug.) IMO we don't need CPU default right now, especially since we already have kubelet flag for limiting pods per machine (though unfortunately it's currently set to 100 because we haven't reconfigured the tests that intentionally run lots of pods per node, to override the flag. we can change the default to 30 and add an override mechanism if this is important.)
Comparing a config file with the live state and expecting them to match is doomed. The recommended approach is described by . Kubelet could store the config as an annotation in the mirror pod and compare with that.
Code change:
containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Related discussion:
text: Does the scheduler favor nodes with the fewest pods even if they don't specify resources? LimitRange does cause that to happen.
Storing the config as an annotation sounds good to me. Filed
Although this issues was closed, it still appears in my environment
Same here, this happens from time to time in our environments, is there a way to debug to find the root cause?
Code change:
}, { manifest: KubeProxyDaemonSet, <del> data: struct{ ImageRepository, Arch, Version, ImageOverride, ClusterCIDR, MasterTaintKey, CloudTaintKey string }{ </del> <ins> data: struct{ ImageRepository, Arch, Version, ImageOverride, ExtraParams, ClusterCIDR, MasterTaintKey, CloudTaintKey string }{ </ins> ImageRepository: "foo", Arch:            "foo", Version:         "foo", ImageOverride:   "foo", <ins> ExtraParams:     "foo", </ins> ClusterCIDR:     "foo", MasterTaintKey:  "foo", CloudTaintKey:   "foo",

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.
Code change:
Description string `json:"description,omitempty"` // Versions are versions for this third party object <del> // +optional </del> Versions []APIVersion `json:"versions,omitempty"` }

Related discussion:
text: FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.
Code change:
nodeTime = time.Now() bootTime, err = util.GetBootTime() <del> gomega.Expect(err).To(gomega.BeNil()) </del> <ins> framework.ExpectNoError(err) </ins> // Set lookback duration longer than node up time. // Assume the test won't take more than 1 hour, in fact it usually only takes 90 seconds.

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.
Code change:
genericConfig.DisabledPostStartHooks.Insert(rbacrest.PostStartHookName) } <del> webhookAuthResolver := func(delegate webhookconfig.AuthenticationInfoResolver) webhookconfig.AuthenticationInfoResolver { return webhookconfig.AuthenticationInfoResolverFunc(func(server string) (*rest.Config, error) { if server == "kubernetes.default.svc" { return genericConfig.LoopbackClientConfig, nil } ret, err := delegate.ClientConfigFor(server) if err != nil { return nil, err } if proxyTransport != nil && proxyTransport.Dial != nil { ret.Dial = proxyTransport.Dial } return ret, err }) </del> <ins> webhookAuthResolverWrapper := func(delegate webhookconfig.AuthenticationInfoResolver) webhookconfig.AuthenticationInfoResolver { return &webhookconfig.AuthenticationInfoResolverDelegator{ ClientConfigForFunc: func(server string) (*rest.Config, error) { if server == "kubernetes.default.svc" { return genericConfig.LoopbackClientConfig, nil } return delegate.ClientConfigFor(server) }, ClientConfigForServiceFunc: func(serviceName, serviceNamespace string) (*rest.Config, error) { if serviceName == "kubernetes" && serviceNamespace == "default" { return genericConfig.LoopbackClientConfig, nil } ret, err := delegate.ClientConfigForService(serviceName, serviceNamespace) if err != nil { return nil, err } if proxyTransport != nil && proxyTransport.Dial != nil { ret.Dial = proxyTransport.Dial } return ret, err }, } </ins> } pluginInitializers, err := BuildAdmissionPluginInitializers( s, client, sharedInformers, serviceResolver, <del> webhookAuthResolver, </del> <ins> webhookAuthResolverWrapper, </ins> ) if err != nil { return nil, nil, nil, nil, nil, fmt.Errorf("failed to create admission plugin initializer: %v", err)

Related discussion:
text: We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close
Code change:
// recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Related discussion:
text: is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.
Code change:
// recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Related discussion:
text: Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.
Code change:
// recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Related discussion:
text: Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.
Code change:
// recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Related discussion:
text: The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.
Code change:
} } <del> func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </del> <ins> func isPkiVolumeMountNeeded() bool { // On some systems were we host-mount /etc/ssl/certs, it is also required to mount /etc/pki. This is needed // due to symlinks pointing from files in /etc/ssl/certs into /etc/pki/ if _, err := os.Stat("/etc/pki"); err == nil { return true } return false } func pkiVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </ins> return api.Volume{ Name: "pki", VolumeSource: api.VolumeSource{ <ins> // TODO(phase1+) make path configurable HostPath: &api.HostPathVolumeSource{Path: "/etc/pki"}, }, } } func pkiVolumeMount() api.VolumeMount { return api.VolumeMount{ Name:      "pki", MountPath: "/etc/pki", } } func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{ Name: "k8s", VolumeSource: api.VolumeSource{ </ins> HostPath: &api.HostPathVolumeSource{Path: kubeadmapi.GlobalEnvParams.KubernetesDir}, }, }

Related discussion:
text: <!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.
Code change:
} } <del> func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </del> <ins> func isPkiVolumeMountNeeded() bool { // On some systems were we host-mount /etc/ssl/certs, it is also required to mount /etc/pki. This is needed // due to symlinks pointing from files in /etc/ssl/certs into /etc/pki/ if _, err := os.Stat("/etc/pki"); err == nil { return true } return false } func pkiVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </ins> return api.Volume{ Name: "pki", VolumeSource: api.VolumeSource{ <ins> // TODO(phase1+) make path configurable HostPath: &api.HostPathVolumeSource{Path: "/etc/pki"}, }, } } func pkiVolumeMount() api.VolumeMount { return api.VolumeMount{ Name:      "pki", MountPath: "/etc/pki", } } func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{ Name: "k8s", VolumeSource: api.VolumeSource{ </ins> HostPath: &api.HostPathVolumeSource{Path: kubeadmapi.GlobalEnvParams.KubernetesDir}, }, }

Related discussion:
text: This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm