Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In some case, controller manager failed to create pod as replicas number defined in ReplicaSet spec. Normally if replicaset controller fails to create a pod, it will try later. () But func in ignore timeout error for some reason(), then if a create pod request really timeout during accessing apiserver, this failed will not add to the queue again. Then the pod should be created will never be created again. () I saw this happened in a production environment. What you expected to happen: Controller manager will retry and create pod again. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7 Kernel (e.g. ): Install tools: Others: /cc
/sig api-machinery
/sig apps /remove-sig api-machinery
In : The controllers ignore the timeout error when creating the pods, depending on how the initialization progress: the initialization is successful in less than 5 mins, the controller will observe the creation via the informer. All is good. the initialization fails, server will delete the pod, but the controller won't receive any event. The controller will not create new pod until the Creation expectation expires in 5 min. the initialization takes too long (5 mins), the Creation expectation expires and the controller will create extra pods. Note 2,3 that if the creation request failed with timeout, the ReplicaSetController will resync: And the expectation creation has already been in if diff < 0:
I got the similar situation. After getting type: 'Warning' reason: 'FailedCreate' Error creating: Timeout: request did not complete within allowed duration, Pod is not created. Its ReplicaSet has .spec.replicas = 0, but .status.replicas stays 0 without any reason. k8s version is 1.11.0.
hello , , i am thinking the code in causes the inconsistency above. it will only RE-enqueue the RS when ReadyReplicas == Replicas && AvailableReplicas != Replicas, while there are situations that ReadyReplicas != Replicas.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Also seeing this in 1.12 on GKE (1.12.8-gke.6).
Just got this now on AWS (v1.12.6-eks-): Using Istio 1.1.7, if that helps at all. I updated some nodes and switched to a new node type, and whilst it was supposed to be bringing these pods back up, it just never did. It's only in our staging environment right now at least!
Ooooh, we're also running Istio. (And I just saw the issue again in a different GKE cluster.) With the automatic sidecar-injection. Maybe an interplay problem with that admission controller.
Hitting the same issue on EKS 1.13.7, and we are using Istio as well (1.2.0). We ran into the issue while doing a rolling-upgrade of the cluster, with a lot of pod churn, so I assume that the above hypothesis around admission-controller / sidecar injection might be correct. :) Edit: we're trying to mitigate this by making the sidecar injector HA; by default it is running a single instance.
did scaling the sidecar injector work? I'm still hitting this issue with Istio 1.2.2, and it's really making me nervous about upgrading the production server. It's consistently not bringing pods back.
it seems to have mitigated the problem for us, but I'm not sure we've done another full rolling-update since my previous message. In any case, scaling up the injector can't hurt
Happens on my cluster two, but we're not using Istio. Message is following, with no additional clue : One thing to note is that it happened after pods were Evicted, and OutofEphemeral condition. Disk was full during the eviction.
I actually found the issue I've been having "with Istio". We're using CloudFormation to create the nodes in AWS (almost the default CloudFormation template given by AWS). The problem with this is that it doesn't drain the nodes before killing them, so even if you replace nodes in batches, you're still killing pods. If you kill the Istio pods, it can then prevent other pods from starting up (giving you the aforementioned error). From that point I think some of the fault lies with Kubernetes too, because there are times when a node has been terminated, and I can go look at a ReplicaSet and it'll think it's got the right amount of pods running, then I can go look at pods, and there'll be nothing there for that service. I think this prevents Istio from recovering as quickly as it could. Istio then seems to be very picky about the order that services start in, so it takes quite a bit of time for Istio to recovery even once the Istio pods start coming back. So what I've done now is labels to the nodes (e.g. or ), and then when I want to upgrade the nodes, I create an entire new node group with the opposite deployment label. Then using that label I taint and drain the old nodes. At this point you may need to scale up some services to allow nodes to be drained because of PodDisruptionBudgets. Once the nodes are drained, I destroy the CloudFormation stack for the old node (alternatively I suppose you could set the desired number of nodes to 0). Luckily this isn't something that really needs to happen very often. Also, I'm pretty sure using would ease some of this, but we're using Terraform for our infrastructure, and I'd rather keep it all in once place... Next step for me is trying to figure out how to replace nodes in batches, because ideally I don't want twice as many nodes as I need running at one point. I might hit some limits in AWS for certain node types.
I think I ran into this situation as well. It was caused by a similar situation as described by . A change to a Terraform module resulted in ASG replacement. The change was supposed to be deployed by hand on each ASG individually, which though not as graceful as using taints to drain the nodes in the old ASG, would have probably allowed Kubernetes to reschedule the unexpectedly terminated pods without too much difficulty. Instead, the change was inadvertently applied to a large number of ASGs all at roughly the same time. This caused many ReplicaSets to report 0 current and 0 ready pods, and to have the "Error creating: the server was unable to return a response in the time allotted, but may still be processing the request (post pods)" event. Luckily, manually scaling each ReplicaSet to 0 and then back to the original scale gave Kubernetes the kick it needed to create the pods.
I think I am hitting this bug, too. My situation is that I have an validating webhook admission controller attached to the namespace. If the admission controller goes offline, the replica set for regular Deployments on that namespace will not be able to create pods as they can't get past the admission controller. They list 'Timeout: request did not complete within allowed duration' errors, and then never try again to create the pod. Once the admission controller is back online. These replica sets just sit there. The only way to get the Deployment to deploy pods again is to delete its replica set. Then, it will make a new replica set and that one will happily deploy the pods. In particular, this is problematic if you have some sort of moderately widespread worker node outage. If a bunch/all of workers go offline at once, you could lose your admission controllers temporarily and then get into a situation where nothing comes back due to stuck replica sets.
Got the same situation in EKS (we have a nice monitoring alert that matches events containing ), server version . Fixed using the commands: Delete stuck replicasets, Kubernetes creates another instead and if you don't have any other problems (like resource quotas, for example), they will start new pods.
Observed on GKE 1.13.6-gke.13 or 1.13.7-gke.24, as it happened during an upgrade. Lots of FailedCreate events. We got scheduling back on track by scaling deployments to zero replicas and then back to the intended number.
Found the following in ReplicaSetController#manageReplicas :
I am hitting this bug, too. If ReplicaSetController create pod request really timeout during accessing apiserver, this failed key will not add to the queue again. Then the pod should be created will never be created again. in manageReplicas, if create pod timeout < 5 minute, the queue has no key, the controller never sync when no key , so it never create a new pod even through the expectation expires. rs status kubectl get rs --all-namespaces NAMESPACE NAME DESIRED CURRENT READY AGE kube-system kubenab- 3 2 2 5d9h kube-controller-manager log: I1204 04:13:23. 1 ] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kubenab-", UID:"-b142-4db2-a914-", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: the server was unable to return a response in the time allotted, but may still be processing the request (post pods) kube-apiserver log: W1204 04:12:53. 1 ] Failed calling webhook, failing open kubenab- failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:12:53. 1 ] failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:13:23. 1 ] apiserver was unable to write a JSON response: http: Handler timeout E1204 04:13:23. 1 ] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"} I1204 04:13:23. 1 ] Trace[]: "Create /api/v1/namespaces/kube-system/pods" (started: 2019-12-04 04:12:23. +0000 UTC m=+.) (total time: 1m0.011414107s): Trace[]: [30.010012786s] [30.00986498s] About to store object in database Trace[]: [1m0.011414107s] [30.001401321s] END W1204 04:13:23. 1 ] Failed calling webhook, failing open kubenab- failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:13:23. 1 ] failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers)
/assign /assign /assign It appears that the timeout code here: may be out-of-date now that initializers have been removed. The solution may be to just remove the code swallowing the timeout.

Discussion:
// this error since all subsequent creations will fail. return } <del> if errors.IsTimeout(err) { // Pod is created but its initialization has timed out. // If the initialization is successful eventually, the // controller will observe the creation via the informer. // If the initialization fails, or if the pod keeps // uninitialized for a long time, the informer will not // receive any update, and the controller will create a new // pod when the expectation expires. return } </del> } if err != nil { klog.V(2).Infof("Failed creation, decrementing expectations for set %q/%q", ds.Namespace, ds.Name)

// this error since all subsequent creations will fail. return } <del> if errors.IsTimeout(err) { // Pod is created but its initialization has timed out. // If the initialization is successful eventually, the // controller will observe the creation via the informer. // If the initialization fails, or if the pod keeps // uninitialized for a long time, the informer will not // receive any update, and the controller will create a new // pod when the expectation expires. return } </del> } if err != nil { defer utilruntime.HandleError(err)

// anything because any creation will fail return nil } <del> if errors.IsTimeout(err) { // Pod is created but its initialization has timed out. // If the initialization is successful eventually, the // controller will observe the creation via the informer. // If the initialization fails, or if the pod keeps // uninitialized for a long time, the informer will not // receive any update, and the controller will create a new // pod when the expectation expires. return nil } </del> } return err })