jiebi/CodeConvo huggingface dataset:
If any of those are a no, then it'll be removed from the pool on the next sync. Perhaps someone from can shed some light. /sig network
cc This problem extends beyond GCP/ingress controller and to anyone using pools to upgrade nodes. A typical workflow I see is you'd cordon a node pool so as you drain each node the pods don't go to another node in the pool that'll shortly be drained as well. The current logic does not support that and complicates cluster administration because cordoning does not respect PDB's.
/remove-sig apps /remove-sig gcp (since the service controller falls under sig networking)
Those labels are not set on the issue: <detailsIn response to : /remove-sig apps /remove-sig gcp Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ref I thought a general practice for draining the old pool is to move over one node at a time in a rolling fashion. Shouldn't there always be certain backend pods being able to receive & serve traffic? cc
WRT draining, it is recommended to do it one node at a time ; however, in real world scenarios doing it one node at a time can cause a lot of avoidable container churn if you don't cordon off the nodes you at least plan to drain so that pods aren't being scheduled on a node you'll be draining after the next one. According to the comments in the service controller it seems the unschedulable part exists because it was how we differentiated masters in the past; however, we have labels for that now. We also have the service exclusion label (can't find when/if that'll be promoted though as i don't think there's a ticket on it).
I'm having problems with this issue as well. The fact that nodes are removed from target pools when cordoned means that I sometimes get brief interruptions of the traffic to a service when I cordon a node. I'm guessing the node isn't gracefully removed from the load balancer which results in some traffic being cut off trying to route through the recently cordoned node. The same problem occasionally happens when deleting nodes / node pools or when the cluster autoscaler deletes a node. We are running GKE ver. 1.10.2-gke.1.
Our best defence against these issue have so far been to limit node changes as much as possible and not use any auto scaling.
I've posted a question regarding this issue in SO some weeks before it was created. The suggested action was to use node taints () instead of cordon, that way nodes will still be marked as .
Thank you for the link, it Certainly helped. Perhaps, if thats the case(as mentioned in the accepted answer of the above link), then to cordon all the nodes of old node pools should not be a recommended action, at least in the GKE upgrade docs.
only says: And I can't find what more it does in the documentation. So I would expect that it has exactly the same behaviour as a taint. Why is removing running pods from the balancer working as intended? If it isn't a bug then the documentation is certainly unclear in this regard.
This just caused a major outage for us too. I agree with the previous posts, while this may be working as the code intended, it does not follow the patterns and documentation k8s provides. When I cordon a node it should not impact the traffic going to the node or any pods on it. I should be able to gradually remove traffic by draining nodes/pods and respecting PDBs.
this issue touches on the fact we are removing masters as well and inadvertently removing cordoned nodes. The logic in the service controller mentions we only filter out unschedulable nodes since that's how masters were setup.
We wanted to replace all of the kubernetes cluster nodes to different node instance types, so we started off by cordoning all of the nodes, but it surprisingly caused an outage for us. As pointed out, the only documentation around cordoning is , which I took to mean that new pods can not be scheduled on those nodes. It was surprising behavior to find out that, not only does it not allow new pods to be scheduled on those nodes, but that it also causes the service controller to remove unschedulable nodes from load balancing pools. I understand that it's working as intended, but if that was also documented as part of the cordon operation, I would have been able to avoid an outage.
I finally caught up to this bug. Fun one. I agree that this is working-as-intended and that it stinks. We can do better.
As a short-term workaround, you can taint the nodes before cordoning one-by-one. That will stop new work from arriving, but will leave old work there. If we agree on an appropriately named taint, we can have nginx tolerate it by default. Am I missing any reason why this would not work? If that flies, we should fix docs ( those are GCP docs). I am happy to help with that. Longer term maybe we can do something smarter. I am not sure what that would be, just now. Ideas?
Is there any reason we can't just leave the nodes in the load balancer pool when the node is cordoned? And when the pods are terminated the nodes can be removed? I may be missing some context here?
The presumption is that a node that is unschedulable is going away, or needs some other repair. Using it as a target for load-balancers means there could be sessions routed THROUGH that node. Even as the pods are drained. Right up until the node suddenly dies, causing an outage of a different flavor. What should really happen is that a node which becomes unschedulable starts draining connections. Once all the pods are moved off, the node can be removed from the pool entirely. I don't think any cloud providers, even GCP, handle this just right today. David Radcliffe <:
Wait, if you brought up a new nodepool, all those new nodes should be in the TargetPool. Cordoning the old nodepool should only terminate the existing connections, new connections should be fine. Something doesn't add up...
Was this using
Yes
Right, well, that explains it. So even draining won't do what you want - you need new connections to be allowed until you have capacity in the new pool. The good news is that you can do this entirely yourself, with taints. We just need nginx to tolerate the taint, which you can also specify yourself. The only real alternative I see would be, as you said, to leave nodes in the LB pool until the node dies. Given that we never specified this, it is probably different on every provider, which would make it hard to change (especially if the provider already provides clean drains while unschedulable).
IIUC, we don't want new pods to land on the tainted/cordoned nodes. In that case, we can place "NoSchedule" taint on the nodes. We don't need any tolerations on nginx. This will prevent new instances of nginx from landing on the tainted nodes, but the existing ones will keep running there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This just surprised us also. I understand the technical reasons behind it but the command name and associated documentation is misleading. To me it isn't intuitive that the command does not remove compute (pods) but does remove some networking. I would expect cordon to keep new pods from getting scheduled to it and keep it from getting to new services/ELBs/etc. The command would then cause pods to be removed and also take the node out of networking paths. Almost need a new state for a node of that is clearly defined.
My main issue is with the docs, and also GKE docs. They should explain this pitfall. After I got aware of the issue I understand why it works as it does. A simple workaround is to taint the nodes you are going to rotate first.
Just chiming in; We as a GKE customer have today been bitten by this issue in production. The documentation of (in the context of GKE) is not extensive enough so we could be aware of this issue. This should be addressed. Luckily the outage was not severe because we were using the node-by-node approach (and accepted the container churn this would cause). We explicitly took this approach because our previous nodepool upgrade was a total outage also caused by in a different way. (for those interested in our total outage, not related to this issue): Due to the fact that also results in the re-calculating the amount of schedulable nodes and thus scaled the amount of pods in our cluster (in our case: from 110 back to 2) resulting for us in major DNS outage internally.
Be warned that for a nodepool upgrade can have a lot of unwanted & unexpected side effects.
It looks like this happens only to AWS and GCP but not on Azure? Can anyone confirm this? I just tested in both AWS and Azure, and while I see logs like in AWS Kubernetes, I do not see any logs like that in Azure. Is it intended? If so isn't behavior different for each cloud provider? i.e.) cordoning a node in AWS Kubernetes prevents loadbalancer from routing traffic to the affected node but cordoning a node in Azure Kubernetes will still let the loadbalancer route traffic to the affected node. Is it intentional?
Besides running a test like the above, how would I determine what the behavior is on AWS? I haven't found anything in the node administration docs or the Service that would imply that would shift traffic away from a Pod. In fact, the doc for explicitly states, emphasis mine: If I hadn't come across this Issue due to reports from GKE, I suppose it would only have been a matter of time until we had a high-impact production outage. There's a disconnect between Kubernetes API and the LoadBalancer as to whether Pods running on a 'd node are Ready. From all the docs I've seen, I would expect that either a) would evict Pods over to a Node before setting the instance as , or b) LoadBalancer would not equate with
Hi All, What approaches people using too workaround this? From reading the comments I was hoping to follow the custom taint approach. However, I've noticed executes a cordon making it difficult to drain the node safely with this approach.
I was lucky in that I experienced this issue while cordoning our testing environment, so we didn't experience a production outage. I think what I'm going to try is the following... it's going to be time-consuming but I think it's the safest bet. Double the number of nodes in the server group to bring online the new nodes first Place a custom taint on each "legacy" node As you said, executes a first, so that's a no-go. This is the time-consuming part... for each "legacy" node I am going to each pod, and let my PDBs take care of maintaining minimum number of pods. I'm hopeful to try this approach this afternoon, I'll comment back here if I do!
Same as myself! Pity neither of us have a story for Since my comment this morning I've been trying a few approaches, I've ended up with the following: Create a new node pool of equal size in GKE with the upgraded kube version Set the current node pool to not be able to grow beyond its current size - if your using regional nodes this can be annoying, let it bring up new nodes anyways and just have them taint as NoSchedule straight away Taint all old node pool nodes with NoSchedule Go through each old node pool node executing a taint of NoExecute which will cause the node to drain and go onto available nodes i.e. your newly provisioned node pool Ensure your nginx ingress is deployed with tolerances:
Hi All, I'm at a bit of a loss on this one. I've started using custom taints to drain the nodes without getting them removed from the load balancer. My application can hold a HTTP connection open for up to 600 seconds. I'm using nginx as a daemonset, have it configured with a keepalive timeout of 620s, and have . I drain the application first using a taint on each node. I have nginx on each node tolerate this taint and remove it separately at a later point using a different taint. I'm waiting 660s between drains of each node for each taint to ensure nothing is dropped - this all works fine as expected. However, once I kill the node pool or do a kubectl drain after the node is empty of all my stuff plus nginx ingress I start seeing connection issues. I'm a bit lost where to go at debugging them, best guess is something to do with kube-proxy. Can anyone shed some light on this? is there anything on the Google Layer 4 load balancer I'm missing here? Is there some way to correctly drain kube-proxy (or does that even make sense?)
This doesn't address the network issues you're seeing, but for those trying to "roll their own cordon" using taints: Using a NoExecute taint to "drain" the node won't be orderly and won't respect any PDBs. You want to replicate the logic from which tries to do evictions of every pod in a loop. (I realize this is a burdensome idea, but it is also the safest idea.)
To add on to ,: One approach that does not use node taints to force migrate, is to change for the requirements of your app. A change in node selector (e.g. based on instance type), along with a PDB (e.g. 66% / out of 3 replicas availability) should migrate you over fine. This worked out better for me than to drain or add taints to nodes to force push apps out. After the migration completes, you should be able to the nodes, and delete the node pool. FWIW, I too saw connection issues during load testing of a migration, and at least on my setup it was due to using an AWS NLB for the NGINX Ingress Controller. Switching to a classic ELB instead worked, as the NGINX LB Service seems to hit some sort of by my accounts when using NLBs. self-plug: I happened to recently cover a similar topic in a that you may find relevant. Hope it helps
For better or worse, for in my case pods that are behind an ingress resource are using node affinity to preferably not schedule them onto the same node. Originally this was done to ensure traffic is balanced across the different nginx ingresses. It has had a positive affect in this case too. Very interesting! Thank you for sharing, I'm not sure that swapping out the Layer4 GCE load balancer is an option here, but you have me thinking. I'm wondering if the TCP keepalive between the LB and nginx is potentially keeping connections open. , and suggest that there may be a 600 second TCP keepalive going on. Will retest doing a sleep of 620s after nginx removal and before node drain.
Has anyone come up with a strategy to safely move workloads between two pools? Reading this bug, it seems there's no way to reliably do this without causing an outage if you follow the recommendation of + . In my case I'm not using an nginx ingress but services with type . From what I can see, adding to a node causes the Service Controller to consider the node not ready and thus removes it from a target group. This would mean that between and there is no way to add a taint to the node that will leave it in the LB until it's drained.
Using doesn't work in this case since the thing I'm draining might just come back and I'm not sure if the service controller will still consider the node ready in that case, but I believe it does. The one strategy I can think of is using labels on the node and an anti-affinity on the pod. You'd then add the label to a node that would push the pod away and then delete the pod. As long as the anti affinity uses this should be safe to do in preparation of a migration. Once the pods have been deleted you could then the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale /remove-lifecycle rotten
- As I mentioned in I set up a script that first taints all nodes in the old node pool and then cordon+taint one by one node. If your ingress services are redundant this should work well.
Is there a plan for cordon behavior to improve on GCP? We have relied on this behavior for years on our AWS clusters and it was very surprising to find out.
This is a bug with service load balancers - not something we should document. We defined a label in that cluster admins can use - Unschedulable has nothing to do with SLB inclusion. which is beta and on by default in 1.19.
Disagree. Unschedulable means "no new workloads". It has nothing to do with existing workloads, that is why we have drain (but we should have something in kube that communicates nodes are draining). I have opened to remove this check - we have an alternate approach now and the old logic is wrong (the comment in the code even says "this is for masters!" and that's wrong now too as part of the kep).
A node status condition?
Possibly, or a taint, or a label. Something concrete.
/milestone v1.19
When using a generic ephemeral volume, the volume binder in kube-scheduler typically processes a new pod before the generic volume controller in kube-controller-manager has a chance to create the PVC. This leads to events for pods using generic ephemeral volumes: It would be nice to suppress these warnings for a short while (duration to be determined) after the pod creation. Later they should be emitted because then they indicate that the generic volume controller isn't working properly. This makes nicer to use.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.
However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm trying to test Eviction functionality in my program, but running into some inconsistencies around how Evictions are handled in the fake SimpleClient in the client-go testing functionality. Most notably, creates a different from , and both are missing information. issues a with set to (seems to be the only Fake that has this mismatch) on the subresource of . It is missing all the extra information in the eviction, i.e. the . issues a on the subresource of the resource. This is wrong, because aren't created, they're only posted and handled immediately. The Action is missing a Namespace. What you expected to happen: I want to be able to add my own reactor to the SimpleClient to handle this special case. I propose we change both cases to a with set to the . This will ensure it's not handled by the current SimpleClient ObjectReaction, and makes it so users can implement whatever they want themselves. I'd be happy to make a PR with these changes. How to reproduce it (as minimally and precisely as possible): Output: Environment: git revision of : <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
POST requests are create requests. It doesn't matter if the object is persisted or not.
/sig api-machinery
cc
Not sure if this concided with a Jenkins restart.
Not sure what we can do about this. Retry a few times?
GitHub seemed to hiccup last night and fail a bunch of our builds. I'm not sure a retry would've helped, unless we kept retrying for many minutes.
This is the sort of thing that just happens occasionally. We've seen this a lot in OpenShift CI runs. I'm not sure there's much you can do other than try to avoid external network access (e.g. bake tools into the VM image).
It seems to have occurred twice in the last 12h so retry w/ timeout can't hurt?
It can't hurt... but it may or may not help depending on the extent of the external issue
If the problem is really bad, we could toss it somewhere on GCS?
Looks like it failed on : We should probably put this in a retry loop or else do something more fancy
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I tried to create a pod with env variables referencing to annotations: What you expected to happen: Should Proceed. As in It stated is a valid option How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig docs
The API docs need to be updated to indicate the correct supported values. See for correct values
You are a lifesaver. Thank you!
/assign I can try to update the api documentation