Change:
Using v1.2.0-beta.1. Deployed a DaemonSet with no node selector, but it's not running in all of them. The two that are running are the ones with SchedulingDisabled.
can I see: Do the nodes have enough resource? Are their host port conflicts?
Output at and parameters removed to avoid potential leaks. Resources look fine, no port conflicts. There is another DaemonSet in the dump, called that is experiencing the same issue.
i am experiencing the same issue with 10 nodes. running latest releases kubernetes 1.2.0 this is the only daemon set i have running
Do the nodes have capacity, are you using host ports?
2 of the nodes DO NOT have capacities, all the other do. i am not using host ports will it be possible to reserve resource for daemonsets? will they get created in the nodes with no capacity when there is?
kube version across the cluster is 1.2 as well as my local kubectl version i started using --kube-reserved=cpu=1000m,memory=1G and launched a DS for fluentd i then launched 3 pods into each node that were consuming about 80% ram and about 70% cpu, the DS staying there and all the logs were in ES. today i second DS for sensu, only the fluentd DS pods, heapster and dashboard pods were running when i this second DS the result was all the pods being created and everything seemed to work, them i proceeded to create again 3 pods into each node that were consuming about 80% ram and about 70% cpu the result of this was that all the DS pods were killed for both DS this is description on one of the nodes where all DS pods were killed
some updates here, after the pods were killed and i removed all the pods on the nodes, the DS pods were not recreated once there was resources available for them. i had to delete the DS and create it again for the pods to be back.
I'm seeing this same behavior in my 1.2 cluster. I have 4 nodes in a cluster, all of which have sufficient space available, but the DS is reporting "desired" and "current" counts of 2. What's worse is that things were properly working a few days ago when I rolled this out, but sometime in the last few days, 2 of the nodes lost their DS pods and they haven't come back.
i gave up and moved to manifests on the nodes
After playing around with label selectors, restarting api server and controller, and deleting/adding a node, finally got another of the 4 nodes to have a DS pod on it after waiting ~5 minutes.
I marked this p0 for triage for now.
Ok - more troubleshooting with Kelsey on slack...deleting the problem nodes by hand, then restarting kubelet on those nodes seemed to fix the issue. The DS scheduled the remaining node once the kubelet had registered itself. Guessing a bad cache somewhere.
I'd love to kubectl get node -o json on a bad node
The last node in the cluster (which came back with re-registering the kubelet) had recently undergone a coreos automatic update - looks like it's running docker 1.9.1 on that node now. The third node (which came online after playing around with label selectors and re-registration) did not receive the upgrade and is still running docker 1.8.3. Perhaps the reboot/registration that happened broke a cache somewhere?
I have this behavior in another cluster as well - the bad node went through a reboot/upgrade cycle early this morning. Here's the node description:
Do you have any failed or succedded pods on those nodes? are you using Job resource for anything? you can see them when you run kubectl get pod -o wide --show-all. adds an event that should give us insight into where these pods are getting caught up. I also opened which looks like a bug.
Lots of happy pods on these nodes, driven from RCs and Deployments. I'm not using Jobs at all yet. On Apr 6, 2016 11:51 AM, "Mike Danese" wrote:
cc
I had two clusters, both with 2 nodes, one where DaemonSets are working and one (older) where they have stopped working. Both have happy (kube-system) pods on them, both have sufficient free capacity (20ish percent cpu and memory allocated on both). Both have used Jobs in the past, but don't have any jobs atm. I tried restarting the kube-controller-manager with no luck, and also using 1.2.1 kcm with no luck. kubectl get ds --kube-system gives: And if I enable on the KCM (where docker-reg and fluentd are the two DS) I see things like: with the key line I believe being: This despite the fact that these DaemonSets have no node selectors (here's one lightly edited one for example): I'm more than happy to debug this interactively with someone on Slack (handle jknight) if they want more info. I was looking at the code last night and my feeling is it might be related to but without more logging output inside there it's hard to tell why.
Can you paste the output of ?
Ohhh... I think I see now, so this would be a result of thinking the nodes are full due to the completed pods? I can verify this on my other cluster by scheduling some jobs... stay tuned. Also, if anyone has a nightly build with in there, I can test that.
Yup, on my working cluster I started with the two DaemonSets successfully running on both nodes, then after running some jobs I see that go down to 1 and then 0. Can I request a cherry-pick for ?
:)
I just grabbed 1.2.2 KCM, and the problem is fixed. Thanks so much
Is 1.2.2 causing issues for people?
2 seems to be stable for me - my DaemonSets are still running pods on each node after a few days.
It is still an issue for me: I have 4 nodes, 1 master and 3 slaves. I deploy following daemonset: It deploys, but has DESIRED set to 3, and not 4: What's the weirdest of all, it deploys on master node and just two slaves..
Forgot to add:
This seems to be related to because describe on daemonset reports: When there's no actual conflict.
I'm seeing a similar issue. I had a disk full issue on a bunch of nodes (unrelated). Some nodes had the daemonsets removed. Other's didn't. The issue is, once I've fixed this, I can't have the nodes reschedule DaemonSets. Short of deleting the node and then restarting kubelet, which isn't much fun.
I've just run into a similar looking issue. I have a DS called with a node selector: There are three nodes with such label: But Kubernetes thinks the DS should only run on two nodes: The DS doesn't run on : After running and rebooting the node, the DS runs as expected on all three nodes: This was working fine for months; it only broke recently, after we upgraded from Kubernetes 1.3 to 1.5. I can't say for sure those things are related though, because I don't know exactly what the DS stopped working correctly.
I'll leave the milestone on this one since it looks like it's still getting attention. clear the milestone if it is not release blocking
I actually expect that this is very much improved in 1.6 so I'll drop the milestone and wait for feedback once people start testing 1.6. cc
Let's just open new issues when people give feedback on 1.6.
For people ending here with a 1.5 cluster and dreading to replace nodes: it might help to just recreate the daemonset itself by using something like This worked for me to bring back a missing kube2iam pod on a node. Unfortunately I don't have the logs any more to see why it got lost in the first place.
Just ran into the same issue with K8s 1.6.4. I had a node being OOD and repaired it manually, when it came back healthy the ds was not scheduled there and the ds controller did not even try that. Fixed it using comment above () This is issue is really bad when the ds in question is for example Calico, which is needed for the pod networking.
I see this happening on 1.6 clusters. How should I debug?
like I said (comment) helps on 1.6 Otherwise, now that we have DS updates with 1.7 an update might trigger the logic, too. Haven't tested that, yet, though.
Seeing this with Openshift 3.7 / K8s 1.7 EDIT: root cause for me was related to a taint on some nodes.
We ran into this problem with v1.6.13 The instructions in this make the daemonset pods start at all nodes. But even after a delete and recreate, I think the DaemonSet is still left in a wrong state Even though all the pods have started daemonset thinks there are 0 desired replicas.
Just hit this problem with v1.14.1. Deployed some identical servers (apart from hostname/IP obviously) from the same configuration management but one was not getting DaemonSets scheduled on it. Comment resolved the issue for us. Still strange it happened and why it only happened to one of them.
same problem with v1.13.5, experienced it over a dozen of times; had to either delete node from apiserver and restart kubelet or kill the kube-controller-manager that's the leader.
We had the same problem on v1.15.10 with Daemon Set only using 3/9 available nodes. After attempting all sorts of solutions. The fix was to roll the nodes and let the Daemon Set find the new nodes.
I had this same problem and managed to solve it. The root cause for me was that one of my nodes had a taint, due to problematic mounting of an AWS EBS volume (related to this other issue: ). The simplest solution was to "force detach" the problematic volume on the AWS console, and afterwards remove the taint from the node: (watch out for the trailing dash symbol in the command above, which means to remove a taint). Another solution if everything else fails is to drain the node and remove it from the cluster.

Discussion:
} _, notFittingCPU, notFittingMemory := predicates.CheckPodsExceedingFreeResources(pods, node.Status.Allocatable) if len(notFittingCPU)+len(notFittingMemory) != 0 { <ins> dsc.eventRecorder.Eventf(ds, api.EventTypeNormal, "FailedPlacement", "failed to place pod on %q: insufficent free resources", node.ObjectMeta.Name) </ins> return false } ports := sets.String{} for _, pod := range pods { if errs := validation.AccumulateUniqueHostPorts(pod.Spec.Containers, &ports, field.NewPath("spec", "containers")); len(errs) > 0 { <ins> dsc.eventRecorder.Eventf(ds, api.EventTypeNormal, "FailedPlacement", "failed to place pod on %q: host port conflict", node.ObjectMeta.Name) </ins> return false } }