Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it. The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign

Discussion:
return dsc.updateDaemonSetStatus(ds) } <ins> // hasIntentionalPredicatesReasons checks if any of the given predicate failure reasons // is intentional. func hasIntentionalPredicatesReasons(reasons []algorithm.PredicateFailureReason) bool { for _, r := range reasons { switch reason := r.(type) { case *predicates.PredicateFailureError: switch reason { // intentional case predicates.ErrNodeSelectorNotMatch, predicates.ErrPodNotMatchHostName, predicates.ErrNodeLabelPresenceViolated, // this one is probably intentional since it's a workaround for not having // pod hard anti affinity. predicates.ErrPodNotFitsHostPorts, // DaemonSet is expected to respect taints and tolerations predicates.ErrTaintsTolerationsNotMatch: return true } } } return false } </ins> // nodeShouldRunDaemonPod checks a set of preconditions against a (node,daemonset) and returns a // summary. Returned booleans are: // * wantToRun:

return false, false, false, err } <ins> // Return directly if there is any intentional predicate failure reason, so that daemonset controller skips // checking other predicate failures, such as InsufficientResourceError and unintentional errors. if hasIntentionalPredicatesReasons(reasons) { return false, false, false, nil } </ins> for _, r := range reasons { glog.V(4).Infof("DaemonSet Predicates failed on node %s for ds '%s/%s' for reason: %v", node.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Name, r.GetReason()) switch reason := r.(type) {

shouldSchedule = false case *predicates.PredicateFailureError: var emitEvent bool <del> // we try to partition predicates into two partitions here: intentional on the part of the operator and not. </del> switch reason { <del> // intentional case predicates.ErrNodeSelectorNotMatch, predicates.ErrPodNotMatchHostName, predicates.ErrNodeLabelPresenceViolated, // this one is probably intentional since it's a workaround for not having // pod hard anti affinity. predicates.ErrPodNotFitsHostPorts, // DaemonSet is expected to respect taints and tolerations predicates.ErrTaintsTolerationsNotMatch: wantToRun, shouldSchedule, shouldContinueRunning = false, false, false // unintentional </del> <ins> // unintentional predicates reasons need to be fired out to event. </ins> case predicates.ErrDiskConflict, predicates.ErrVolumeZoneConflict,

type daemonSetsController struct { *DaemonSetsController <del> dsStore   cache.Store podStore  cache.Store nodeStore cache.Store </del> <ins> dsStore      cache.Store podStore     cache.Store nodeStore    cache.Store fakeRecorder *record.FakeRecorder </ins> } func newTestController(initialObjects ...runtime.Object) (*daemonSetsController, *fakePodControl, *fake.Clientset) {

informerFactory.Core().V1().Nodes(), clientset, ) <del> manager.eventRecorder = record.NewFakeRecorder(100) </del> <ins> fakeRecorder := record.NewFakeRecorder(100) manager.eventRecorder = fakeRecorder </ins> manager.podStoreSynced = alwaysReady manager.nodeStoreSynced = alwaysReady

informerFactory.Extensions().V1beta1().DaemonSets().Informer().GetStore(), informerFactory.Core().V1().Pods().Informer().GetStore(), informerFactory.Core().V1().Nodes().Informer().GetStore(), <ins> fakeRecorder, </ins> }, podControl, clientset }

} } <ins> func resourcePodSpecWithoutNodeName(memory, cpu string) v1.PodSpec { return v1.PodSpec{ Containers: []v1.Container{{ Resources: v1.ResourceRequirements{ Requests: allocatableResources(memory, cpu), }, }}, } } </ins> func allocatableResources(memory, cpu string) v1.ResourceList { return v1.ResourceList{ v1.ResourceMemory: resource.MustParse(memory),

} } <ins> // DaemonSets should only place onto nodes with sufficient free resource and matched node selector func TestInsufficientCapacityNodeSufficientCapacityWithNodeLabelDaemonLaunchPod(t *testing.T) { podSpec := resourcePodSpecWithoutNodeName("50M", "75m") ds := newDaemonSet("foo") ds.Spec.Template.Spec = podSpec ds.Spec.Template.Spec.NodeSelector = simpleNodeLabel manager, podControl, _ := newTestController(ds) node1 := newNode("not-enough-resource", nil) node1.Status.Allocatable = allocatableResources("10M", "20m") node2 := newNode("enough-resource", simpleNodeLabel) node2.Status.Allocatable = allocatableResources("100M", "200m") manager.nodeStore.Add(node1) manager.nodeStore.Add(node2) manager.dsStore.Add(ds) syncAndValidateDaemonSets(t, manager, ds, podControl, 1, 0) // we do not expect any event for insufficient free resource if len(manager.fakeRecorder.Events) != 0 { t.Fatalf("unexpected events, got %v, expected %v: %+v", len(manager.fakeRecorder.Events), 0, manager.fakeRecorder.Events) } } </ins> func TestSufficientCapacityWithTerminatedPodsDaemonLaunchesPod(t *testing.T) { for _, strategy := range updateStrategies() { podSpec := resourcePodSpec("too-much-mem", "75M", "75m")