Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet. If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.

Discussion:
} } <ins> // TestMatchNodeSelectorTermsStateless ensures MatchNodeSelectorTerms() // is invoked in a "stateless" manner, i.e. nodeSelectorTerms should NOT // be deeply modifed after invoking func TestMatchNodeSelectorTermsStateless(t *testing.T) { type args struct { nodeSelectorTerms []v1.NodeSelectorTerm nodeLabels        labels.Set nodeFields        fields.Set } tests := []struct { name string args args want []v1.NodeSelectorTerm }{ { name: "nil terms", args: args{ nodeSelectorTerms: nil, nodeLabels:        nil, nodeFields:        nil, }, want: nil, }, { name: "nodeLabels: preordered matchExpressions and nil matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, }, }, nodeLabels: map[string]string{"label_1": "label_1_val"}, nodeFields: nil, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, }, }, }, { name: "nodeLabels: unordered matchExpressions and nil matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, }, }, nodeLabels: map[string]string{"label_1": "label_1_val"}, nodeFields: nil, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, }, }, }, { name: "nodeFields: nil matchExpressions and preordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, nodeLabels: nil, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, }, { name: "nodeFields: nil matchExpressions and unordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, nodeLabels: nil, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, }, { name: "nodeLabels and nodeFields: ordered matchExpressions and ordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, nodeLabels: map[string]string{ "label_1": "label_1_val", }, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, }, { name: "nodeLabels and nodeFields: ordered matchExpressions and unordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, nodeLabels: map[string]string{ "label_1": "label_1_val", }, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_1_val", "label_2_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, }, { name: "nodeLabels and nodeFields: unordered matchExpressions and ordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, nodeLabels: map[string]string{ "label_1": "label_1_val", }, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_1", "host_2"}, }}, }, }, }, { name: "nodeLabels and nodeFields: unordered matchExpressions and unordered matchFields", args: args{ nodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, nodeLabels: map[string]string{ "label_1": "label_1_val", }, nodeFields: map[string]string{ "metadata.name": "host_1", }, }, want: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{{ Key:      "label_1", Operator: v1.NodeSelectorOpIn, Values:   []string{"label_2_val", "label_1_val"}, }}, MatchFields: []v1.NodeSelectorRequirement{{ Key:      "metadata.name", Operator: v1.NodeSelectorOpIn, Values:   []string{"host_2", "host_1"}, }}, }, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { MatchNodeSelectorTerms(tt.args.nodeSelectorTerms, tt.args.nodeLabels, tt.args.nodeFields) if !apiequality.Semantic.DeepEqual(tt.args.nodeSelectorTerms, tt.want) { // fail when tt.args.nodeSelectorTerms is deeply modified t.Errorf("MatchNodeSelectorTerms() got = %v, want %v", tt.args.nodeSelectorTerms, tt.want) } }) } } </ins> func TestMatchTopologySelectorTerms(t *testing.T) { type args struct { topologySelectorTerms []v1.TopologySelectorTerm

package v1 import ( <ins> "fmt" </ins> "reflect" "strings" "testing"

} for i, tc := range tc { <ins> inCopy := tc.in.DeepCopy() </ins> out, err := LabelSelectorAsSelector(tc.in) <ins> // after calling LabelSelectorAsSelector, tc.in shouldn't be modified if !reflect.DeepEqual(inCopy, tc.in) { t.Errorf("[%v]expected:nt%#vnbut got:nt%#v", i, inCopy, tc.in) } </ins> if err == nil && tc.expectErr { t.Errorf("[%v]expected error but got none.", i) } if err != nil && !tc.expectErr { t.Errorf("[%v]did not expect error but got: %v", i, err) } <del> if !reflect.DeepEqual(out, tc.out) { t.Errorf("[%v]expected:nt%+vnbut got:nt%+v", i, tc.out, out) </del> <ins> // fmt.Sprint() over String() as nil.String() will panic if fmt.Sprint(out) != fmt.Sprint(tc.out) { t.Errorf("[%v]expected:nt%snbut got:nt%s", i, fmt.Sprint(tc.out), fmt.Sprint(out)) </ins> } } }

return nil, err } } <del> sort.Strings(vals) </del> return &Requirement{key: key, operator: op, strValues: vals}, nil }

if len(r.strValues) == 1 { buffer.WriteString(r.strValues[0]) } else { // only > 1 since == 0 prohibited by NewRequirement <del> buffer.WriteString(strings.Join(r.strValues, ",")) </del> <ins> // normalizes value order on output, without mutating the in-memory selector representation // also avoids normalization when it is not required, and ensures we do not mutate shared data buffer.WriteString(strings.Join(safeSort(r.strValues), ",")) </ins> } switch r.operator {

return buffer.String() } <ins> // safeSort sort input strings without modification func safeSort(in []string) []string { if sort.StringsAreSorted(in) { return in } out := make([]string, len(in)) copy(out, in) sort.Strings(out) return out } </ins> // Add adds requirements to the selector. It copies the current selector returning a new one func (lsel internalSelector) Add(reqs ...Requirement) Selector { var sel internalSelector

} } } <ins> func TestSafeSort(t *testing.T) { tests := []struct { name   string in     []string inCopy []string want   []string }{ { name:   "nil strings", in:     nil, inCopy: nil, want:   nil, }, { name:   "ordered strings", in:     []string{"bar", "foo"}, inCopy: []string{"bar", "foo"}, want:   []string{"bar", "foo"}, }, { name:   "unordered strings", in:     []string{"foo", "bar"}, inCopy: []string{"foo", "bar"}, want:   []string{"bar", "foo"}, }, { name:   "duplicated strings", in:     []string{"foo", "bar", "foo", "bar"}, inCopy: []string{"foo", "bar", "foo", "bar"}, want:   []string{"bar", "bar", "foo", "foo"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := safeSort(tt.in); !reflect.DeepEqual(got, tt.want) { t.Errorf("safeSort() = %v, want %v", got, tt.want) } if !reflect.DeepEqual(tt.in, tt.inCopy) { t.Errorf("after safeSort(), input = %v, want %v", tt.in, tt.inCopy) } }) } } </ins>