Change:
https://k8s- This test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all. /kind bug /priority failing-test /sig storage /assign /cc /milestone v1.12
looks like some permissions issue after upgrade /assign
has triaged this to an understanding that the failures happen because the pod is scheduled on the master and getting logs through the method they're using is disabled on master nodes. This makes me think this is similar to and we have non-release-blocking issues in test cases not being quite robust enough now in the face of node taints.
It is interesting that in some places this test is always failing, some it is always succeeding, and even in one it flakes but with a high success rate: https://k8s-
Adding on to comment: The pod that we are creating does not have any special "master schedulable" annotations or tolerations, so from my understanding this should still be not scheduled to the master for any reason. The Pod Spec in question: Potentially relevant scheduler logs.
The fix mentioned in is only handling DaemonSets. I don't think it will fix this issue.
Do we need to get sig-scheduling involved to see why the master is showing up as a choice for this pod?
/sig scheduling Can you triage this for 1.12 release blocking?
It seems that this pvc-tester- pod is only scheduled to master in every run which caused the test to fail consistently.
However in this test suite, it seems like the test passes most of the time (pod is scheduled to a node), then occasionally flakes (pod scheduled to master).: https://k8s- Whereas in this one it doesn't seem to fail at all!: https://k8s- I talked to about why the behavior might be different across different test suites but we were unable to come up with anything concrete.
Is the master node in your test environment tainted properly?
When we were doing unrelated development with DaemonSets, we noticed a similar issue where our DaemonSet pod would get scheduled on gce environment using hack/cluster-, and we noticed masters brought up using that script were not tainted
What release was the tainting master logic ?
cc we had similar issue that gpu driver installer is being scheduled on master at , can also be related?
Not sure if this is a source of truth on the cluster but the test artifacts here on a successful run show that the master node doesn't have a taint: and on a failed run ALSO no taint: But I'm not really sure where this file comes from. And whether this is the cluster before or after the upgrade
ie:
Don't think will help. It looks specific to gpu labeling
Also ^^ is GKE specific, and GKE docu says no taints are inherited to a pool from the cluster, so shouldn't be something from that direction.
Some log snippets around the area of interest:
/priority critical-urgent
I think the fix is obvious: we need to taint the master node on the gce master setup in cluster/. What I still don't understand is how did we get this far without anything else failing, and why only this one test?
Do taints persist across master upgrades, or do they need to be re-applied again?
Based on Slack discussion and the triage so far, this looks like it's a case of unfortunate cluster/* directory bit rot. We can add the taint and have time before the release still to see clear CI signal, but I would not rate this as a release blocker, especially considering node-"" appears to have been introduced in 1.6.
/assign I'm going to contribute in my daytime :)
I think there's what happened: only update status when upgrading from 1.11 to 1.13, as the node was previously registered (1.11); so is not updated, including unschedulable taint. scheduler is 1.13 now, TaintNodesByCondition is enabled by default and NodeConditionCheck is disabled by default, so it ignores controller manager is 1.13 now, it'll taint nodes according to node's configurations, e.g. . Because (including unschedulable) is not updated by kubelet when upgrading, it has to dependent on controller manager (nodelifecycle) to update it which has race condition with kube-scheduler. There's an another issue that, we'll reject taints update to Node in noderestriction admission controller :( For the solution, I think we need to patch (at least ) and add a whitelist to noderestriction admission controller for known taints key; we can not use to start the kubelet, as it also be skipped (only update node status). Refer to the following logs for what happened: All logs are from https://k8s- Build- The taint of master; the taint will include if tainted by controller manager. scheduler-
/sig node
if no comments on the solution, i will open a pr for it.
cc on relaxing node restriction for this upgrade case Could another potential solution be to keep the previous scheduler node.unschedulable logic for one more release in order to handle this 2 version upgrade scenario?
Actually to clarify, is this both a 1.10 to 1.12 and 1.11 to 1.12 issue?
please get a PR started so we can get review happening ASAP
Thanks, for the investigation. Initially, I wanted to suggest to keep the logic of checking node condition in the scheduler along side TaintNodeByCondition for one more release, but I guess there would be conflicts. For example, a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition. So, looks like we'll have to fix the tainting logic.
can you clarify on what fixing the tainting logic would entail? If I understand your example, wouldn't the Pod object need to be updated to also tolerate the taint that the condition got converted to?
I was specifically referring to what suggested: "I think we need to patch node (at least node.Spec.Taints) and add a whitelist to noderestriction admission controller for known taints key." Yes, the pod needs to have the right toleration for the taint. We already create our critical system pods with proper tolerations for certain node taints, such as unreachable, etc.
I'm going back to feeling this is actually a release blocking upgrade issue. please clarify in more detail your concern for breakages, the possible paths for resolution, and preferred path.
I agree with you that this is release blocking. We commented at the same time. I mentioned above that I think we need to address tainting of the nodes in kubelet.
Who's the right owner for this now? I'm assuming we've lost Klaus to the weekend in his timezone, but some of us still have a few work hours yet on Friday...
I already pinged Dawn as our Node guru, but didn't get any reply. She might be on vacation. Let me see if I can find other folks here.
cc for what we can do on the node side. cc for the security/policy related matters associated with adding taints for node condition
help please
regarding your example "a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition", could this be an API compatibility issue? Will a pod that tolerates a node condition in 1.11 fail to schedule in 1.12 because the scheduler no longer looks at node conditions and only node taints?
Pods were not able to tolerate node conditions by setting some API fields. We had some annotations, such as pod critical annotation and other hard-coded logic in DS controller, etc. that would have the same effect of tolerating conditions, but they were not in our API.
Thanks for the explanation. It looks like the critical pod annotation was alpha, so does not have a backwards compatibility requirement, and we've fixed all the known users of it.
xref: - cc Should this be handled by the node controller? I.e. if the Unschedulable bool is set on the node, add the unschedulable taint?
I don't believe the kubelet has the ability to update its taints anymore. Assuming this is the case, this seems like a version skew problem between master components. I.E. the 1.11 controller manager isn't compatible with the 1.13 scheduler because it doesn't add taints that the scheduler requires. This means that if the scheduler is upgraded before the controller manager (or before it has the chance to perform updates), we can run into issues. I think making the scheduler respect conditions in 1.13, and thus making it compatible with the 1.11 controller manager, seems like the best remedy to this. Regardless of this issue, user pods would need to include both the critical annotation as well as the toleration in order to maintain the same behavior across releases that transition from conditions to taints.
Oh, I see that the node controller is already doing that, but there's a race condition (), sorry I missed that earlier.
Yes, translation of conditions and unschedulable field to taints is done by the controller
I wonder if we can change our upgrade flow to ensure that the controller manager is upgraded and becomes stable (adds the taints) before the scheduler is upgraded.
The kubelet has never required the ability to update taints. Was that a typo, supposed to be 1.12? Have we ever supported two version skew between those components? Under what circumstances do we upgrade to 1.13 scheduler while a 1.11 controller manager is still running? Same question about two version skew between these two components. That's more concerning… is that something all user pods would be exposed to and required to change?
+1 to suggestion in to unblock the release
What about HA master upgrade? scheduler and controller-manager each have independent leader elections, so during a HA rolling upgrade, the current leaders for scheduler and controller managers could be on different releases in either direction
I wouldn't expect an upgrade from 1.12 to 1.13 to begin until the upgrade from 1.11 to 1.12 was complete/successful
Thanks for correcting me. It wasn't a typo, but just a lack of knowledge about how multi-version upgrades are performed. In that case we need to support 1 version skew. Yeah, it should be 1.12, not 1.13. But generally, the newer scheduler should still be compatible with the state left by the previous version's controller manager, even if it isn't running anymore. That's more concerning… is that something all user pods would be exposed to and required to change? Actually, I agree. have we officially deprecated the critical pod annotation? We should ideally add the correct tolerations to pods with the critical pod annotation to ensure backwards compatibility.
Bobby commented that the system pods have already been corrected. I also noted that the critical pod annotation is alpha.
I mean doing an HA rolling upgrade from 1.11 to 1.12. Let's say we have 3 masters, A, B, C, all on 1.11. is upgraded to 1.12, B and C are still on 1.11. leader election could pick scheduler A as the leader, controller-manager leader election could pick controller-manager B as the leader. Now we have version skew between master components.
That feature (critical pods) was an alpha feature and we have already replaced it. The dilemma is that I am not so sure if critical pods were the only kind of pods with a "hack" for tolerating node conditions. Let me dig a bit more.
One version of skew, absolutely. comment made me concerned two versions of skew support were being claimed/proposed
I looked the scheduler code and the DaemonSet controller code to find if adding "CheckNodeCondition" could cause issues in presence of TaintNodeByCondition and I didn't find anything particularly concerning. So, I created . Let's see if it breaks tests.
that will break schedule daemonset by scheduler. and registerwithtaints of kubelet is not worked in upgrade.
I suspected that it would conflict with pods which now have tolerations and the scheduler should ignore node conditions for, but I thought it would be limited to critical pods. Thanks for pointing it out. I closed the PR. As an alternative solution, I suggested changing the order of upgrading server components in our upgrade process, so that the controller manager is upgraded before the scheduler, but I am not sure how easy it is to implement. Do you have any alternative fix in mind?
unschedulable is taint by controller & kubelet; taints node with unschedulable to avoid race condition between controller and scheduler, it dependents on of kubelet to taint when start up; but does not work in upgrade case as mentioned above (we only update status when upgrading). IMO, it should be fixed; it'll also fix upgrade e2e test but may not work in real scenario which upgrade master firstly. let me think about it, I'll propose one today.
The race that is important to resolve at creation time is that of a new node object appearing untainted. That race does not apply in an upgrade scenario, where the node object already exists, and can already be observed and acted upon. Relying on the condition-to-taint controller in upgrade scenarios is reasonable.
The race after kubelet start up is well known; but we did not handle startup case well. I think what we're doing is to deprecate , and replace it with unschedulable taint. We need to keep backward compatibility for it.

Discussion:
return false, []algorithm.PredicateFailureReason{ErrNodeUnknownCondition}, nil } <del> if nodeInfo.Node().Spec.Unschedulable { </del> <ins> // If pod tolerate unschedulable taint, it's also tolerate `node.Spec.Unschedulable`. podToleratesUnschedulable := v1helper.TolerationsTolerateTaint(pod.Spec.Tolerations, &v1.Taint{ Key:    algorithm.TaintNodeUnschedulable, Effect: v1.TaintEffectNoSchedule, }) // TODO (k82cn): deprecates `node.Spec.Unschedulable` in 1.13. if nodeInfo.Node().Spec.Unschedulable && !podToleratesUnschedulable { </ins> return false, []algorithm.PredicateFailureReason{ErrNodeUnschedulable}, nil }

os.Setenv(KubeMaxPDVols, previousValue) } } <ins> func TestCheckNodeUnschedulablePredicate(t *testing.T) { testCases := []struct { name string pod  *v1.Pod node *v1.Node fit  bool }{ { name: "Does not schedule pod to unschedulable node (node.Spec.Unschedulable==true)", pod:  &v1.Pod{}, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: true, }, }, fit: false, }, { name: "Schedule pod to normal node", pod:  &v1.Pod{}, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: false, }, }, fit: true, }, { name: "Schedule pod with toleration to unschedulable node (node.Spec.Unschedulable==true)", pod: &v1.Pod{ Spec: v1.PodSpec{ Tolerations: []v1.Toleration{ { Key:    algorithm.TaintNodeUnschedulable, Effect: v1.TaintEffectNoSchedule, }, }, }, }, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: true, }, }, fit: true, }, } for _, test := range testCases { nodeInfo := schedulercache.NewNodeInfo() nodeInfo.SetNode(test.node) fit, _, err := CheckNodeUnschedulablePredicate(test.pod, nil, nodeInfo) if err != nil { t.Fatalf("Failed to check node unschedulable: %v", err) } if fit != test.fit { t.Errorf("Unexpected fit: expected %v, got %v", test.fit, fit) } } } </ins>

// Fit is determined based on whether a pod can tolerate all of the node's taints factory.RegisterMandatoryFitPredicate(predicates.PodToleratesNodeTaintsPred, predicates.PodToleratesNodeTaints) <ins> // Fit is determined based on whether a pod can tolerate unschedulable of node factory.RegisterMandatoryFitPredicate(predicates.CheckNodeUnschedulablePred, predicates.CheckNodeUnschedulablePredicate) </ins> // Insert Key "PodToleratesNodeTaints" and "CheckNodeUnschedulable" To All Algorithm Provider // The key will insert to all providers which in algorithmProviderMap[] // if you just want insert to specific provider, call func InsertPredicateKeyToAlgoProvider() factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.PodToleratesNodeTaintsPred) <ins> factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.CheckNodeUnschedulablePred) </ins> glog.Warningf("TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory") }