Issue 1:
Problem

Some predicates may need precomputations or "preconditions" to be met before they can run.

As of now, those preconditions may or may not be gauranteed to exist in the predicateMetadata struct.

Some of those predicates may be able to bootstrap some of their own data (by redoing a query, if the predicate metadata is incomplete).

Solution

We should decide wether (1) predicates should try to create data which is precomputed and (2) if so, what the best way to do this is which is scalable and doesnt lead to duplication of error handling/querying.

Two possible solutions:

make any predicate's precompute function (as of now, really we only have one - but there may be others) stand alone - so that it can be called outside of the standard precomputation steps provided by the scheduler framework (this might help in corner cases - i.e. if a query fails randomly, the predicate has a retry built-in - but it will lead to duplication of logic).

Each predicates precompute function stays encapsulated, and predicates simply have preconditions of checking for missing data (this is simpler to maintain from the code perspective, but might be some corner cases where it hurts us). Once we decide, we would update the predicates_test functions where necessary to include test cases for when precomputation is enabled.

@wojtek-t Suggesting this as a follow on to #33763 .

Issue 2:
Currently in the existing implementation, how should a taint be unique (i.e. by key or <key, effect>) on a node is not clear,

server side validation don't actually check if there are duplicated taints any reject the request.

kubectl command has an implicit semantic that taint is unique by key.

# Update node 'foo' with a taint with key 'dedicated' and value 'special-user' and effect 'NoScheduleNoAdmitNoExecute'.
# If a taint with that key already exists, its value and effect are replaced as specified.
$ kubectl taint nodes foo dedicated=special-user:NoScheduleNoAdmitNoExecute

# Remove from node 'foo' the taint with key 'dedicated' if one exists.
$ kubectl taint nodes foo dedicated-

Discussed with @davidopp and @bgrant0607 , David's idea is:

We cannot make taint key be unique on a node; we can only make <taint, effect> be unique. The reason is that we want to be able to express all of these (I am copying this from my comment here):

don't schedule me there and evict me if I'm already running there

don't schedule me there but leave me bound if I'm already running there

schedule me there and leave me bound if I'm already running there

This means we need to decouple "no schedule" and "no execute" effects (let's ignore the NoAdmit part)

I think there are only two ways to do it:

(original plan) we have NoSchedule effect and NoScheduleNoExecute effect

(suggested new plan) we have NoSchedule effect and NoExecute effect

In either approach, you need to put two separate taints, one with each of the effects, on a node if you want to allow some pods to request only "no schedule" (i.e. leave me bound if already running) and some pods to request "no schedule and no execute." (The only difference between the two approaches is which tolerations you need for the different use cases, but in both approaches you need two different taints with the same key and different effects if you want to allow the full flexibility.)

As discussed, we should make taints unique by <key, effect> on a node, basically, the following two steps are needed:

add some validation check that reject duplicated taints that have the same <key, effect>

modify the existing semantic of kubectl taint command to:

# Update node 'foo' with a taint with key 'dedicated' and value 'special-user' and effect 'NoScheduleNoAdmit'.
# If a taint with that key and effect already exists, its value is replaced as specified.
$ kubectl taint nodes foo dedicated=special-user:NoScheduleNoAdmit

# Remove from node 'foo' the taint with key 'dedicated' and effect 'NoScheduleNoAdmit' if one exists.
$ kubectl taint nodes foo dedicated:NoScheduleNoAdmit-

If the effect is omitted in the kubectlcommand line, treat that as a wildcard (all effects). That would be especially useful for removing taints.

# Remove from node 'foo' all the taints with key 'dedicated'
$ kubectl taint nodes foo dedicated-