Change:
Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!

Discussion:
// undesired becomes completely unused, all the config objects are // read and processed as a whole. <ins> // The funcs in this package follow the naming convention that the suffix // "Locked" means the relevant mutex must be locked at the start of each // call and will be locked upon return.  For a configController, the // suffix "ReadLocked" stipulates a read lock while just "Locked" // stipulates a full lock.  Absence of either suffix means that either // (a) the lock must NOT be held at call time and will not be held // upon return or (b) locking is irrelevant. </ins> // StartFunction begins the process of handling a request.  If the // request gets queued then this function uses the given hashValue as // the source of entropy as it shuffle-shards the request into a

// requestWaitLimit comes from server configuration. requestWaitLimit time.Duration <ins> // watchTracker implements the necessary WatchTracker interface. WatchTracker // the most recent update attempts, ordered by increasing age. // Consumer trims to keep only the last minute's worth of entries. // The controller uses this to limit itself to at most six updates // to a given FlowSchema in any minute. // This may only be accessed from the one and only worker goroutine. mostRecentUpdates []updateAttempt </ins> // This must be locked while accessing flowSchemas or <del> // priorityLevelStates. lock sync.Mutex </del> <ins> // priorityLevelStates.  A lock for writing is needed // for writing to any of the following: // - the flowSchemas field // - the slice held in the flowSchemas field // - the priorityLevelStates field // - the map held in the priorityLevelStates field // - any field of a priorityLevelState held in that map lock sync.RWMutex </ins> // flowSchemas holds the flow schema objects, sorted by increasing // numerical (decreasing logical) matching precedence.  Every

// name to the state for that level.  Every name referenced from a // member of `flowSchemas` has an entry here. priorityLevelStates map[string]*priorityLevelState <del> // the most recent update attempts, ordered by increasing age. // Consumer trims to keep only the last minute's worth of entries. // The controller uses this to limit itself to at most six updates // to a given FlowSchema in any minute. // This may only be accessed from the one and only worker goroutine. mostRecentUpdates []updateAttempt // watchTracker implements the necessary WatchTracker interface. WatchTracker </del> } type updateAttempt struct {

} func (cfgCtlr *configController) updateObservations() { <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> for _, plc := range cfgCtlr.priorityLevelStates { if plc.queues != nil { plc.queues.UpdateObservations()

// waiting in its queue, or `Time{}` if this did not happen. func (cfgCtlr *configController) startRequest(ctx context.Context, rd RequestDigest, queueNoteFn fq.QueueNoteFn) (fs *flowcontrol.FlowSchema, pl *flowcontrol.PriorityLevelConfiguration, isExempt bool, req fq.Request, startWaitingTime time.Time, flowDistinguisher string) { klog.V(7).Infof("startRequest(%#+v)", rd) <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> var selectedFlowSchema, catchAllFlowSchema *flowcontrol.FlowSchema for _, fs := range cfgCtlr.flowSchemas { if matchesFlowSchema(rd, fs) {

klog.V(7).Infof("startRequest(%#+v) => fsName=%q, distMethod=%#+v, plName=%q, numQueues=%d", rd, selectedFlowSchema.Name, selectedFlowSchema.Spec.DistinguisherMethod, plName, numQueues) req, idle := plState.queues.StartRequest(ctx, &rd.WorkEstimate, hashValue, flowDistinguisher, selectedFlowSchema.Name, rd.RequestInfo, rd.User, queueNoteFn) if idle { <del> cfgCtlr.maybeReapLocked(plName, plState) </del> <ins> cfgCtlr.maybeReapReadLocked(plName, plState) </ins> } return selectedFlowSchema, plState.pl, false, req, startWaitingTime, flowDistinguisher }

// priority level if it has no more use.  Call this after getting a // clue that the given priority level is undesired and idle. func (cfgCtlr *configController) maybeReap(plName string) { <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> plState := cfgCtlr.priorityLevelStates[plName] if plState == nil { klog.V(7).Infof("plName=%s, plState==nil", plName)

// it has no more use.  Call this if both (1) plState.queues is // non-nil and reported being idle, and (2) cfgCtlr's lock has not // been released since then. <del> func (cfgCtlr *configController) maybeReapLocked(plName string, plState *priorityLevelState) { </del> <ins> func (cfgCtlr *configController) maybeReapReadLocked(plName string, plState *priorityLevelState) { </ins> if !(plState.quiescing && plState.numPending == 0) { return }