Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned . Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to . hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .

Discussion:
"container/list" ) <del> // removeFromFIFOFunc removes a designated element from the list. // The complexity of the runtime cost is O(1) // It returns the request that has been removed from the list, // it returns nil if the request has already been removed. </del> <ins> // removeFromFIFOFunc removes a designated element from the list // if that element is in the list. // The complexity of the runtime cost is O(1). // The returned value is the element removed, if indeed one was removed, // otherwise `nil`. </ins> type removeFromFIFOFunc func() *request // walkFunc is called for each request in the list in the // oldest -> newest order. // ok: if walkFunc returns false then the iteration stops immediately. <ins> // walkFunc may remove the given request from the fifo, // but may not mutate the fifo in any othe way. </ins> type walkFunc func(*request) (ok bool) // Internal interface to abstract out the implementation details

} func (l *requestFIFO) Walk(f walkFunc) { <del> for current := l.Front(); current != nil; current = current.Next() { </del> <ins> var next *list.Element for current := l.Front(); current != nil; current = next { next = current.Next() // f is allowed to remove current </ins> if r, ok := current.Value.(*request); ok { if !f(r) { return

removeFn = append(removeFn, list.Enqueue(arrival[i])) } <del> dequeued := make([]*request, 0) for _, f := range removeFn { dequeued = append(dequeued, f()) </del> <ins> expected := append([]*request{}, arrival...) for idx, f := range removeFn { if a := f(); a != arrival[idx] { t.Errorf("Removal %d returned %v instead of expected pointer", idx, a) } if a := f(); a != nil { t.Errorf("Redundant removal %d returned %v instead of expected nil", idx, a) } expected = expected[1:] actual := walkAll(list) verifyOrder(t, expected, actual) </ins> } if list.Length() != 0 { t.Errorf("Expected length: %d, but got: %d)", 0, list.Length()) } <del> verifyOrder(t, arrival, dequeued) </del> } func TestFIFORemoveFromFIFOFunc(t *testing.T) {

removeFn = append(removeFn, list.Enqueue(arrival[i])) } <del> dequeued := make([]*request, 0) </del> <ins> expected := append([]*request{}, arrival...) </ins> r := rand.New(rand.NewSource(time.Now().UnixNano())) <del> randomIndices := r.Perm(len(removeFn)) t.Logf("Random remove order: %v", randomIndices) for i := range randomIndices { dequeued = append(dequeued, removeFn[i]()) </del> <ins> for range arrival { idx := r.Intn(len(expected)) t.Logf("Removing random index %d", idx) if e, a := expected[idx], removeFn[idx](); e != a { t.Errorf("Removal of %d returned %v instead of expected pointer %v", idx, a, e) } if e, a := (*request)(nil), removeFn[idx](); e != a { t.Errorf("Redundant removal of %d returned %v instead of expected nil pointer", idx, a) } expected = append(expected[:idx], expected[idx+1:]...) actual := walkAll(list) verifyOrder(t, expected, actual) removeFn = append(removeFn[:idx], removeFn[idx+1:]...) </ins> } if list.Length() != 0 { t.Errorf("Expected length: %d, but got: %d)", 0, list.Length()) } <del> verifyOrder(t, arrival, dequeued) </del> } func TestFIFOWithRemoveIsIdempotent(t *testing.T) {

// can short circuit loop (break) if oldest requests are not timing out // as newer requests also will not have timed out <del> // now - requestWaitLimit = waitLimit waitLimit := now.Add(-qs.qCfg.RequestWaitLimit) </del> <ins> // now - requestWaitLimit = arrivalLimit arrivalLimit := now.Add(-qs.qCfg.RequestWaitLimit) </ins> reqs.Walk(func(req *request) bool { <del> if waitLimit.After(req.arrivalTime) { req.decision.Set(decisionReject) timeoutCount++ metrics.AddRequestsInQueues(req.ctx, qs.qCfg.Name, req.fsName, -1) req.NoteQueued(false) </del> <ins> if arrivalLimit.After(req.arrivalTime) { if req.decision.Set(decisionReject) && req.removeFromQueueLocked() != nil { timeoutCount++ req.NoteQueued(false) metrics.AddRequestsInQueues(req.ctx, qs.qCfg.Name, req.fsName, -1) } </ins> // we need to check if the next request has timed out. return true } // since reqs are sorted oldest -> newest, we are done here. return false }) // remove timed out requests from queue if timeoutCount > 0 { <del> // The number of requests we have timed out is timeoutCount, // so, let's dequeue the exact number of requests for this queue. for i := 0; i < timeoutCount; i++ { queue.requests.Dequeue() } // decrement the # of requestsEnqueued </del> qs.totRequestsWaiting -= timeoutCount qs.obsPair.RequestsWaiting.Add(float64(-timeoutCount)) }

qs.obsPair.RequestsWaiting.Add(1) } <del> // dispatchAsMuchAsPossibleLocked runs a loop, as long as there // are non-empty queues and the number currently executing is less than the // assured concurrency value.  The body of the loop uses the fair queuing // technique to pick a queue, dequeue the request at the head of that // queue, increment the count of the number executing, and send true // to the request's channel. </del> <ins> // dispatchAsMuchAsPossibleLocked does as many dispatches as possible now. </ins> func (qs *queueSet) dispatchAsMuchAsPossibleLocked() { <del> for qs.totRequestsWaiting != 0 && qs.totSeatsInUse < qs.dCfg.ConcurrencyLimit { ok := qs.dispatchLocked() if !ok { break } </del> <ins> for qs.totRequestsWaiting != 0 && qs.totSeatsInUse < qs.dCfg.ConcurrencyLimit && qs.dispatchLocked() { </ins> } }

// dispatchLocked uses the Fair Queuing for Server Requests method to // select a queue and dispatch the oldest request in that queue.  The <del> // return value indicates whether a request was dispatched; this will // be false when there are no requests waiting in any queue. </del> <ins> // return value indicates whether a request was dequeued; this will // be false when either all queues are empty or the request at the head // of the next queue cannot be dispatched. </ins> func (qs *queueSet) dispatchLocked() bool { queue, request := qs.findDispatchQueueLocked() if queue == nil {

if request == nil { // This should never happen.  But if it does... return false } <ins> qs.totRequestsWaiting-- metrics.AddRequestsInQueues(request.ctx, qs.qCfg.Name, request.fsName, -1) request.NoteQueued(false) qs.obsPair.RequestsWaiting.Add(-1) defer qs.boundNextDispatchLocked(queue) if !request.decision.Set(decisionExecute) { return true } </ins> request.startTime = qs.clock.Now() // At this moment the request leaves its queue and starts // executing.  We do not recognize any interim state between // "queued" and "executing".  While that means "executing" // includes a little overhead from this package, this is not a // problem because other overhead is also included. <del> qs.totRequestsWaiting-- </del> qs.totRequestsExecuting++ qs.totSeatsInUse += request.MaxSeats() queue.requestsExecuting++ queue.seatsInUse += request.MaxSeats() <del> metrics.AddRequestsInQueues(request.ctx, qs.qCfg.Name, request.fsName, -1) request.NoteQueued(false) </del> metrics.AddRequestsExecuting(request.ctx, qs.qCfg.Name, request.fsName, 1) metrics.AddRequestConcurrencyInUse(qs.qCfg.Name, request.fsName, request.MaxSeats()) <del> qs.obsPair.RequestsWaiting.Add(-1) </del> qs.obsPair.RequestsExecuting.Add(1) if klog.V(6).Enabled() { klog.Infof("QS(%s) at t=%s R=%v: dispatching request %#+v %#+v work %v from queue %d with start R %v, queue will have %d waiting & %d requests occupying %d seats, set will have %d seats occupied",

} // When a request is dequeued for service -> qs.virtualStart += G * width queue.nextDispatchR += request.totalWork() <del> qs.boundNextDispatchLocked(queue) request.decision.Set(decisionExecute) </del> return true }