Change:
server, requestGetter := newHTTP2ServerWithClient(handler, requestTimeout*2) defer server.Close() <del> // we send two requests, each with a client timeout of requestTimeout*2 seconds // this ensures the test does not block indefinitely if the server does not respond. //  - first request (expected to timeout as designed) sent from a new goroutine //  - second request (expected to be enqueued) is sent from a new goroutine </del> <ins> // This test involves two requests sent to the same priority level, which has 1 queue and // a concurrency limit of 1.  The handler chain include the timeout filter. // Each request is sent from a separate goroutine, with a client-side timeout that is // double the timeout filter's limit. // The first request should get dispatched immediately; execution (a) starts with closing // the channel that triggers the second client goroutine to send its request and then (b) // waits for both client goroutines to have gotten a response (expected to be timeouts). // The second request sits in the queue until the timeout filter does its thing, which // it does concurrently to both requests.  For the first request this should make the client // get a timeout response without directly affecting execution.  For the second request, the // fact that the timeout filter closes the request's Context.Done() causes the request to be // promptly ejected from its queue.  The goroutine doing the APF handling writes an HTTP // response message with status 429. // The timeout handler invokes its inner handler in one goroutine while reacting to the // passage of time in its original goroutine.  That reaction to a time out consists of either // (a) writing an HTTP response message with status 504 to indicate the timeout or (b) doing an // HTTP/2 stream close; the latter is done if either the connection has been "hijacked" or some // other goroutine (e.g., the one running the inner handler) has started to write a response. // In the scenario tested here, there is thus a race between two goroutines to respond to // the second request and any of their responses is allowed by the test. </ins> firstReqResultCh, secondReqResultCh := make(chan result, 1), make(chan result, 1) go func() { defer close(firstReqRoundTripDoneCh) <del> t.Logf("Sending request: %q", firstRequestTimesOutPath) </del> <ins> t.Logf("At %s, Sending request: %q", time.Now().Format(timeFmt), firstRequestTimesOutPath) </ins> resp, err := requestGetter(firstRequestTimesOutPath) <del> t.Logf("RoundTrip of request: %q has completed", firstRequestTimesOutPath) </del> <ins> t.Logf("At %s, RoundTrip of request: %q has completed", time.Now().Format(timeFmt), firstRequestTimesOutPath) </ins> firstReqResultCh <- result{err: err, response: resp} }() go func() {

Discussion:
pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.