Change:
return nil, err } <del> return &GenericWebhook{restClient, initialBackoff}, nil </del> <ins> return &GenericWebhook{restClient, initialBackoff, DefaultShouldRetry}, nil </ins> } // WithExponentialBackoff will retry webhookFn() up to 5 times with exponentially increasing backoff when <del> // it returns an error for which apierrors.SuggestsClientDelay() or apierrors.IsInternalError() returns true. </del> <ins> // it returns an error for which this GenericWebhook's ShouldRetry function returns true, confirming it to // be retriable. If no ShouldRetry has been defined for the webhook, then the default one is used (DefaultShouldRetry). </ins> func (g *GenericWebhook) WithExponentialBackoff(ctx context.Context, webhookFn func() rest.Result) rest.Result { var result rest.Result <ins> shouldRetry := g.ShouldRetry if shouldRetry == nil { shouldRetry = DefaultShouldRetry } </ins> WithExponentialBackoff(ctx, g.InitialBackoff, func() error { result = webhookFn() return result.Error() <del> }) </del> <ins> }, shouldRetry) </ins> return result } <del> // WithExponentialBackoff will retry webhookFn() up to 5 times with exponentially increasing backoff when // it returns an error for which apierrors.SuggestsClientDelay() or apierrors.IsInternalError() returns true. func WithExponentialBackoff(ctx context.Context, initialBackoff time.Duration, webhookFn func() error) error { </del> <ins> // WithExponentialBackoff will retry webhookFn up to 5 times with exponentially increasing backoff when // it returns an error for which shouldRetry returns true, confirming it to be retriable. func WithExponentialBackoff(ctx context.Context, initialBackoff time.Duration, webhookFn func() error, shouldRetry func(error) bool) error { </ins> backoff := wait.Backoff{ Duration: initialBackoff, Factor:   1.5,

Discussion:
What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery