Change:
Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher. One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers. Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.

Discussion:
"//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/version:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <ins> "//staging/src/k8s.io/apiserver/pkg/authentication/user:go_default_library", </ins> "//staging/src/k8s.io/apiserver/pkg/endpoints/discovery:go_default_library", "//staging/src/k8s.io/apiserver/pkg/features:go_default_library", "//staging/src/k8s.io/apiserver/pkg/storage/names:go_default_library",

flowcontrol "k8s.io/api/flowcontrol/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apiserver/pkg/authentication/user" </ins> "k8s.io/client-go/rest" "k8s.io/kubernetes/test/e2e/framework" ) const ( requestConcurrencyLimitMetricName      = "apiserver_flowcontrol_request_concurrency_limit" <del> requestConcurrencyLimitMetricLabelName = "priorityLevel" </del> <ins> requestConcurrencyLimitMetricLabelName = "priority_level" </ins> ) var _ = SIGDescribe("API priority and fairness", func() {

nonMatchingUsername := "foo" ginkgo.By("creating a testing prioritylevel") <del> createdPriorityLevel, err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Create( context.TODO(), &flowcontrol.PriorityLevelConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: testingPriorityLevelName, }, Spec: flowcontrol.PriorityLevelConfigurationSpec{ Type: flowcontrol.PriorityLevelEnablementLimited, Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ AssuredConcurrencyShares: 1, // will have at minimum 1 concurrency share LimitResponse: flowcontrol.LimitResponse{ Type: flowcontrol.LimitResponseTypeReject, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func() { // clean-ups err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Delete(context.TODO(), testingPriorityLevelName, metav1.DeleteOptions{}) framework.ExpectNoError(err) err = f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Delete(context.TODO(), testingFlowSchemaName, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() </del> <ins> createdPriorityLevel, cleanup := createPriorityLevel(f, testingPriorityLevelName, 1) defer cleanup() </ins> ginkgo.By("creating a testing flowschema") <del> createdFlowSchema, err := f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Create( context.TODO(), &flowcontrol.FlowSchema{ ObjectMeta: metav1.ObjectMeta{ Name: testingFlowSchemaName, }, Spec: flowcontrol.FlowSchemaSpec{ MatchingPrecedence: 1000, // a rather higher precedence to ensure it make effect PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{ Name: testingPriorityLevelName, }, DistinguisherMethod: &flowcontrol.FlowDistinguisherMethod{ Type: flowcontrol.FlowDistinguisherMethodByUserType, }, Rules: []flowcontrol.PolicyRulesWithSubjects{ { Subjects: []flowcontrol.Subject{ { Kind: flowcontrol.SubjectKindUser, User: &flowcontrol.UserSubject{ Name: matchingUsername, }, }, }, NonResourceRules: []flowcontrol.NonResourcePolicyRule{ { Verbs:           []string{flowcontrol.VerbAll}, NonResourceURLs: []string{flowcontrol.NonResourceAll}, }, }, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("response headers should contain flow-schema/priority-level uid") </del> <ins> createdFlowSchema, cleanup := createFlowSchema(f, testingFlowSchemaName, 1000, testingPriorityLevelName, matchingUsername) defer cleanup() ginkgo.By("checking response headers contain flow-schema/priority-level uid") </ins> if !testResponseHeaderMatches(f, matchingUsername, string(createdPriorityLevel.UID), string(createdFlowSchema.UID)) { framework.Failf("matching user doesnt received UID for the testing priority-level and flow-schema") }

} }) <del> ginkgo.It("should ensure that requests can't be drowned out", func() { </del> <ins> // This test creates two flow schemas and a corresponding priority level for // each flow schema. One flow schema has a higher match precedence. With two // clients making requests at different rates, we test to make sure that the // higher QPS client cannot drown out the other one despite having higher // priority. ginkgo.It("should ensure that requests can't be drowned out (priority)", func() { </ins> flowSchemaNamePrefix := "e2e-testing-flowschema" priorityLevelNamePrefix := "e2e-testing-prioritylevel" loadDuration := 10 * time.Second

completedRequests     int32 } clients := []client{ <del> // "elephant" refers to a client that creates requests at a much higher </del> <ins> // "highqps" refers to a client that creates requests at a much higher </ins> // QPS than its counter-part and well above its concurrency share limit. <del> // In contrast, the mouse stays under its concurrency shares. // Additionally, the "elephant" client also has a higher matching </del> <ins> // In contrast, "lowqps" stays under its concurrency shares. // Additionally, the "highqps" client also has a higher matching </ins> // precedence for its flow schema. <del> {username: "elephant", qps: 100.0, concurrencyMultiplier: 2.0, matchingPrecedence: 999}, {username: "mouse", qps: 5.0, concurrencyMultiplier: 0.5, matchingPrecedence: 1000}, </del> <ins> {username: "highqps", qps: 100.0, concurrencyMultiplier: 2.0, matchingPrecedence: 999}, {username: "lowqps", qps: 5.0, concurrencyMultiplier: 0.5, matchingPrecedence: 1000}, </ins> } ginkgo.By("creating test priority levels and flow schemas") for i := range clients { clients[i].priorityLevelName = fmt.Sprintf("%s-%s", priorityLevelNamePrefix, clients[i].username) framework.Logf("creating PriorityLevel %q", clients[i].priorityLevelName) <del> _, err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Create( context.TODO(), &flowcontrol.PriorityLevelConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: clients[i].priorityLevelName, }, Spec: flowcontrol.PriorityLevelConfigurationSpec{ Type: flowcontrol.PriorityLevelEnablementLimited, Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ AssuredConcurrencyShares: 1, LimitResponse: flowcontrol.LimitResponse{ Type: flowcontrol.LimitResponseTypeReject, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func(name string) { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Delete(context.TODO(), name, metav1.DeleteOptions{})) }(clients[i].priorityLevelName) </del> <ins> _, cleanup := createPriorityLevel(f, clients[i].priorityLevelName, 1) defer cleanup() </ins> clients[i].flowSchemaName = fmt.Sprintf("%s-%s", flowSchemaNamePrefix, clients[i].username) framework.Logf("creating FlowSchema %q", clients[i].flowSchemaName) <del> _, err = f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Create( context.TODO(), &flowcontrol.FlowSchema{ ObjectMeta: metav1.ObjectMeta{ Name: clients[i].flowSchemaName, }, Spec: flowcontrol.FlowSchemaSpec{ MatchingPrecedence: clients[i].matchingPrecedence, PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{ Name: clients[i].priorityLevelName, }, DistinguisherMethod: &flowcontrol.FlowDistinguisherMethod{ Type: flowcontrol.FlowDistinguisherMethodByUserType, }, Rules: []flowcontrol.PolicyRulesWithSubjects{ { Subjects: []flowcontrol.Subject{ { Kind: flowcontrol.SubjectKindUser, User: &flowcontrol.UserSubject{ Name: clients[i].username, }, }, }, NonResourceRules: []flowcontrol.NonResourcePolicyRule{ { Verbs:           []string{flowcontrol.VerbAll}, NonResourceURLs: []string{flowcontrol.NonResourceAll}, }, }, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func(name string) { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Delete(context.TODO(), name, metav1.DeleteOptions{})) }(clients[i].flowSchemaName) </del> <ins> _, cleanup = createFlowSchema(f, clients[i].flowSchemaName, clients[i].matchingPrecedence, clients[i].priorityLevelName, clients[i].username) defer cleanup() </ins> } ginkgo.By("getting request concurrency from metrics") for i := range clients { <del> resp, err := f.ClientSet.CoreV1().RESTClient().Get().RequestURI("/metrics").DoRaw(context.TODO()) framework.ExpectNoError(err) sampleDecoder := expfmt.SampleDecoder{ Dec:  expfmt.NewDecoder(bytes.NewBuffer(resp), expfmt.FmtText), Opts: &expfmt.DecodeOptions{}, </del> <ins> realConcurrency := getPriorityLevelConcurrency(f, clients[i].priorityLevelName) clients[i].concurrency = int32(float64(realConcurrency) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 } framework.Logf("request concurrency for %q will be %d (concurrency share = %d)", clients[i].username, clients[i].concurrency, realConcurrency) } ginkgo.By(fmt.Sprintf("starting uniform QPS load for %s", loadDuration.String())) var wg sync.WaitGroup for i := range clients { wg.Add(1) go func(c *client) { defer wg.Done() framework.Logf("starting uniform QPS load for %q: concurrency=%d, qps=%.1f", c.username, c.concurrency, c.qps) c.completedRequests = uniformQPSLoadConcurrent(f, c.username, c.concurrency, c.qps, loadDuration) }(&clients[i]) } wg.Wait() ginkgo.By("checking completed requests with expected values") for _, client := range clients { // Each client should have 95% of its ideal number of completed requests. maxCompletedRequests := float64(client.concurrency) * client.qps * float64(loadDuration/time.Second) fractionCompleted := float64(client.completedRequests) / maxCompletedRequests framework.Logf("client %q completed %d/%d requests (%.1f%%)", client.username, client.completedRequests, int32(maxCompletedRequests), 100*fractionCompleted) if fractionCompleted < 0.95 { framework.Failf("client %q: got %.1f%% completed requests, want at least 95%%", client.username, 100*fractionCompleted) </ins> } <del> for { var v model.Vector err := sampleDecoder.Decode(&v) if err == io.EOF { break } framework.ExpectNoError(err) for _, metric := range v { if string(metric.Metric[model.MetricNameLabel]) != requestConcurrencyLimitMetricName { continue } if string(metric.Metric[requestConcurrencyLimitMetricLabelName]) != clients[i].priorityLevelName { continue } clients[i].concurrency = int32(float64(metric.Value) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 } framework.Logf("request concurrency for %q will be %d (concurrency share = %d)", clients[i].username, clients[i].concurrency, int32(metric.Value)) } </del> <ins> } }) // This test has two clients (different usernames) making requests at // different rates. Both clients' requests get mapped to the same flow schema // and priority level. We expect APF's "ByUser" flow distinguisher to isolate // the two clients and not allow one client to drown out the other despite // having a higher QPS. ginkgo.It("should ensure that requests can't be drowned out (fairness)", func() { priorityLevelName := "e2e-testing-prioritylevel" flowSchemaName := "e2e-testing-flowschema" loadDuration := 10 * time.Second framework.Logf("creating PriorityLevel %q", priorityLevelName) _, cleanup := createPriorityLevel(f, priorityLevelName, 1) defer cleanup() framework.Logf("creating FlowSchema %q", flowSchemaName) _, cleanup = createFlowSchema(f, flowSchemaName, 1000, priorityLevelName, "*") defer cleanup() type client struct { username              string qps                   float64 concurrencyMultiplier float64 concurrency           int32 completedRequests     int32 } clients := []client{ {username: "highqps", qps: 100.0, concurrencyMultiplier: 2.0}, {username: "lowqps", qps: 5.0, concurrencyMultiplier: 0.5}, } framework.Logf("getting real concurrency") realConcurrency := getPriorityLevelConcurrency(f, priorityLevelName) for i := range clients { clients[i].concurrency = int32(float64(realConcurrency) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 </ins> } <ins> framework.Logf("request concurrency for %q will be %d", clients[i].username, clients[i].concurrency) </ins> } ginkgo.By(fmt.Sprintf("starting uniform QPS load for %s", loadDuration.String()))

}) }) <ins> // createPriorityLevel creates a priority level with the provided assured // concurrency share. func createPriorityLevel(f *framework.Framework, priorityLevelName string, assuredConcurrencyShares int32) (*flowcontrol.PriorityLevelConfiguration, func()) { createdPriorityLevel, err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Create( context.TODO(), &flowcontrol.PriorityLevelConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: priorityLevelName, }, Spec: flowcontrol.PriorityLevelConfigurationSpec{ Type: flowcontrol.PriorityLevelEnablementLimited, Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ AssuredConcurrencyShares: assuredConcurrencyShares, LimitResponse: flowcontrol.LimitResponse{ Type: flowcontrol.LimitResponseTypeReject, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) return createdPriorityLevel, func() { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Delete(context.TODO(), priorityLevelName, metav1.DeleteOptions{})) } } func getPriorityLevelConcurrency(f *framework.Framework, priorityLevelName string) int32 { resp, err := f.ClientSet.CoreV1().RESTClient().Get().RequestURI("/metrics").DoRaw(context.TODO()) framework.ExpectNoError(err) sampleDecoder := expfmt.SampleDecoder{ Dec:  expfmt.NewDecoder(bytes.NewBuffer(resp), expfmt.FmtText), Opts: &expfmt.DecodeOptions{}, } for { var v model.Vector err := sampleDecoder.Decode(&v) if err == io.EOF { break } framework.ExpectNoError(err) for _, metric := range v { if string(metric.Metric[model.MetricNameLabel]) != requestConcurrencyLimitMetricName { continue } if string(metric.Metric[requestConcurrencyLimitMetricLabelName]) != priorityLevelName { continue } return int32(metric.Value) } } framework.ExpectNoError(fmt.Errorf("cannot find metric %q with matching priority level name label %q", requestConcurrencyLimitMetricName, priorityLevelName)) return 0 } // createFlowSchema creates a flow schema referring to a particular priority // level and matching the username provided. func createFlowSchema(f *framework.Framework, flowSchemaName string, matchingPrecedence int32, priorityLevelName string, matchingUsername string) (*flowcontrol.FlowSchema, func()) { var subjects []flowcontrol.Subject if matchingUsername == "*" { subjects = append(subjects, flowcontrol.Subject{ Kind: flowcontrol.SubjectKindGroup, Group: &flowcontrol.GroupSubject{ Name: user.AllAuthenticated, }, }) } else { subjects = append(subjects, flowcontrol.Subject{ Kind: flowcontrol.SubjectKindUser, User: &flowcontrol.UserSubject{ Name: matchingUsername, }, }) } createdFlowSchema, err := f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Create( context.TODO(), &flowcontrol.FlowSchema{ ObjectMeta: metav1.ObjectMeta{ Name: flowSchemaName, }, Spec: flowcontrol.FlowSchemaSpec{ MatchingPrecedence: matchingPrecedence, PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{ Name: priorityLevelName, }, DistinguisherMethod: &flowcontrol.FlowDistinguisherMethod{ Type: flowcontrol.FlowDistinguisherMethodByUserType, }, Rules: []flowcontrol.PolicyRulesWithSubjects{ { Subjects: subjects, NonResourceRules: []flowcontrol.NonResourcePolicyRule{ { Verbs:           []string{flowcontrol.VerbAll}, NonResourceURLs: []string{flowcontrol.NonResourceAll}, }, }, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) return createdFlowSchema, func() { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Delete(context.TODO(), flowSchemaName, metav1.DeleteOptions{})) } } </ins> // makeRequests creates a request to the API server and returns the response. func makeRequest(f *framework.Framework, username string) *http.Response { <del> config := rest.CopyConfig(f.ClientConfig()) </del> <ins> config := f.ClientConfig() </ins> config.Impersonate.UserName = username <ins> config.Impersonate.Groups = []string{"system:authenticated"} </ins> roundTripper, err := rest.TransportFor(config) framework.ExpectNoError(err)

}() } wg.Wait() <del> return atomic.LoadInt32(&completed) </del> <ins> return completed </ins> }