Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While I was watching grafana for a single run, I noticed that GET requests are reported as LIST requests, e.g. grafana shows ~1K QPS for LIST of pods ! while kube-apiserver's logs show: (there are no pod lists that time). I'm not sure if this is intended change to metrics reporting (kind of unexpected to be honest), but this definitely affects way our scalability tests validate latency metrics: We do not validate GET API call latency at all Validation for LIST requests is also affected, as usually GET API calls are way cheaper than LIST requests, so grouping GET-LISTs and LIST-LISTs together lowers p99 for LISTs. Also perf-dash shows that GET requests simply disappeared at some point: http://perf- ! What you expected to happen: GET requests should be reported with verb=GET in apiserverrequest_total metric. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:

FYI
I managed to narrow the issue by bisecting the runs of which runs load tests using master k/k every 1 hour. This is the last test run that had GETs reported correctly: This is the first test run that has no GETs reported: Looking at the k/k repo commits' hashes inside artifacts in both runs and at what happened between each of them it looks as if might be the culprit.
From the former run's artifacts: This is (among others) what we miss in the latter run - this is understandable from the point of view of this if condition: Why did the reported GETs have as the scope instead of in the first place, however??
It seems that our handing is incorrect: vs Or my understanding of it was incorrect. It seems that the semantic is: cluster - for cluster-scope requests namespace - for namespace-scoped requests request - for subresources I think the assumption in the monitoring path was: cluster - for cluster-scope requests namespace - for namespace-scope requests requests - for requests touching a single resource (not necessary subresource) I'm not 100% sure which semantic we should use - based on the finding we should fix one or the other place...
cc There should probably be a unit test mapping URL paths to the expected label values.
is opened to fix that.
/triage accepted

Discussion:
"//staging/src/k8s.io/component-base/metrics/...", "//test/e2e_node", "//test/integration/apiserver/flowcontrol", <ins> "//test/integration/metrics", </ins> "//vendor/...", ], )

requestScope = "resource" operationSuffix = operationSuffix + "WithPath" } <ins> if strings.Index(action.Path, "/{name}") != -1 || action.Verb == "POST" { requestScope = "resource" } </ins> if action.AllNamespaces { requestScope = "cluster" operationSuffix = operationSuffix + "ForAllNamespaces"

embed = [":go_default_library"], tags = ["integration"], deps = [ <ins> "//staging/src/k8s.io/api/core/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/integration/framework:go_default_library", <ins> "//vendor/github.com/prometheus/common/model:go_default_library", </ins> ], )

"runtime" "testing" <ins> "github.com/prometheus/common/model" v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" clientset "k8s.io/client-go/kubernetes"

"etcd_request_duration_seconds_sum", }) } <ins> func TestApiserverMetricsLabels(t *testing.T) { _, s, closeFn := framework.RunAMaster(nil) defer closeFn() client, err := clientset.NewForConfig(&restclient.Config{Host: s.URL, QPS: -1}) if err != nil { t.Fatalf("Error in create clientset: %v", err) } expectedMetrics := []model.Metric{} metricLabels := func(group, version, resource, subresource, scope, verb string) model.Metric { return map[model.LabelName]model.LabelValue{ model.LabelName("group"):       model.LabelValue(group), model.LabelName("version"):     model.LabelValue(version), model.LabelName("resource"):    model.LabelValue(resource), model.LabelName("subresource"): model.LabelValue(subresource), model.LabelName("scope"):       model.LabelValue(scope), model.LabelName("verb"):        model.LabelValue(verb), } } callOrDie := func(_ interface{}, err error) { if err != nil { t.Fatalf("unexpected error: %v", err) } } appendExpectedMetric := func(metric model.Metric) { expectedMetrics = append(expectedMetrics, metric) } // Call appropriate endpoints to ensure particular metrics will be exposed // Namespace-scoped resource c := client.CoreV1().Pods(metav1.NamespaceDefault) makePod := func(labelValue string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "foo", Labels: map[string]string{"foo": labelValue}, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "container", Image: "image", }, }, }, } } callOrDie(c.Create(context.TODO(), makePod("foo"), metav1.CreateOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "resource", "POST")) callOrDie(c.Update(context.TODO(), makePod("bar"), metav1.UpdateOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "resource", "PUT")) callOrDie(c.UpdateStatus(context.TODO(), makePod("bar"), metav1.UpdateOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "status", "resource", "PUT")) callOrDie(c.Get(context.TODO(), "foo", metav1.GetOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "resource", "GET")) callOrDie(c.List(context.TODO(), metav1.ListOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "namespace", "LIST")) callOrDie(nil, c.Delete(context.TODO(), "foo", metav1.DeleteOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "resource", "DELETE")) // cluster-scoped LIST of namespace-scoped resources callOrDie(client.CoreV1().Pods(metav1.NamespaceAll).List(context.TODO(), metav1.ListOptions{})) appendExpectedMetric(metricLabels("", "v1", "pods", "", "cluster", "LIST")) // Cluster-scoped resource cn := client.CoreV1().Namespaces() makeNamespace := func(labelValue string) *v1.Namespace { return &v1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name:   "foo", Labels: map[string]string{"foo": labelValue}, }, } } callOrDie(cn.Create(context.TODO(), makeNamespace("foo"), metav1.CreateOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "", "resource", "POST")) callOrDie(cn.Update(context.TODO(), makeNamespace("bar"), metav1.UpdateOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "", "resource", "PUT")) callOrDie(cn.UpdateStatus(context.TODO(), makeNamespace("bar"), metav1.UpdateOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "status", "resource", "PUT")) callOrDie(cn.Get(context.TODO(), "foo", metav1.GetOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "", "resource", "GET")) callOrDie(cn.List(context.TODO(), metav1.ListOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "", "cluster", "LIST")) callOrDie(nil, cn.Delete(context.TODO(), "foo", metav1.DeleteOptions{})) appendExpectedMetric(metricLabels("", "v1", "namespaces", "", "resource", "DELETE")) // Verify if all metrics were properly exported. metrics, err := scrapeMetrics(s) if err != nil { t.Fatal(err) } samples, ok := metrics["apiserver_request_total"] if !ok { t.Fatalf("apiserver_request_total metric not exposed") } hasLabels := func(current, expected model.Metric) bool { for key, value := range expected { if current[key] != value { return false } } return true } for _, expectedMetric := range expectedMetrics { found := false for _, sample := range samples { if hasLabels(sample.Metric, expectedMetric) { found = true break } } if !found { t.Errorf("No sample found for %#v", expectedMetric) } } } </ins>