Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In Kubernetes 1.11.x, the controller-manager metrics are providing histogram records for every api endpoint ever called during namespace cleanup. For every namespace that is deleted since a controller has been the leader, 1000+ new lines of metrics are created / served indefinitely. When creating/deleting namespaces at a rate of 100/hour, that's 100,000+ new metrics lines being served for every request of controller-manager/metrics. spun up a 1.10 environment, and says it looks like the namespace names were replaced with {namespace}: In 1.11 this has changed (short clip of 1 histogram): What you expected to happen: Don't generate so many metrics. How to reproduce it (as minimally and precisely as possible): To replicate in 1.11.x, simply create and then delete a namespace and query the metrics endpoint of your controller-manager. Anything else we need to know?: Repro steps and bug reported by /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
did some more debugging and believes that the root cause is He says that PR switched to the new dynamic client so when a request is made, this function is called: He explains that function in the old client checks if is set, the following is an example call: In the new client, the following is an example call: In this new dynamic client, the delete operation calls: whereas in the old client, it ran: The , he says, is the critical component, which sets the request client namespace field without namespace being set, when is called then the metrics are populated without renaming the Namespace to which causes tons of unique metrics to be generated and cause metrics data grow very quickly. This seems to be a pretty critical bug. Will mark it for v1.12. Please triage. /milestone v1.12 CC
- friendly reminder that Code Freeze is next week; if this is critical enough for 1.12 it needs to be triaged accordingly.
This isn't a regression (1.11 had it), but it's a big enough bug that I'd take a fix for it in 1.12. I think that a fix should take the form of fixing the to handle cases where the abspath is set. That would fix it for all future callers instead of just a one off for this client. That method is only called for metrics and our api paths follow a very fixed format. I expect that it should be fairly easy to recognize and you only have to do the analysis if you have no hits for the namespace being set.
please apply a priority/critical-urgent label to keep track for 1.12
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I applied the label. We will have to un-apply it if a targeted fix doesn't show up very soon :)
If no one has started working on it, I am on this now. I will try fix it ASAP.
please do!
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
please update on a daily basis, as we're in Code Freeze. :)
I have reproduced the problem and also wrote a unit test case for confirmation. Basically with the current dynamic client implementation, the actually string of namespace is in url here: , so when finalURLTemplate function checks (), it returns false and would not replace namespace with "{namespace}". I am working on fixing it by replacing the url segment after /namespaces/ by . I should be able to submit the PR tomorrow if not today.

Discussion:
// finalURLTemplate is similar to URL(), but will make all specific parameter values equal // - instead of name or namespace, "{name}" and "{namespace}" will be used, and all query <del> // parameters will be reset. This creates a copy of the request so as not to change the // underlying object.  This means some useful request info (like the types of field // selectors in use) will be lost. // TODO: preserve field selector keys </del> <ins> // parameters will be reset. This creates a copy of the url so as not to change the // underlying object. </ins> func (r Request) finalURLTemplate() url.URL { <del> if len(r.resourceName) != 0 { r.resourceName = "{name}" } if r.namespaceSet && len(r.namespace) != 0 { r.namespace = "{namespace}" } </del> newParams := url.Values{} v := []string{"{value}"} for k := range r.params {

} r.params = newParams url := r.URL() <ins> segments := strings.Split(r.URL().Path, "/") groupIndex := 0 index := 0 if r.URL() != nil && r.baseURL != nil && strings.Contains(r.URL().Path, r.baseURL.Path) { groupIndex += len(strings.Split(r.baseURL.Path, "/")) } if groupIndex >= len(segments) { return *url } const CoreGroupPrefix = "api" const NamedGroupPrefix = "apis" isCoreGroup := segments[groupIndex] == CoreGroupPrefix isNamedGroup := segments[groupIndex] == NamedGroupPrefix if isCoreGroup { // checking the case of core group with /api/v1/... format index = groupIndex + 2 } else if isNamedGroup { // checking the case of named group with /apis/apps/v1/... format index = groupIndex + 3 } else { // this should not happen that the only two possibilities are /api... and /apis..., just want to put an // outlet here in case more API groups are added in future if ever possible: // https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups // if a wrong API groups name is encountered, return the {prefix} for url.Path url.Path = "/{prefix}" url.RawQuery = "" return *url } //switch segLength := len(segments) - index; segLength { switch { // case len(segments) - index == 1: // resource (with no name) do nothing case len(segments)-index == 2: // /$RESOURCE/$NAME: replace $NAME with {name} segments[index+1] = "{name}" case len(segments)-index == 3: if segments[index+2] == "finalize" || segments[index+2] == "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+1] = "{name}" } else { // /namespace/$NAMESPACE/$RESOURCE: replace $NAMESPACE with {namespace} segments[index+1] = "{namespace}" } case len(segments)-index >= 4: segments[index+1] = "{namespace}" // /namespace/$NAMESPACE/$RESOURCE/$NAME: replace $NAMESPACE with {namespace},  $NAME with {name} if segments[index+3] != "finalize" && segments[index+3] != "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+3] = "{name}" } } url.Path = path.Join(segments...) </ins> return *url }

} func TestURLTemplate(t *testing.T) { <del> uri, _ := url.Parse("http://localhost") r := NewRequest(nil, "POST", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0) r.Prefix("pre1").Resource("r1").Namespace("ns").Name("nm").Param("p0", "v0") full := r.URL() if full.String() != "http://localhost/pre1/namespaces/ns/r1/nm?p0=v0" { t.Errorf("unexpected initial URL: %s", full) } actualURL := r.finalURLTemplate() actual := actualURL.String() expected := "http://localhost/pre1/namespaces/%7Bnamespace%7D/r1/%7Bname%7D?p0=%7Bvalue%7D" if actual != expected { t.Errorf("unexpected URL template: %s %s", actual, expected) } if r.URL().String() != full.String() { t.Errorf("creating URL template changed request: %s -> %s", full.String(), r.URL().String()) </del> <ins> uri, _ := url.Parse("http://localhost/some/base/url/path") testCases := []struct { Request          *Request ExpectedFullURL  string ExpectedFinalURL string }{ { // non dynamic client Request: NewRequest(nil, "POST", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("api", "v1").Resource("r1").Namespace("ns").Name("nm").Param("p0", "v0"), ExpectedFullURL:  "http://localhost/some/base/url/path/api/v1/namespaces/ns/r1/nm?p0=v0", ExpectedFinalURL: "http://localhost/some/base/url/path/api/v1/namespaces/%7Bnamespace%7D/r1/%7Bname%7D?p0=%7Bvalue%7D", }, { // non dynamic client with wrong api group Request: NewRequest(nil, "POST", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("pre1", "v1").Resource("r1").Namespace("ns").Name("nm").Param("p0", "v0"), ExpectedFullURL:  "http://localhost/some/base/url/path/pre1/v1/namespaces/ns/r1/nm?p0=v0", ExpectedFinalURL: "http://localhost/%7Bprefix%7D", }, { // dynamic client with core group + namespace + resourceResource (with name) // /api/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/api/v1/namespaces/ns/r1/name1"), ExpectedFullURL:  "http://localhost/some/base/url/path/api/v1/namespaces/ns/r1/name1", ExpectedFinalURL: "http://localhost/some/base/url/path/api/v1/namespaces/%7Bnamespace%7D/r1/%7Bname%7D", }, { // dynamic client with named group + namespace + resourceResource (with name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/g1/v1/namespaces/ns/r1/name1"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/g1/v1/namespaces/ns/r1/name1", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/g1/v1/namespaces/%7Bnamespace%7D/r1/%7Bname%7D", }, { // dynamic client with core group + namespace + resourceResource (with NO name) // /api/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/api/v1/namespaces/ns/r1"), ExpectedFullURL:  "http://localhost/some/base/url/path/api/v1/namespaces/ns/r1", ExpectedFinalURL: "http://localhost/some/base/url/path/api/v1/namespaces/%7Bnamespace%7D/r1", }, { // dynamic client with named group + namespace + resourceResource (with NO name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/g1/v1/namespaces/ns/r1"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/g1/v1/namespaces/ns/r1", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/g1/v1/namespaces/%7Bnamespace%7D/r1", }, { // dynamic client with core group + resourceResource (with name) // /api/$RESOURCEVERSION/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/api/v1/r1/name1"), ExpectedFullURL:  "http://localhost/some/base/url/path/api/v1/r1/name1", ExpectedFinalURL: "http://localhost/some/base/url/path/api/v1/r1/%7Bname%7D", }, { // dynamic client with named group + resourceResource (with name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/g1/v1/r1/name1"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/g1/v1/r1/name1", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/g1/v1/r1/%7Bname%7D", }, { // dynamic client with named group + namespace + resourceResource (with name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME/$SUBRESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces/finalize"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces/finalize", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bnamespace%7D/namespaces/%7Bname%7D/finalize", }, { // dynamic client with named group + namespace + resourceResource (with name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bnamespace%7D/namespaces/%7Bname%7D", }, { // dynamic client with named group + namespace + resourceResource (with NO name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%SUBRESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/namespaces/finalize"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/namespaces/finalize", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bnamespace%7D/namespaces/finalize", }, { // dynamic client with named group + namespace + resourceResource (with NO name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%SUBRESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/namespaces/status"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/namespaces/status", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bnamespace%7D/namespaces/status", }, { // dynamic client with named group + namespace + resourceResource (with no name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/namespaces"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/namespaces", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bnamespace%7D/namespaces", }, { // dynamic client with named group + resourceResource (with name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/finalize"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/finalize", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bname%7D/finalize", }, { // dynamic client with named group + resourceResource (with name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces/status"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces/status", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bname%7D/status", }, { // dynamic client with named group + resourceResource (with name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces/namespaces"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/namespaces", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces/%7Bname%7D", }, { // dynamic client with named group + resourceResource (with no name) // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/$RESOURCE/%NAME Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/apis/namespaces/namespaces/namespaces"), ExpectedFullURL:  "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces", ExpectedFinalURL: "http://localhost/some/base/url/path/apis/namespaces/namespaces/namespaces", }, { // dynamic client with wrong api group + namespace + resourceResource (with name) + subresource // /apis/$NAMEDGROUPNAME/$RESOURCEVERSION/namespaces/$NAMESPACE/$RESOURCE/%NAME/$SUBRESOURCE Request: NewRequest(nil, "DELETE", uri, "", ContentConfig{GroupVersion: &schema.GroupVersion{Group: "test"}}, Serializers{}, nil, nil, 0). Prefix("/pre1/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces/finalize"), ExpectedFullURL:  "http://localhost/some/base/url/path/pre1/namespaces/namespaces/namespaces/namespaces/namespaces/namespaces/finalize", ExpectedFinalURL: "http://localhost/%7Bprefix%7D", }, } for i, testCase := range testCases { r := testCase.Request full := r.URL() if full.String() != testCase.ExpectedFullURL { t.Errorf("%d: unexpected initial URL: %s %s", i, full, testCase.ExpectedFullURL) } actualURL := r.finalURLTemplate() actual := actualURL.String() if actual != testCase.ExpectedFinalURL { t.Errorf("%d: unexpected URL template: %s %s", i, actual, testCase.ExpectedFinalURL) } if r.URL().String() != full.String() { t.Errorf("%d, creating URL template changed request: %s -> %s", i, full.String(), r.URL().String()) } </ins> } }