Code change:
Hello, we have been using client-go to measure the latency for all client-go traffic from Cilium to kube-apiserver. We have that since v1.24.0 the URL path is no longer aggregated in a way that this cardinality would no longer happen as presented in the following image: <img width="330" alt="Screen Shot 2022-08-08 at 10 18 24 AM" src="https://user-"It seems this functionality was removed as a part of as it seems that PR only took into account that users of the would only be using it to measure latency to a particular host and not for a particular URL path as we are doing in . Would it be reasonable to revert commit in order to fix this regression? Thanks cc
/sig api-machinery /sig instrumentation
But reverting will bring back the cardinality problem, will not? The latency of each URL path is available in the apiserver metrics ...
We have never any cardinality problem with the previous code since we were not using the "host" field which is what I assume the original issue about cardinality was trying to fix. We only used the "path" of an URL, which was aggregated in a way that would not be facing any cardinality, like the figure attached in this GH description. Now, starting with = 1.24.0, each and is actually a different value which does cause the cardinality problem to appear.
ah, I see the disconnection, I wasn't understanding the issue sorry. The Interface uses the url as parameter to keep backwards compatibility, but the method that gathers the metrics only uses the host to avoid the cardinality problem. You have to update the method that gathers the metrics to only use the host part of the url
Filed
should fix the code in kubernetes ? however, that will break consumer of the Interface that still wants to process the path, but we know using the path will cause issues with cardinality :shrug:
I would be fine with adding a new label to mitigate that issue since originally the cardinality explosion that was reported was caused by the query string and not the path: When trimming down the whole URL to only the host we even had a discussion to keep the path, but we decided not to since we didn't have any use case for this data. If you have a use case where you need the path in order to get more granularity on the latency of your network then I would be fine with adding the label to Kubernetes, but another path forward could be to also remove the path information from cilium if deemed unnecessary. In the most common scenarios, you will only need the host to get the network latency based on different proxy routes.
yeah we can't really do much about it, it is either we leak the unsanitized paths or we prevent the users from using the path completely. Since we want to not break the interface and keep the URL parameter, we might just have to keep sanitizing the paths for our consumers even though we are not using the path anymore.
IIUIC the RESTClient is used in the dynamic client-go too, for these users with 1900 CRDs, the path is really going to explode for them
Then it seems that the best option to not break users would be to revert , but at the same time I don't really like the idea of spending CPU time sanitizing something that will never be used in the RESTClient
A suggestion would be to create a new interface similar to the that would have the following method: Thus, those users with 1900 CRDs would use the instead of the .
after digging a little bit the underlying issue it looks that the problem was initially caused by a kubelet bug as pointed out . The dependents of client-go don't have access to all the fields of the struct which uses heavily to generate the aggregated URL.
well, it is clear now that current status is wrong, we pass the whole URL to the interface and rely on the to use only the Host part of the URL :smile: I'm fine with whatever you decide, to restore or just completely strip the Path Sorry for the unfortunate consequence, at that time I didn't consider that the Interface/Method drift and thought that function was orphan
/triage accepted /assign

Related discussion:
text: return finalURL } <ins> // finalURLTemplate is similar to URL(), but will make all specific parameter values equal // - instead of name or namespace, "{name}" and "{namespace}" will be used, and all query // parameters will be reset. This creates a copy of the url so as not to change the // underlying object. func (r Request) finalURLTemplate() url.URL { newParams := url.Values{} v := []string{"{value}"} for k := range r.params { newParams[k] = v } r.params = newParams u := r.URL() if u == nil { return url.URL{} } segments := strings.Split(u.Path, "/") groupIndex := 0 index := 0 trimmedBasePath := "" if r.c.base != nil && strings.Contains(u.Path, r.c.base.Path) { p := strings.TrimPrefix(u.Path, r.c.base.Path) if !strings.HasPrefix(p, "/") { p = "/" + p } // store the base path that we have trimmed so we can append it // before returning the URL trimmedBasePath = r.c.base.Path segments = strings.Split(p, "/") groupIndex = 1 } if len(segments) <= 2 { return *u } const CoreGroupPrefix = "api" const NamedGroupPrefix = "apis" isCoreGroup := segments[groupIndex] == CoreGroupPrefix isNamedGroup := segments[groupIndex] == NamedGroupPrefix if isCoreGroup { // checking the case of core group with /api/v1/... format index = groupIndex + 2 } else if isNamedGroup { // checking the case of named group with /apis/apps/v1/... format index = groupIndex + 3 } else { // this should not happen that the only two possibilities are /api... and /apis..., just want to put an // outlet here in case more API groups are added in future if ever possible: // https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups // if a wrong API groups name is encountered, return the {prefix} for url.Path u.Path = "/{prefix}" u.RawQuery = "" return *u } // switch segLength := len(segments) - index; segLength { switch { // case len(segments) - index == 1: // resource (with no name) do nothing case len(segments)-index == 2: // /$RESOURCE/$NAME: replace $NAME with {name} segments[index+1] = "{name}" case len(segments)-index == 3: if segments[index+2] == "finalize" || segments[index+2] == "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+1] = "{name}" } else { // /namespace/$NAMESPACE/$RESOURCE: replace $NAMESPACE with {namespace} segments[index+1] = "{namespace}" } case len(segments)-index >= 4: segments[index+1] = "{namespace}" // /namespace/$NAMESPACE/$RESOURCE/$NAME: replace $NAMESPACE with {namespace},  $NAME with {name} if segments[index+3] != "finalize" && segments[index+3] != "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+3] = "{name}" } } u.Path = path.Join(trimmedBasePath, path.Join(segments...)) return *u } </ins> func (r *Request) tryThrottleWithInfo(ctx context.Context, retryInfo string) error { if r.rateLimiter == nil { return nil