Code change:
} // Ensure that the if the APIService OpenAPI is updated, the // aggregated OpenAPI is also updated. <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path2") expectNoPath(t, swagger, "/apis/apiservicegroup/v1/path1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") } // Tests that an APIService that registers OpenAPI will only have the OpenAPI // for its specific group version served registered. // See https://github.com/kubernetes/kubernetes/pull/123570 for full context. func TestAPIServiceOpenAPIServiceMismatch(t *testing.T) { mux := http.NewServeMux() var delegationHandlers []http.Handler delegate1 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/foo/v1": {}, }, }, }, }} delegationHandlers = append(delegationHandlers, delegate1) s := buildAndRegisterSpecAggregator(delegationHandlers, mux) apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v1", Service: &v1.ServiceReference{Name: "dummy"}, }, } apiService.Name = "apiservice" apiService2 := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v2", Service: &v1.ServiceReference{Name: "dummy2"}, }, } apiService2.Name = "apiservice2" handler := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/apiservicegroup/v1": {}, }, }, }, }} handler2 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/a":                  {}, "/apis/apiservicegroup/v1": {}, "/apis/apiservicegroup/v2": {}, }, }, }, }} if err := s.AddUpdateAPIService(apiService, handler); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService.Name); err != nil { t.Error(err) } if err := s.AddUpdateAPIService(apiService2, handler2); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService2.Name); err != nil { t.Error(err) } swagger, err := fetchOpenAPI(mux) if err != nil { t.Error(err) } expectPath(t, swagger, "/apis/apiservicegroup/v1") </ins> expectPath(t, swagger, "/apis/apiservicegroup/v2") <ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Logf("Remove APIService %s", apiService.Name) s.RemoveAPIService(apiService.Name) swagger, err = fetchOpenAPI(mux) if err != nil { t.Error(err) } // Ensure that the if the APIService is added then removed, the OpenAPI disappears from the aggregated OpenAPI as well. </ins> expectNoPath(t, swagger, "/apis/apiservicegroup/v1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") }

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in