Change:
reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded. I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.
adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.

Discussion:
admissionapi "k8s.io/pod-security-admission/api" samplev1alpha1 "k8s.io/sample-apiserver/pkg/apis/wardle/v1alpha1" "k8s.io/utils/pointer" <del> "k8s.io/utils/strings/slices" </del> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega"

framework.ExpectNoError(err, "failed to count the required APIServices") framework.Logf("APIService %s has been deleted.", apiServiceName) <del> ginkgo.By("Confirm that the group path of " + apiServiceName + " was removed from root paths") groupPath := "/apis/" + apiServiceGroupName err = wait.PollUntilContextTimeout(ctx, apiServiceRetryPeriod, apiServiceRetryTimeout, true, func(ctx context.Context) (done bool, err error) { rootPaths := metav1.RootPaths{} statusContent, err = restClient.Get(). AbsPath("/"). SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } err = json.Unmarshal(statusContent, &rootPaths) if err != nil { return false, err } return !slices.Contains(rootPaths.Paths, groupPath), nil }) framework.ExpectNoError(err, "Expected to not find %s from root paths", groupPath) </del> cleanupSampleAPIServer(ctx, client, aggrclient, n, apiServiceName) }

"k8s.io/apimachinery/pkg/runtime/schema" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" utilruntime "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> discoveryendpoint "k8s.io/apiserver/pkg/endpoints/discovery/aggregated" genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"

// For each groupversion served by our resourcemanager, create an APIService // object connected to our fake APIServer <ins> var groupVersions []metav1.GroupVersion </ins> for _, versionInfo := range basicTestGroup.Versions { groupVersion := metav1.GroupVersion{ Group:   basicTestGroup.Name,

} require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

"fmt" "reflect" "strings" <ins> "testing" </ins> "time" apiextensions "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"

"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes"

} <ins> func WaitForRootPaths(t *testing.T, ctx context.Context, client testClient, requirePaths, forbidPaths sets.Set[string]) error { return wait.PollUntilContextTimeout(ctx, 250*time.Millisecond, maxTimeout, true, func(ctx context.Context) (done bool, err error) { statusContent, err := client.Discovery().RESTClient().Get().AbsPath("/").SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } rootPaths := metav1.RootPaths{} if err := json.Unmarshal(statusContent, &rootPaths); err != nil { return false, err } paths := sets.New(rootPaths.Paths...) if missing := requirePaths.Difference(paths); len(missing) > 0 { t.Logf("missing required root paths %v", sets.List(missing)) return false, nil } if present := forbidPaths.Intersection(paths); len(present) > 0 { t.Logf("present forbidden root paths %v", sets.List(present)) return false, nil } return true, nil }) } </ins> func WaitForGroups(ctx context.Context, client testClient, groups ...apidiscoveryv2beta1.APIGroupDiscovery) error { return WaitForResultWithCondition(ctx, client, func(groupList apidiscoveryv2beta1.APIGroupDiscoveryList) bool { for _, searchGroup := range groups {