Change:
1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back. edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?

Discussion:
package apimachinery import ( <ins> "crypto/rand" </ins> "crypto/x509" "encoding/json" <ins> "fmt" </ins> "io/ioutil" <ins> "math/big" </ins> "os" "strings" "time"

By("Registering the sample API server.") cleanTest(f, true) client := f.ClientSet <ins> restClient := client.Discovery().RESTClient() </ins> iclient := f.InternalClientset aggrclient := f.AggregatorClient

// We seem to need to do additional waiting until the extension api service is actually up. err = wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) { <del> request := client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders") </del> <ins> request := restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders") </ins> request.SetHeader("Accept", "application/json") _, err := request.DoRaw() if err != nil {

}) framework.ExpectNoError(err, "gave up waiting for apiservice wardle to come up successfully") <ins> flunderName := generateFlunderName("rest-flunder") </ins> // kubectl create -f flunders-1.yaml -v 9 // curl -k -v -XPOST https://localhost/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders // Request Body: {"apiVersion":"wardle.k8s.io/v1alpha1","kind":"Flunder","metadata":{"labels":{"sample-label":"true"},"name":"test-flunder","namespace":"default"}} <del> flunder := `{"apiVersion":"wardle.k8s.io/v1alpha1","kind":"Flunder","metadata":{"labels":{"sample-label":"true"},"name":"test-flunder","namespace":"default"}}` result := client.Discovery().RESTClient().Post().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").Body([]byte(flunder)).Do() </del> <ins> flunder := `{"apiVersion":"wardle.k8s.io/v1alpha1","kind":"Flunder","metadata":{"labels":{"sample-label":"true"},"name":"` + flunderName + `","namespace":"default"}}` result := restClient.Post().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").Body([]byte(flunder)).Do() </ins> framework.ExpectNoError(result.Error(), "creating a new flunders resource") var statusCode int result.StatusCode(&statusCode)

framework.Failf("Flunders client creation response was status %d, not 201", statusCode) } <ins> pods, err := client.CoreV1().Pods(namespace).List(metav1.ListOptions{}) framework.ExpectNoError(result.Error(), "getting pods for flunders service") </ins> // kubectl get flunders -v 9 // curl -k -v -XGET https://localhost/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders <del> contents, err := client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </del> <ins> contents, err := restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </ins> framework.ExpectNoError(err, "attempting to get a newly created flunders resource") var flundersList samplev1alpha1.FlunderList err = json.Unmarshal(contents, &flundersList) <del> framework.ExpectNoError(err, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </del> <ins> validateErrorWithDebugInfo(f, err, pods, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </ins> if len(flundersList.Items) != 1 { framework.Failf("failed to get back the correct flunders list %v", flundersList) } // kubectl delete flunder test-flunder -v 9 // curl -k -v -XDELETE  https://35.193.112.40/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/test-flunder <del> _, err = client.Discovery().RESTClient().Delete().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/test-flunder").DoRaw() framework.ExpectNoError(err, "attempting to delete a newly created flunders resource") </del> <ins> _, err = restClient.Delete().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/" + flunderName).DoRaw() validateErrorWithDebugInfo(f, err, pods, "attempting to delete a newly created flunders(%v) resource", flundersList.Items) </ins> // kubectl get flunders -v 9 // curl -k -v -XGET https://localhost/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders <del> contents, err = client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </del> <ins> contents, err = restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </ins> framework.ExpectNoError(err, "confirming delete of a newly created flunders resource") err = json.Unmarshal(contents, &flundersList) <del> framework.ExpectNoError(err, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </del> <ins> validateErrorWithDebugInfo(f, err, pods, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </ins> if len(flundersList.Items) != 0 { framework.Failf("failed to get back the correct deleted flunders list %v", flundersList) } <ins> flunderName = generateFlunderName("dynamic-flunder") </ins> // Rerun the Create/List/Delete tests using the Dynamic client. resources, err := client.Discovery().ServerPreferredNamespacedResources() framework.ExpectNoError(err, "getting server preferred namespaces resources for dynamic client")

Kind:       "Flunder", APIVersion: "wardle.k8s.io/v1alpha1", }, <del> ObjectMeta: metav1.ObjectMeta{Name: "test-flunder"}, </del> <ins> ObjectMeta: metav1.ObjectMeta{Name: flunderName}, </ins> Spec:       samplev1alpha1.FlunderSpec{}, } jsonFlunder, err := json.Marshal(testFlunder)

obj, err := dynamicClient.Resource(&apiResource, namespace).List(metav1.ListOptions{}) framework.ExpectNoError(err, "listing flunders using dynamic client") unstructuredList, ok := obj.(*unstructuredv1.UnstructuredList) <del> framework.ExpectNoError(err, "casting flunders list(%T) as unstructuredList using dynamic client", obj) </del> <ins> validateErrorWithDebugInfo(f, err, pods, "casting flunders list(%T) as unstructuredList using dynamic client", obj) </ins> if len(unstructuredList.Items) != 1 { framework.Failf("failed to get back the correct flunders list %v from the dynamic client", unstructuredList) } // kubectl delete flunder test-flunder <del> err = dynamicClient.Resource(&apiResource, namespace).Delete("test-flunder", &metav1.DeleteOptions{}) framework.ExpectNoError(err, "deleting flunders using dynamic client") </del> <ins> err = dynamicClient.Resource(&apiResource, namespace).Delete(flunderName, &metav1.DeleteOptions{}) validateErrorWithDebugInfo(f, err, pods, "deleting flunders(%v) using dynamic client", unstructuredList.Items) </ins> // kubectl get flunders obj, err = dynamicClient.Resource(&apiResource, namespace).List(metav1.ListOptions{}) framework.ExpectNoError(err, "listing flunders using dynamic client") unstructuredList, ok = obj.(*unstructuredv1.UnstructuredList) <del> framework.ExpectNoError(err, "casting flunders list(%T) as unstructuredList using dynamic client", obj) </del> <ins> validateErrorWithDebugInfo(f, err, pods, "casting flunders list(%T) as unstructuredList using dynamic client", obj) </ins> if len(unstructuredList.Items) != 0 { framework.Failf("failed to get back the correct deleted flunders list %v from the dynamic client", unstructuredList) } cleanTest(f, true) } <ins> func validateErrorWithDebugInfo(f *framework.Framework, err error, pods *v1.PodList, msg string, fields ...interface{}) { if err != nil { namespace := "sample-system" msg := fmt.Sprintf(msg, fields...) msg += fmt.Sprintf(" but received unexpected error:n%v", err) client := f.ClientSet ep, err := client.CoreV1().Endpoints(namespace).Get("sample-api", metav1.GetOptions{}) if err == nil { msg += fmt.Sprintf("nFound endpoints for sample-api:n%v", ep) } pds, err := client.CoreV1().Pods(namespace).List(metav1.ListOptions{}) if err == nil { msg += fmt.Sprintf("nFound pods in sample-system:n%v", pds) msg += fmt.Sprintf("nOriginal pods in sample-system:n%v", pods) } framework.Failf(msg) } } func generateFlunderName(base string) string { id, err := rand.Int(rand.Reader, big.NewInt(2147483647)) if err != nil { return base } return fmt.Sprintf("%s-%d", base, id) } </ins>