Change:
I am suspicious that CRD watch is not working, but was unable to find e2e test coverage to persuade myself that it is working. cf .
(in particular, I'm looking for the update tests.)
I think this was due to enabling protobuf, though it also failed in a curl for me, so something strange is happening. (Maybe we have to explicitly ask for json?)
We test CRD watch in and in , but not in . I agree we should add e2e tests as a requirement before GA, and also for the sake of defining conformance.
/area custom-resources
Are we talking about CRD or CR watch and protobuf? The later with protobuf obviously does not work, as commented.
I'm not sure "obviously" is correct - list works, for example. IIRC watch with protobuf just silently fails to do anything - it doesn't return an error. And yes I mean CR watch :-)
it should obviously not work with protobuf, but in the sense that the negotiating serializer switches to JSON if you offer it json and protobuf as alternatives. This seems to be broken.
/cc
I'd like to take this up. I'm familiarising myself with the apiextension code base, please give me some time to come up with a PR :)
We already have integration tests for this. Is an e2e test really needed?
I think we do need an e2e test for CRD watch (among other things) as a requirement for CRD going GA. The most pressing need is to define conformance (which only e2e tests can do). Currently we only test that you can create and delete a CRD, but we don't test that you can actually do anything with it: We should expand the conformance (e2e) test for CRD to include checking that the custom resource gets installed and is usable, including watches. If not for conformance, I think an e2e test for CRD watch would still be appropriate since it's an important end-user flow (a thing we expect users to do on purpose; not an edge case). For example, the apiextensions-apiserver integration test might miss problems that come from embedding into kube-apiserver. Similarly, the kubectl tests are meant to test kubectl client functionality, not to protect against regressions on the server side. Someone might reasonably change those tests in a way that suits kubectl, without realizing that doing so could leave some server code untested at the e2e level. We also need the e2e test to make sure we honor the guarantees of the raw API, to ensure clients other than kubectl will work.
For our developer purpose I would say that e2e tests are not needed when integration tests are in-place. We can even test with a real kube-apiserver including embdded into kube-apiserver if we want that. With conformance tests might have a point. Though I wonder where the line is between e2e and features we test in integration or unit tests only. We don't test all behaviour in e2e today, by far.
Since this has the help-wanted label, adding some useful info for new contributors to get started: About CRDs: If you are new to the Kubernetes API in general, I'd recommend reading the first. CustomResourceDefinition (CRD) is a way to extend the Kubernetes API. You can read more about it here: If you'd like to do some additional reading about this, I had written something up during the time I did GSoC (shamless plug): About testing: There are lots of integration tests for CRDs and you can get a general idea about testing for CRDs and apiextensions-apiserver in general. There is for CRDs. You can use this and other e2e tests to understand how to write them. contains useful information about how you can run tests. gives general info about e2e tests and mentions tests about writing good e2e tests. Also, you can ping me on Slack (I am there) if you have further questions or need help! :)

Discussion:
"aggregator.go", "certs.go", "chunking.go", <ins> "crd_watch.go", </ins> "custom_resource_definition.go", "etcd_failure.go", "framework.go",

"//vendor/k8s.io/apiextensions-apiserver/test/integration/testserver:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/api/meta:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1beta1:go_default_library",

<ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apimachinery import ( "fmt" apiextensionsv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset" "k8s.io/apiextensions-apiserver/test/integration/testserver" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/dynamic" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = SIGDescribe("CustomResourceDefinition Watch", func() { f := framework.NewDefaultFramework("crd-watch") Context("CustomResourceDefinition Watch", func() { /* Testname: crd-watch Description: Create a Custom Resource Definition and make sure watches observe events on create/delete. */ It("watch on custom resource definition objects", func() { framework.SkipUnlessServerVersionGTE(crdVersion, f.ClientSet.Discovery()) const ( watchCRNameA = "name1" watchCRNameB = "name2" ) config, err := framework.LoadConfig() if err != nil { framework.Failf("failed to load config: %v", err) } apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { framework.Failf("failed to initialize apiExtensionClient: %v", err) } noxuDefinition := testserver.NewNoxuCustomResourceDefinition(apiextensionsv1beta1.ClusterScoped) noxuVersionClient, err := testserver.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, f.ClientPool) if err != nil { framework.Failf("failed to create CustomResourceDefinition: %v", err) } defer func() { err = testserver.DeleteCustomResourceDefinition(noxuDefinition, apiExtensionClient) if err != nil { framework.Failf("failed to delete CustomResourceDefinition: %v", err) } }() ns := "" noxuResourceClient := newNamespacedCustomResourceClient(ns, noxuVersionClient, noxuDefinition) watchA, err := watchCRWithName(noxuResourceClient, watchCRNameA) Expect(err).NotTo(HaveOccurred()) watchB, err := watchCRWithName(noxuResourceClient, watchCRNameB) Expect(err).NotTo(HaveOccurred()) testCrA := testserver.NewNoxuInstance(ns, watchCRNameA) testCrB := testserver.NewNoxuInstance(ns, watchCRNameB) By("Creating first CR ") testCrA, err = instantiateCustomResource(testCrA, noxuResourceClient, noxuDefinition) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Added, testCrA) expectNoEvent(watchB, watch.Added, testCrA) By("Creating second CR") testCrB, err = instantiateCustomResource(testCrB, noxuResourceClient, noxuDefinition) Expect(err).NotTo(HaveOccurred()) expectEvent(watchB, watch.Added, testCrB) expectNoEvent(watchA, watch.Added, testCrB) By("Deleting first CR") err = deleteCustomResource(noxuResourceClient, watchCRNameA) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Deleted, nil) expectNoEvent(watchB, watch.Deleted, nil) By("Deleting second CR") err = deleteCustomResource(noxuResourceClient, watchCRNameB) Expect(err).NotTo(HaveOccurred()) expectEvent(watchB, watch.Deleted, nil) expectNoEvent(watchA, watch.Deleted, nil) }) }) }) func watchCRWithName(crdResourceClient dynamic.ResourceInterface, name string) (watch.Interface, error) { return crdResourceClient.Watch( metav1.ListOptions{ FieldSelector:  "metadata.name=" + name, TimeoutSeconds: int64ptr(600), }, ) } func instantiateCustomResource(instanceToCreate *unstructured.Unstructured, client dynamic.ResourceInterface, definition *apiextensionsv1beta1.CustomResourceDefinition) (*unstructured.Unstructured, error) { createdInstance, err := client.Create(instanceToCreate) if err != nil { return nil, err } createdObjectMeta, err := meta.Accessor(createdInstance) if err != nil { return nil, err } // it should have a UUID if len(createdObjectMeta.GetUID()) == 0 { return nil, fmt.Errorf("missing uuid: %#v", createdInstance) } createdTypeMeta, err := meta.TypeAccessor(createdInstance) if err != nil { return nil, err } if e, a := definition.Spec.Group+"/"+definition.Spec.Version, createdTypeMeta.GetAPIVersion(); e != a { return nil, fmt.Errorf("expected %v, got %v", e, a) } if e, a := definition.Spec.Names.Kind, createdTypeMeta.GetKind(); e != a { return nil, fmt.Errorf("expected %v, got %v", e, a) } return createdInstance, nil } func deleteCustomResource(client dynamic.ResourceInterface, name string) error { return client.Delete(name, &metav1.DeleteOptions{}) } func newNamespacedCustomResourceClient(ns string, client dynamic.Interface, definition *apiextensionsv1beta1.CustomResourceDefinition) dynamic.ResourceInterface { return client.Resource(&metav1.APIResource{ Name:       definition.Spec.Names.Plural, Namespaced: definition.Spec.Scope == apiextensionsv1beta1.NamespaceScoped, }, ns) } </ins>

} func expectEvent(w watch.Interface, eventType watch.EventType, object runtime.Object) { <del> if event, ok := waitForEvent(w, eventType, object); !ok { </del> <ins> if event, ok := waitForEvent(w, eventType, object, 1*time.Minute); !ok { </ins> framework.Failf("Timed out waiting for expected event: %v", event) } } func expectNoEvent(w watch.Interface, eventType watch.EventType, object runtime.Object) { <del> if event, ok := waitForEvent(w, eventType, object); ok { </del> <ins> if event, ok := waitForEvent(w, eventType, object, 10*time.Second); ok { </ins> framework.Failf("Unexpected event occurred: %v", event) } } <del> func waitForEvent(w watch.Interface, expectType watch.EventType, expectObject runtime.Object) (watch.Event, bool) { stopTimer := time.NewTimer(1 * time.Minute) </del> <ins> func waitForEvent(w watch.Interface, expectType watch.EventType, expectObject runtime.Object, duration time.Duration) (watch.Event, bool) { stopTimer := time.NewTimer(duration) </ins> defer stopTimer.Stop() for { select { <del> case actual := <-w.ResultChan(): </del> <ins> case actual, ok := <-w.ResultChan(): if ok { framework.Logf("Got : %v %v", actual.Type, actual.Object) } else { framework.Failf("Watch closed unexpectedly") } </ins> if expectType == actual.Type && (expectObject == nil || apiequality.Semantic.DeepEqual(expectObject, actual.Object)) { return actual, true }