Change:
What happened: Prior to 1.14, when you created a watch on a resource coming from an aggregated API, you would instantly obtain the object, and could stop it when you want. Since 1.14, when you do the same, the method returns when the server sends its first event. Resources directly hosted by the central API-server (core resources, crds, etc.) are not touched by this issue. When you use higher level APIs like informers, situations like this can make your code hang, as attempts to stop the informer seem ignored (the informer cancellation logic only works once the underlying watch.Interface has been retrieved), and your code hang. This is hitting , and a workaround has been found here: However this change of behavior means that current versions of the CLI can hang when deploying Kubernetes workloads. What you expected to happen: As with resources hosted by the central API server, the watch.Interface object should be instantly returned How to reproduce it (as minimally and precisely as possible): I'll try to setup an automated test case for that with compose-on-kubernetes. In the mean-time: deploy a kube 1.14 cluster with kind / minikube / ... deploy compose-on-kubernetes on it (see ) download the latest official docker cli release and try to deploy a simple compose file to kubernetes using Anything else we need to know?: After digging a little with a debugging http proxy, I saw that the big difference seem to be that: prior to 1.14, when such a watch is issued, the response headers are immediately sent to the client with 1.14, on central-API resources, it does not change with 1.14, on aggregated API resources, the reponse headers are sent with the first event Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Using Kind, also happen with internal Docker Desktop builds OS (e.g: ): Both Kind, and Linuxkit Kernel (e.g. ): Install tools: Others:
/sig api-machinery
cc
a fix similar to might be required, where headers are flushed after write
looks like watch is flushed here:
please note that the problem manifests itself only for aggregated APIs. My guess is that there is something in the proxy created by kube-aggregator that does not propagate the flush down to the client.
Not seeing issues there, it explicitly flushes on an interval as well: Also, that code hasn't changed since 1.13 Will try to reproduce locally
Actually, looks like the golang reverse proxy reworked flushing in go1.12: the flushing now takes effect on each body write, but I don't see anything forcing a flush after headers are written I'll take the issue upstream /assign
reported upstream as
Really good catch!
workaround in until we pick up a fix from golang
picked to 1.14.1 in

Discussion:
proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: h.Location.Scheme, Host: h.Location.Host}) proxy.Transport = h.Transport proxy.FlushInterval = h.FlushInterval <del> proxy.ServeHTTP(w, newReq) </del> <ins> proxy.ServeHTTP(maybeWrapFlushHeadersWriter(w), newReq) } // maybeWrapFlushHeadersWriter wraps the given writer to force flushing headers prior to writing the response body. // if the given writer does not support http.Flusher, http.Hijacker, and http.CloseNotifier, the original writer is returned. // TODO(liggitt): drop this once https://github.com/golang/go/issues/31125 is fixed func maybeWrapFlushHeadersWriter(w http.ResponseWriter) http.ResponseWriter { flusher, isFlusher := w.(http.Flusher) hijacker, isHijacker := w.(http.Hijacker) closeNotifier, isCloseNotifier := w.(http.CloseNotifier) // flusher, hijacker, and closeNotifier are all used by the ReverseProxy implementation. // if the given writer can't support all three, return the original writer. if !isFlusher || !isHijacker || !isCloseNotifier { return w } return &flushHeadersWriter{w, flusher, hijacker, closeNotifier} } type flushHeadersWriter struct { http.ResponseWriter http.Flusher http.Hijacker http.CloseNotifier } func (w *flushHeadersWriter) WriteHeader(code int) { w.ResponseWriter.WriteHeader(code) w.Flusher.Flush() </ins> } // tryUpgrade returns true if the request was handled.

} } <ins> func TestFlushIntervalHeaders(t *testing.T) { const expected = "hi" stopCh := make(chan struct{}) backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Add("MyHeader", expected) w.WriteHeader(200) w.(http.Flusher).Flush() <-stopCh })) defer backend.Close() defer close(stopCh) backendURL, err := url.Parse(backend.URL) if err != nil { t.Fatal(err) } proxyHandler := NewUpgradeAwareHandler(backendURL, nil, false, false, nil) frontend := httptest.NewServer(proxyHandler) defer frontend.Close() req, _ := http.NewRequest("GET", frontend.URL, nil) req.Close = true ctx, cancel := context.WithTimeout(req.Context(), 10*time.Second) defer cancel() req = req.WithContext(ctx) res, err := frontend.Client().Do(req) if err != nil { t.Fatalf("Get: %v", err) } defer res.Body.Close() if res.Header.Get("MyHeader") != expected { t.Errorf("got header %q; expected %q", res.Header.Get("MyHeader"), expected) } } </ins> // exampleCert was generated from crypto/tls/generate_cert.go with the following command: //    go run generate_cert.go  --rsa-bits 512 --host example.com --ca --start-date "Jan 1 00:00:00 1970" --duration=1000000h var exampleCert = []byte(`-----BEGIN CERTIFICATE-----