Change:
When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth. Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)

Discussion:
import ( "fmt" <ins> "net/http" </ins> "reflect" "strconv" "strings"

"time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/rest" <ins> "k8s.io/kubernetes/pkg/api/unversioned" </ins> "k8s.io/kubernetes/pkg/client/cache" "k8s.io/kubernetes/pkg/conversion" "k8s.io/kubernetes/pkg/runtime"

defer c.watchCache.RUnlock() initEvents, err := c.watchCache.GetAllEventsSinceThreadUnsafe(watchRV) if err != nil { <del> return nil, err </del> <ins> // To match the uncached watch implementation, once we have passed authn/authz/admission, // and successfully parsed a resource version, other errors must fail with a watch event of type ERROR, // rather than a directly returned error. return newErrWatcher(err), nil </ins> } c.Lock()

return lw.storage.WatchList(context.TODO(), lw.resourcePrefix, options.ResourceVersion, Everything) } <ins> // cacherWatch implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event } func newErrWatcher(err error) *errWatcher { // Create an error event errEvent := watch.Event{Type: watch.Error} switch err := err.(type) { case runtime.Object: errEvent.Object = err case *errors.StatusError: errEvent.Object = &err.ErrStatus default: errEvent.Object = &unversioned.Status{ Status:  unversioned.StatusFailure, Message: err.Error(), Reason:  unversioned.StatusReasonInternalError, Code:    http.StatusInternalServerError, } } // Create a watcher with room for a single event, populate it, and close the channel watcher := &errWatcher{result: make(chan watch.Event, 1)} watcher.result <- errEvent close(watcher.result) return watcher } // Implements watch.Interface. func (c *errWatcher) ResultChan() <-chan watch.Event { return c.result } // Implements watch.Interface. func (c *errWatcher) Stop() { // no-op } </ins> // cacherWatch implements watch.Interface type cacheWatcher struct { sync.Mutex

"time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/testapi" apitesting "k8s.io/kubernetes/pkg/api/testing"

verifyWatchEvent(t, watcher, watch.Added, podFoo) verifyWatchEvent(t, watcher, watch.Modified, podFooPrime) <del> // Check whether we get too-old error. _, err = cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err == nil { t.Errorf("Expected 'error too old' error") </del> <ins> // Check whether we get too-old error via the watch channel tooOldWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err != nil { t.Fatalf("Expected no direct error, got %v", err) </ins> } <ins> defer tooOldWatcher.Stop() // Ensure we get a "Gone" error expectedGoneError := errors.NewGone("").(*errors.StatusError).ErrStatus verifyWatchEvent(t, tooOldWatcher, watch.Error, &expectedGoneError) </ins> initialWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", fooCreated.ResourceVersion, storage.Everything) if err != nil {