Issue 1:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

"too old resource version"

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:13:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

gcloud container clusters list

NAME ZONE MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
k0 us-west1-b 1.4.1 104.199.114.88 n1-standard-1 1.4.1 3 RUNNING

What happened:

When the resource version is too old the API server returns HTTP 200 OK.

$ curl -i http://127.0.0.1:8001/api/v1/watch/namespaces/default/endpoints/nginx?resourceVersion=240385

HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 18 Oct 2016 21:47:16 GMT
Content-Length: 176

{"type":"ERROR","object":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"too old resource version: 240385 (250923)","reason":"Gone","code":410}}

What you expected to happen:

When the resource version is too old the API server returns HTTP 410 to match the error code in the response body.

How to reproduce it (as minimally and precisely as possible):

Create a deployment and expose it.

Scale up the deployment to 3 nodes

Wait ~30 mins and get the endpoints backing the service

Perform a watch on the endpoint using the resource version from the previous get

Anything else do we need to know:

No.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

upgrade dns

upgrade external ip

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

1.3.7

Environment:

Cloud provider or hardware configuration: Google Cloud Container Engine

OS (e.g. from /etc/os-release): Debian GNU/Linux 7 (wheezy)

Kernel (e.g. uname -a): Linux gke-trace-pool-1-dc497b52-0ptp 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

Install tools: -

Others: -

What happened:

new node pool added to the cluster via GC Container Service Dashboard (new pool 1.3.7, default pool is: 1.3.6, master is: 1.3.7)

nodes in old (default) pool set to un-schedulable and drained

default pool removed via GC Container Service Dashboard

Services with External IP stopped working from the public IP, all requests were hanging (while pods could reach each other via internal IP and service worked with internal IP as well)

I could solve the issue only with removing and re-creating all services with External IP

What you expected to happen:

"Zero down time" upgrading with no manual step involved

How to reproduce it (as minimally and precisely as possible):

I didn't try it again, probably with upgrade and pool remove.

Anything else do we need to know:

At my previous upgrade I also experienced issues with the DNS service, that time I simply pressed upgrade on the GC Container Engine UI, but after the upgrade, some of the pods had to be manually re-created otherwise they weren't able to communicate with each other.