Change:
kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18, 0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4, 0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not? I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!

Discussion:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

// Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }