Issue 1:
Currently, each RESTStorage implementation has its own storage object, which eventually ends up being the same etcd instance for all components.

We should consider instead having apiserver pass a storage destination into each RESTStorage method. This would make it much more clear how to write plugins, both those that you compile into the master binary, and those that are external binaries.

This would also let us eventually do fancy things like sharding (or isolating) keys across different etcd clusters without having to make sweeping changes to all the storage objects.

(See #1355, which sparked this)

Issue 2:
k8s has been working with builds up to yesterday. Today I did a pull, cluster/kube-down.sh, hack/dev-build-and-up.sh

No apparent errors during build but the "waiting for cluster initialization" never finishes.

There are errors in the master apiserver.log, controller-manager.log and scheduler.log. I have included snippets of each log below. I can post entire log files if needed.

The last commit on this pull: commit 87278a6

scheduler.log

E0904 17:56:20.755368 14056 reflector.go:77] unexpected watch close
E0904 17:56:20.755474 14056 reflector.go:77] unexpected watch close
E0904 17:56:21.756530 14056 reflector.go:77] unexpected watch close

controller-manager.log

E0904 17:56:05.192929 11254 replication_controller.go:185] Synchronization error: request [&http.Request{Method:"GET", URL:(*url.URL)(0xc21000b4d0), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.
ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"127.0.0.1:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:
"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {"kind":"Status","creationTimestamp":null,"apiVersion":"v1beta1","status":"failure","message":"501: All the given peers are not reac
hable (Tried to connect to each peer twice and failed) [0]","code":500} (&errors.errorString{s:"request [&http.Request{Method:\"GET\", URL:(*url.URL)(0xc21000b4d0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.
Header{}, Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:\"127.0.0.1:8080\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Head
er(nil), RemoteAddr:\"\", RequestURI:\"\", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {\"kind\":\"Status\",\"creationTimestamp\":null,\"apiVersion\":\"v1beta1\",\"status\":\"failure\",\"message\
":\"501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\",\"code\":500}"})

apiserver.log (pattern repeated many times)

I0904 17:55:06.740711 08443 apiserver.go:62] -machines is overwritten by -minion_regexp
E0904 17:55:07.656846 08443 healthy_registry.go:77] kubernetes-minion-1.c.rxwikiborg.internal failed health check with error: Get http://kubernetes-minion-1.c.rxwikiborg.internal:10250/healthz: dial tcp 10.240.28.63:10250: connection refused
E0904 17:55:07.714646 08443 healthy_registry.go:77] kubernetes-minion-2.c.rxwikiborg.internal failed health check with error: Get http://kubernetes-minion-2.c.rxwikiborg.internal:10250/healthz: dial tcp 10.240.68.185:10250: connection refused
E0904 17:55:07.939884 08443 healthy_registry.go:77] kubernetes-minion-3.c.rxwikiborg.internal failed health check with error: Get http://kubernetes-minion-3.c.rxwikiborg.internal:10250/healthz: dial tcp 10.240.49.203:10250: connection refused
E0904 17:55:07.944329 08443 healthy_registry.go:77] kubernetes-minion-4.c.rxwikiborg.internal failed health check with error: Get http://kubernetes-minion-4.c.rxwikiborg.internal:10250/healthz: dial tcp 10.240.22.199:10250: connection refused
E0904 17:55:07.944772 08443 pod_cache.go:77] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
I0904 17:55:07.945144 08443 log.go:134] GET /api/v1beta1/services?labels=: (288.377us) 500
goroutine 25 [running]:
github.com/GoogleCloudPlatform/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc2102f82a0, 0x1f4)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/httplog/log.go:153 +0x9c
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.writeJSON(0x1f4, 0x7fa5d4188770, 0xc21001fb40, 0x7023a0, 0xc2102ebbe0, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:157 +0x11f
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.errorJSON(0x7fa5d4199330, 0xc2102faba0, 0x7fa5d4188770, 0xc21001fb40, 0x7fa5d4199410, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:164 +0x7d
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).handleRESTStorage(0xc2102fa000, 0xc2102fd180, 0x1, 0x1, 0xc2101088f0, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:79 +0x496
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).ServeHTTP(0xc2102fa000, 0x7fa5d4199410, 0xc2102f82a0, 0xc2101088f0)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:49 +0x280
net/http.func·010(0x7fa5d4199410, 0xc2102f82a0, 0xc2101088f0)
/usr/local/go/src/pkg/net/http/server.go:1252 +0xb0
net/http.HandlerFunc.ServeHTTP(0xc210309840, 0x7fa5d4199410, 0xc2102f82a0, 0xc2101088f0)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc2102fa030, 0x7fa5d4199410, 0xc2102f82a0, 0xc2101088f0)
/usr/local/go/src/pkg/net/http/server.go:1496 +0x163
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.func·002(0x7fa5d41993d8, 0xc2102ebaa0, 0xc2101088f0)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:140 +0x197
net/http.HandlerFunc.ServeHTTP(0xc21030c430, 0x7fa5d41993d8, 0xc2102ebaa0, 0xc2101088f0)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*defaultAPIServer).ServeHTTP(0xc2
E0904 17:55:07.945493 08443 endpoints_controller.go:51] Failed to list services: request [&http.Request{Method:"GET", URL:(*url.URL)(0xc21010e700), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"127.0.0.1:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {"kind":"Status","creationTimestamp":null,"apiVersion":"v1beta1","status":"failure","message":"501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]","code":500}
I0904 17:55:17.946028 08443 log.go:134] GET /api/v1beta1/services?labels=: (222.446us) 500
goroutine 25 [running]:
github.com/GoogleCloudPlatform/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc2100388a0, 0x1f4)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/httplog/log.go:153 +0x9c
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.writeJSON(0x1f4, 0x7fa5d4188770, 0xc21001fb40, 0x7023a0, 0xc2102c3960, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:157 +0x11f
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.errorJSON(0x7fa5d4199330, 0xc2102e6db0, 0x7fa5d4188770, 0xc21001fb40, 0x7fa5d4199410, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:164 +0x7d
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).handleRESTStorage(0xc2102fa000, 0xc2102ee160, 0x1, 0x1, 0xc210037750, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:79 +0x496
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).ServeHTTP(0xc2102fa000, 0x7fa5d4199410, 0xc2100388a0, 0xc210037750)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:49 +0x280
net/http.func·010(0x7fa5d4199410, 0xc2100388a0, 0xc210037750)
/usr/local/go/src/pkg/net/http/server.go:1252 +0xb0
net/http.HandlerFunc.ServeHTTP(0xc210309840, 0x7fa5d4199410, 0xc2100388a0, 0xc210037750)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc2102fa030, 0x7fa5d4199410, 0xc2100388a0, 0xc210037750)
/usr/local/go/src/pkg/net/http/server.go:1496 +0x163
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.func·002(0x7fa5d41993d8, 0xc2102c3780, 0xc210037750)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:140 +0x197
net/http.HandlerFunc.ServeHTTP(0xc21030c430, 0x7fa5d41993d8, 0xc2102c3780, 0xc210037750)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*defaultAPIServer).ServeHTTP(0xc2
E0904 17:55:17.946325 08443 endpoints_controller.go:51] Failed to list services: request [&http.Request{Method:"GET", URL:(*url.URL)(0xc210071a10), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"127.0.0.1:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"
", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {"kind":"Status","creationTimestamp":null,"apiVersion":"v1beta1","status":"failure","message":"501: All the given peers are not reach
able (Tried to connect to each peer twice and failed) [0]","code":500}
I0904 17:55:27.947293 08443 log.go:134] GET /api/v1beta1/services?labels=: (327.905us) 500
goroutine 25 [running]:
github.com/GoogleCloudPlatform/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc210201000, 0x1f4)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/httplog/log.go:153 +0x9c
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.writeJSON(0x1f4, 0x7fa5d4188770, 0xc21001fb40, 0x7023a0, 0xc2102993c0, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:157 +0x11f
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.errorJSON(0x7fa5d4199330, 0xc2102d3780, 0x7fa5d4188770, 0xc21001fb40, 0x7fa5d4199410, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:164 +0x7d
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).handleRESTStorage(0xc2102fa000, 0xc2102eefb0, 0x1, 0x1, 0xc21028b340, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:79 +0x496
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).ServeHTTP(0xc2102fa000, 0x7fa5d4199410, 0xc210201000, 0xc21028b340)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:49 +0x280
net/http.func·010(0x7fa5d4199410, 0xc210201000, 0xc21028b340)
/usr/local/go/src/pkg/net/http/server.go:1252 +0xb0
net/http.HandlerFunc.ServeHTTP(0xc210309840, 0x7fa5d4199410, 0xc210201000, 0xc21028b340)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc2102fa030, 0x7fa5d4199410, 0xc210201000, 0xc21028b340)
/usr/local/go/src/pkg/net/http/server.go:1496 +0x163
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.func·002(0x7fa5d41993d8, 0xc2102991e0, 0xc21028b340)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:140 +0x197
net/http.HandlerFunc.ServeHTTP(0xc21030c430, 0x7fa5d41993d8, 0xc2102991e0, 0xc21028b340)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*defaultAPIServer).ServeHTTP(0xc2
E0904 17:55:27.947560 08443 endpoints_controller.go:51] Failed to list services: request [&http.Request{Method:"GET", URL:(*url.URL)(0xc21000b850), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.R
eadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"127.0.0.1:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"
", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}] failed (500) 500 Internal Server Error: {"kind":"Status","creationTimestamp":null,"apiVersion":"v1beta1","status":"failure","message":"501: All the given peers are not reach
able (Tried to connect to each peer twice and failed) [0]","code":500}
E0904 17:55:37.945823 08443 pod_cache.go:77] Error synchronizing container list: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
I0904 17:55:37.948247 08443 log.go:134] GET /api/v1beta1/services?labels=: (261.546us) 500
goroutine 25 [running]:
github.com/GoogleCloudPlatform/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc2101db6c0, 0x1f4)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/httplog/log.go:153 +0x9c
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.writeJSON(0x1f4, 0x7fa5d4188770, 0xc21001fb40, 0x7023a0, 0xc210299aa0, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:157 +0x11f
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.errorJSON(0x7fa5d4199330, 0xc2102c2660, 0x7fa5d4188770, 0xc21001fb40, 0x7fa5d4199410, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:164 +0x7d
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).handleRESTStorage(0xc2102fa000, 0xc2102d5060, 0x1, 0x1, 0xc21028b8f0, ...)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:79 +0x496
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.(*RESTHandler).ServeHTTP(0xc2102fa000, 0x7fa5d4199410, 0xc2101db6c0, 0xc21028b8f0)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/resthandler.go:49 +0x280
net/http.func·010(0x7fa5d4199410, 0xc2101db6c0, 0xc21028b8f0)
/usr/local/go/src/pkg/net/http/server.go:1252 +0xb0
net/http.HandlerFunc.ServeHTTP(0xc210309840, 0x7fa5d4199410, 0xc2101db6c0, 0xc21028b8f0)
/usr/local/go/src/pkg/net/http/server.go:1220 +0x40
net/http.(*ServeMux).ServeHTTP(0xc2102fa030, 0x7fa5d4199410, 0xc2101db6c0, 0xc21028b8f0)
/usr/local/go/src/pkg/net/http/server.go:1496 +0x163
github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver.func·002(0x7fa5d41993d8, 0xc210299960, 0xc21028b8f0)
/var/src/apiserver/src/github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver/apiserver.go:140 +0x197
net/http.HandlerFunc.ServeHTTP(0xc21030c430, 0x7fa5d41993d8, 0xc210299960, 0xc21028b8f0)