Change:
See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).

Discussion:
tags = ["automanaged"], deps = [ "//cmd/kube-apiserver/app/options:go_default_library", <ins> "//cmd/kube-apiserver/app/preflight:go_default_library", </ins> "//pkg/api:go_default_library", "//pkg/apis/batch:go_default_library", "//pkg/capabilities:go_default_library",

srcs = [ ":package-srcs", "//cmd/kube-apiserver/app/options:all-srcs", <ins> "//cmd/kube-apiserver/app/preflight:all-srcs", </ins> ], tags = ["automanaged"], )

<ins> package(default_visibility = ["//visibility:public"]) licenses(["notice"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", "go_test", ) go_library( name = "go_default_library", srcs = ["checks.go"], tags = ["automanaged"], ) go_test( name = "go_default_test", srcs = ["checks_test.go"], library = ":go_default_library", tags = ["automanaged"], deps = ["//vendor:k8s.io/apimachinery/pkg/util/wait"], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </ins>

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package preflight import ( "fmt" "net" "net/url" "time" ) const connectionTimeout = 1 * time.Second type connection interface { serverReachable(address string) bool parseServerList(serverList []string) error CheckEtcdServers() (bool, error) } type EtcdConnection struct { ServerList []string } func (EtcdConnection) serverReachable(address string) bool { if conn, err := net.DialTimeout("tcp", address, connectionTimeout); err == nil { defer conn.Close() return true } return false } func parseServerURI(serverURI string) (string, error) { connUrl, err := url.Parse(serverURI) if err != nil { return "", fmt.Errorf("unable to parse etcd url: %v", err) } return connUrl.Host, nil } // CheckEtcdServers will attempt to reach all etcd servers once. If any // can be reached, return true. func (con EtcdConnection) CheckEtcdServers() (done bool, err error) { // Attempt to reach every Etcd server in order for _, serverUri := range con.ServerList { host, err := parseServerURI(serverUri) if err != nil { return false, err } if con.serverReachable(host) { return true, nil } } return false, nil } </ins>

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package preflight import ( "testing" "time" utilwait "k8s.io/apimachinery/pkg/util/wait" ) func TestParseServerURIGood(t *testing.T) { host, err := parseServerURI("https://127.0.0.1:2379") if err != nil { t.Fatalf("unexpected error: %v", err) } reference := "127.0.0.1:2379" if host != reference { t.Fatal("server uri was not parsed correctly") } } func TestParseServerURIBad(t *testing.T) { _, err := parseServerURI("-invalid uri$@#%") if err == nil { t.Fatal("expected bad uri to raise parse error") } } func TestEtcdConnection(t *testing.T) { etcd := new(EtcdConnection) result := etcd.serverReachable("-not a real network address-") if result { t.Fatal("checkConnection should not have succeeded") } } func TestCheckEtcdServersEmpty(t *testing.T) { etcd := new(EtcdConnection) result, err := etcd.CheckEtcdServers() if err != nil { t.Fatalf("unexpected error: %v", err) } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestCheckEtcdServersUri(t *testing.T) { etcd := new(EtcdConnection) etcd.ServerList = []string{"-invalid uri$@#%"} result, err := etcd.CheckEtcdServers() if err == nil { t.Fatalf("expected bad uri to raise parse error") } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestCheckEtcdServers(t *testing.T) { etcd := new(EtcdConnection) etcd.ServerList = []string{""} result, err := etcd.CheckEtcdServers() if err != nil { t.Fatalf("unexpected error: %v", err) } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestPollCheckServer(t *testing.T) { err := utilwait.PollImmediate(1*time.Microsecond, 2*time.Microsecond, EtcdConnection{ServerList: []string{""}}.CheckEtcdServers) if err == nil { t.Fatal("expected check to time out") } } </ins>

utilnet "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> utilwait "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apiserver/pkg/admission" genericapiserver "k8s.io/apiserver/pkg/server" "k8s.io/apiserver/pkg/server/filters" serverstorage "k8s.io/apiserver/pkg/server/storage" "k8s.io/kubernetes/cmd/kube-apiserver/app/options" <ins> "k8s.io/kubernetes/cmd/kube-apiserver/app/preflight" </ins> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/capabilities"

"k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap" ) <ins> const etcdRetryLimit = 60 const etcdRetryInterval = 1 * time.Second </ins> // NewAPIServerCommand creates a *cobra.Command object with default parameters func NewAPIServerCommand() *cobra.Command { s := options.NewServerRunOptions()

if err := s.Features.ApplyTo(genericConfig); err != nil { return nil, nil, err } <ins> if err := utilwait.PollImmediate(etcdRetryInterval, etcdRetryLimit*etcdRetryInterval, preflight.EtcdConnection{ServerList: s.Etcd.StorageConfig.ServerList}.CheckEtcdServers); err != nil { return nil, nil, fmt.Errorf("error waiting for etcd connection: %v", err) } </ins> // Use protobufs for self-communication. // Since not every generic apiserver has to support protobufs, we