Change:
In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Discussion:
<del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

], ) <ins> go_test( name = "go_default_test", srcs = ["core_test.go"], embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/client-go/discovery:go_default_library", "//staging/src/k8s.io/client-go/discovery/fake:go_default_library", "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", ], ) </ins> filegroup( name = "package-srcs", srcs = glob(["**"]),

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package app import ( "testing" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/discovery" fakediscovery "k8s.io/client-go/discovery/fake" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes" fakeclientset "k8s.io/client-go/kubernetes/fake" restclient "k8s.io/client-go/rest" ) // TestClientBuilder inherits ClientBuilder and can accept a given fake clientset. type TestClientBuilder struct { clientset clientset.Interface } func (TestClientBuilder) Config(name string) (*restclient.Config, error)  { return nil, nil } func (TestClientBuilder) ConfigOrDie(name string) *restclient.Config      { return nil } func (TestClientBuilder) Client(name string) (clientset.Interface, error) { return nil, nil } func (m TestClientBuilder) ClientOrDie(name string) clientset.Interface { return m.clientset } // FakeDiscoveryWithError inherits DiscoveryInterface(via FakeDiscovery) with some methods accepting testing data. type FakeDiscoveryWithError struct { fakediscovery.FakeDiscovery PossibleResources []*metav1.APIResourceList Err               error } func (d FakeDiscoveryWithError) ServerPreferredNamespacedResources() ([]*metav1.APIResourceList, error) { return d.PossibleResources, d.Err } // FakeDiscoveryWithError inherits Clientset(via FakeClientset) with overridden Discovery method. type FakeClientSet struct { fakeclientset.Clientset DiscoveryObj *FakeDiscoveryWithError } func (c *FakeClientSet) Discovery() discovery.DiscoveryInterface { return c.DiscoveryObj } // Create a fake Clientset with its Discovery method overridden. func NewFakeClientset(fakeDiscovery FakeDiscoveryWithError) *FakeClientSet { cs := &FakeClientSet{} cs.DiscoveryObj = &fakeDiscovery return cs } func TestStartResourceQuotaController_DiscoveryError(t *testing.T) { tcs := map[string]struct { discoveryError    error expectedErr       bool possibleResources []*metav1.APIResourceList }{ "No Discovery Error": { discoveryError:    nil, possibleResources: nil, expectedErr:       false, }, "Discovery Calls Partially Failed": { discoveryError: new(discovery.ErrGroupDiscoveryFailed), possibleResources: []*metav1.APIResourceList{ { GroupVersion: "create/v1", APIResources: []metav1.APIResource{ { Name:       "jobs", Verbs:      []string{"create", "list", "watch", "delete"}, ShortNames: []string{"jz"}, Categories: []string{"all"}, }, }, }, }, expectedErr: false, }, } for name, test := range tcs { testDiscovery := FakeDiscoveryWithError{Err: test.discoveryError, PossibleResources: test.possibleResources} testClientset := NewFakeClientset(testDiscovery) testClientBuilder := TestClientBuilder{clientset: testClientset} ctx := ControllerContext{ ClientBuilder:    testClientBuilder, InformerFactory:  informers.NewSharedInformerFactoryWithOptions(testClientset, time.Duration(1)), InformersStarted: make(chan struct{}), } _, _, err := startResourceQuotaController(ctx) if test.expectedErr != (err != nil) { t.Errorf("test failed for use case: %v", name) } } } </ins>