Change:
Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state. I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).
Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
"//staging/src/k8s.io/api/storage/v1:go_default_library", "//staging/src/k8s.io/api/storage/v1alpha1:go_default_library", "//staging/src/k8s.io/api/storage/v1beta1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library",

import ( "encoding/json" "fmt" <del> "reflect" </del> "time" corev1 "k8s.io/api/core/v1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

return err } <del> if !reflect.DeepEqual(existing.Data, data) { </del> <ins> if !apiequality.Semantic.DeepEqual(existing.Data, data) { </ins> existing.Data = data _, err = client.ConfigMaps(metav1.NamespaceSystem).Update(existing) }

name = "go_default_test", srcs = [ "admission_test.go", <ins> "broken_webhook_test.go", </ins> "main_test.go", ], tags = [

], deps = [ "//cmd/kube-apiserver/app/options:go_default_library", <ins> "//cmd/kube-apiserver/app/testing:go_default_library", </ins> "//staging/src/k8s.io/api/admission/v1beta1:go_default_library", "//staging/src/k8s.io/api/admissionregistration/v1beta1:go_default_library", <ins> "//staging/src/k8s.io/api/apps/v1:go_default_library", </ins> "//staging/src/k8s.io/api/apps/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/extensions/v1beta1:go_default_library",

<ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package admissionwebhook import ( "fmt" "testing" "time" admissionregistrationv1beta1 "k8s.io/api/admissionregistration/v1beta1" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" "k8s.io/kubernetes/test/integration/framework" ) var ( brokenWebhookName    = "integration-broken-webhook-test-webhook-config" deploymentNamePrefix = "integration-broken-webhook-test-deployment" ) func TestBrokenWebhook(t *testing.T) { var tearDownFn kubeapiservertesting.TearDownFunc defer func() { if tearDownFn != nil { tearDownFn() } }() etcdConfig := framework.SharedEtcd() server := kubeapiservertesting.StartTestServerOrDie(t, nil, nil, etcdConfig) tearDownFn = server.TearDownFn client, err := kubernetes.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("unexpected error: %v", err) } t.Logf("Creating Deployment to ensure apiserver is functional") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(0))) if err != nil { t.Fatalf("Failed to create deployment: %v", err) } t.Logf("Creating Broken Webhook that will block all operations on all objects") _, err = client.AdmissionregistrationV1beta1().ValidatingWebhookConfigurations().Create(brokenWebhookConfig(brokenWebhookName)) if err != nil { t.Fatalf("Failed to register broken webhook: %v", err) } // There is no guarantee on how long it takes the apiserver to honor the configuration and there is // no API to determine if the configuration is being honored, so we will just wait 10s, which is long enough // in most cases. time.Sleep(10 * time.Second) // test whether the webhook blocks requests t.Logf("Attempt to create Deployment which should fail due to the webhook") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(1))) if err == nil { t.Fatalf("Expected the broken webhook to cause creating a deployment to fail, but it succeeded.") } t.Logf("Restarting apiserver") tearDownFn = nil server.TearDownFn() server = kubeapiservertesting.StartTestServerOrDie(t, nil, nil, etcdConfig) tearDownFn = server.TearDownFn client, err = kubernetes.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("unexpected error: %v", err) } // test whether the webhook still blocks requests after restarting t.Logf("Attempt again to create Deployment which should fail due to the webhook") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(2))) if err == nil { t.Fatalf("Expected the broken webhook to cause creating a deployment to fail, but it succeeded.") } t.Logf("Deleting the broken webhook to fix the cluster") err = client.AdmissionregistrationV1beta1().ValidatingWebhookConfigurations().Delete(brokenWebhookName, nil) if err != nil { t.Fatalf("Failed to delete broken webhook: %v", err) } // The webhook deletion is honored in 10s. time.Sleep(10 * time.Second) // test if the deleted webhook no longer blocks requests t.Logf("Creating Deployment to ensure webhook is deleted") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(3))) if err != nil { t.Fatalf("Failed to create deployment: %v", err) } } func generateDeploymentName(suffix int) string { return fmt.Sprintf("%v-%v", deploymentNamePrefix, suffix) } func exampleDeployment(name string) *appsv1.Deployment { var replicas int32 = 1 return &appsv1.Deployment{ TypeMeta: metav1.TypeMeta{ Kind:       "Deployment", APIVersion: "apps/v1", }, ObjectMeta: metav1.ObjectMeta{ Namespace: "default", Name:      name, }, Spec: appsv1.DeploymentSpec{ Replicas: &replicas, Selector: &metav1.LabelSelector{ MatchLabels: map[string]string{"foo": "bar"}, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"foo": "bar"}, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:  "foo", Image: "foo", }, }, }, }, }, } } func brokenWebhookConfig(name string) *admissionregistrationv1beta1.ValidatingWebhookConfiguration { var path string var failurePolicy admissionregistrationv1beta1.FailurePolicyType = admissionregistrationv1beta1.Fail return &admissionregistrationv1beta1.ValidatingWebhookConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Webhooks: []admissionregistrationv1beta1.Webhook{ { Name: "broken-webhook.k8s.io", Rules: []admissionregistrationv1beta1.RuleWithOperations{{ Operations: []admissionregistrationv1beta1.OperationType{admissionregistrationv1beta1.OperationAll}, Rule: admissionregistrationv1beta1.Rule{ APIGroups:   []string{"*"}, APIVersions: []string{"*"}, Resources:   []string{"*/*"}, }, }}, // This client config references a non existent service // so it should always fail. ClientConfig: admissionregistrationv1beta1.WebhookClientConfig{ Service: &admissionregistrationv1beta1.ServiceReference{ Namespace: "default", Name:      "invalid-webhook-service", Path:      &path, }, CABundle: nil, }, FailurePolicy: &failurePolicy, }, }, } } </ins>