Change:
Testing how kubelet client rotation works, I discovered that on startup kubelet uses the cert in () and not the new cert in () This prevent kubelet to start kubelet uses the latest client certificate a 1 node cluster with kubespray (kubeadm should do) ntpd/chronyd the date 11 months in the future master certs kubelet that you have a new client cert the date 11 months more in the future and restart kubelet kubelet fails to start because it uses the cert embedded in instead of Both and are set, if I remove kubelet boots but fails to renew the cert Here an easy fix: Also I noticed that is RSA where is EC/P-256 Kubernetes version (use ): v1.18.15 (I can retest in more recent version but haven't seen anything in the release notes) Cloud provider or hardware configuration: VM OS (e.g: ): Kernel (e.g. ): Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle (not sure if this is more kubelet or kubeadm)
is the embedding certificates? see the second warning under here:
you can call on the same node to make the point to the rotatable file. newer versions of kubeadm do this change automatically.
i don't see a core bug here. i missed that you are using kubespray, which uses kubeadm. please report any issues with the kubespray issue tracker first: /kind support /close
Closing this issue. <detailsIn response to : i don't see a core bug here. please triage any issues with the kubespray issue tracker first: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks kubespray indeed doesn't call anywhere, I'll go look at how to fix that
like i said, newer versions of do this automatically. this might be the result of an old cluster being upgrade using but the is remaining stale. the manual step in or calling the on a node is required.
It was a fresh deployment of k8s 1.18 / kubespray 2.15, I'm doing a fresh deployment of k8s 1.20 / kubespray master right now, but maybe kubespray is using kubeadm in weird ways / include old workarounds that leave us with having embedded certs instead of pointing to .
Found the culprit: Kubespray regenerate the conf files using kubeadm to make sure the server is up to date Thanks for the help and sorry for the spam
it would make sense to just include the kubelet-finalize phase in there too.
I prefer to just change the server using one of lineinfile/sed/kubectl so next time there is a new step like kubelet-finalize we don't break again.

Discussion:
var _ = Describe("Docker Containers", func() { var c *client.Client <ins> var ns string </ins> BeforeEach(func() { var err error c, err = loadClient() <ins> ns_, err := createTestingNS("containers", c) ns = ns_.Name </ins> Expect(err).NotTo(HaveOccurred()) }) <ins> AfterEach(func() { if err := c.Namespaces().Delete(ns); err != nil { Failf("Couldn't delete ns %s", err) } }) </ins> It("should use the image defaults if command and args are blank", func() { <del> testContainerOutput("use defaults", c, entrypointTestPod(), []string{ </del> <ins> testContainerOutputInNamespace(ns, "use defaults", c, entrypointTestPod(), []string{ </ins> "[/ep default arguments]", }) })

pod := entrypointTestPod() pod.Spec.Containers[0].Args = []string{"override", "arguments"} <del> testContainerOutput("override arguments", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override arguments", c, pod, []string{ </ins> "[/ep override arguments]", }) })

pod := entrypointTestPod() pod.Spec.Containers[0].Command = []string{"/ep-2"} <del> testContainerOutput("override command", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override command", c, pod, []string{ </ins> "[/ep-2]", }) })

pod.Spec.Containers[0].Command = []string{"/ep-2"} pod.Spec.Containers[0].Args = []string{"override", "arguments"} <del> testContainerOutput("override all", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override all", c, pod, []string{ </ins> "[/ep-2 override arguments]", }) })

for start := time.Now(); time.Since(start) < podStartTimeout; time.Sleep(5 * time.Second) { pod, err := c.Pods(ns).Get(podName) if err != nil { <del> Logf("Get pod failed, ignoring for 5s: %v", err) </del> <ins> Logf("Get pod %v in ns %v failed, ignoring for 5s: %v", podName, ns, err) </ins> continue } done, err := condition(pod)

containerName := pod.Spec.Containers[0].Name // Wait for client pod to complete. <del> expectNoError(waitForPodSuccess(c, pod.Name, containerName)) </del> <ins> expectNoError(waitForPodSuccessInNamespace(c, pod.Name, containerName, ns)) </ins> // Grab its logs.  Get host first. podStatus, err := c.Pods(ns).Get(pod.Name)