Change:
Updated description: The default path points to a transient location () that is removed on reboot. The default location points to a persistent location. This means that after a reboot, a kubelet can be stuck with configuration that is missing credentials needed to reach the API server. users: Updated kubeadm packages (deb , rpm ) have been released with a config change that prevents this issue from occurring. If you have a node that is already in this state: Verify this is the issue you are experiencing: the file references the file does not exist update the kubeadm package remove the file restart the kubelet to allow it to bootstrap Custom deployments should set the flag to a non-transient location, like In 1.8.0, TLS bootstrapping (using ) writes certs to the path and a kubeconfig referencing those certs to the path. That means that on reboot: the client certs are removed is ignored, because exists cannot be loaded, which causes the kubelet to log an error (but not exit) In 1.7, TLS bootstrapping inlined the certificates in the written , which sidestepped the issue with the getting removed. Original description: /sig-autoscaling /kind bug What happened: I have 1 master(174) and 2 nodes(175,171). I am restarting one of the node(171), when it came back, the kubelet on this server does not get started with errors: Sep 30 19:54:46 kube171 kubelet[2458]: error: failed to run Kubelet: no client provided, cannot use webhook authorization ! when I look into the /var/run/kubernetes directory, two files are missing compare to pre-restart root ls -alh total 16K drwxr-xr-x 2 root root 120 Sep 30 19:43 . drwxr-xr-x 27 root root 1020 Sep 30 19:43 .. -rw-r--r-- 1 root root 887 Sep 30 19:43 kubelet- -rw------- 1 root root 227 Sep 30 19:43 kubelet- -rw-r--r-- 1 root root 1.1K Sep 30 19:43 -rw------- 1 root root 1.7K Sep 30 19:43 And after restarting xx cd /var/run/kubernetes xx ls -alh total 8.0K drwxr-xr-x 2 root root 80 Sep 30 19:45 . drwxr-xr-x 25 root root 960 Sep 30 19:45 .. -rw-r--r-- 1 root root 1.1K Sep 30 19:45 -rw------- 1 root root 1.7K Sep 30 19:45 xx What you expected to happen: Does the node suppose to be back be "READY" status automatically? How to reproduce it (as minimally and precisely as possible): Just power off / restart the node can reproduce the issue Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration*: Bare Metal - Self hosted VMs OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.2 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux kube171 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm, kubectl, etc. Nothing special Others: None !
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig-autoscaling
What is the full invocation of the command and content of the (and if used) files (with confidential data redacted)?
cc
Here is the 10- file content which I think might be the kubelet command you were asking for: ! the file: ! And the bootstrap- ! No worry about data, this is just a local experimental environment. Hope these will do the help.
I have the same issue. /var/run/kubernetes/kubelet- and /var/run/kubernetes/kubelet- gets deleted when I reboot the node. Restarting kubelet only works fine. I can run kubelet after restoring these files from a backup.
Did the restart occur after an upgrade from 1.7.x, or was the kubelet at 1.8 before and after the restart?
Kubelet 1.8 is the only one version on my node (before and after the restart).
cc as this bug affects kubeadm a bit as well
this is a clean 1.8 installation with kubeadm, not upgrade from any version.
Thank you. It turns out that there is a bug in a feature kubeadm uses (Kubelet client certificate rotation). Meanwhile, can you remove the flag flag in please and see if that "fixes" the issue? We might release new debs tomorrow or so removing that flag as we don't want to enable a feature that does not work. We also hope that we can fix the underlying cert rotation issue in v1.8.1, but that is not clear yet... Thanks again for reporting this bug!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/auth sig/cluster-lifecycle: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
cc
its my pleasure. But remove the --rotate-certificates flag Or set --rotate-certificates=false Or remove the environment variable KUBELETCERTIFICATEARGS from the kubelet ExecStart do't work, the error is still the same after reboot. !
I'm working on recreating the scenario as well, but just to be sure, did you run after modifying the unit file?
actually, I think this is an issue with being a transient folder, completely removed/recreated on reboot (not on kubelet restart). Looking at the timestamps, the whole folder is being recreated. The serving cert/key are regenerated at kubelet start, but the client certs obtained via bootstrapping remain missing. In short, the issue is: defaults to a transient directory (/var/run/kubernetes) places the file in a non-transient directory (), so it survives a reboot, but references files that get deleted on reboot reboot, the cert files are removed along with the rest of time the kubelet starts: exists, so it skips obtaining a client cert using references a missing client cert/key, so the kubelet fails to contact the API server, logs a warning (not an error): there is no serving cert/key, so the kubelet creates one That leaves us with the observed state: exists with only the self-signed cert/key (with a newer timestamp) the client cert/key are gone the kubelet is running in a state that does not allow it to reach the API server In 1.7, this was still occurring with the server cert/key, but the client certificate/key data was inlined in the generated kubeconfig, rather than being placed in cert/key files in the specified The most obvious immediate resolution would be to set to a non-transient location, like , etc. Note that gce/gke Long-term, I'd expect the following code fixes: change the default to a non-transient location (since also defaults to a non-transient location) error if cannot be loaded, rather than continue with broken config
I was seeing this too. I can confirm setting --cert-dir to a non-transient dir, deleting and restarting results in a working node.
so setting to in the meantime in debs/rpms will miligate this issue, right? Let's make sure sees this as well ^ Thanks for the digging
Yes. 1.7 systems will tolerate that change (since the only thing that was used for in 1.7.x was self-signed serving certs, which will happily regenerate in the new location) New 1.8.0 systems will happily generate TLS-bootstrapped certs in the new location Existing 1.8.0 systems that already bootstrapped will have to remove their file, since their files have the bad location persisted in them.
Opened several PRs: adds the issue and workaround to the 1.8.0 known issues list changes the default for the kubelet, will pick to 1.8.x changes the for kubeadm = 1.8.0
Using kubeadm here and the proposed solution (referenced after this) hasn't worked for me: update the kubeadm package to 1.8.0-1 remove the file restart the kubelet to allow it to bootstrap I'm getting after restarting kubelet and the following entry in the API server logs: I tried making the kubelet to bootstrap itself again using but no luck (and got a deprecation warning). Any recommendations? EDIT: Since this was a production server (and today is Sunday) I had to find a way to get it back ASAP. This is what I did to get it back: Generate a new CSR based from the kubelet- setting the Organization as and the FQDN as Get that to my laptop (where I have the ) Send the CSR to the cluster to be auto-approved and Issued Get the CRT: (I'm on Mac so I had to use instead of ) Create the in the node with the output from the previous step Restart kubelet Keep enjoying my Sunday However, I have the feeling that if this node reboots I'll have to manually do this all over again...
In case anyone else is facing similar issues on an older kubeadm cluster. I fixed this by doing on the node, then making a new token on a master using and re-joining the node using
How I wish I found this issue yesterday Thanks to god ,I just use this commands by myself but A large waste of time for about 2 hours
I just copy and from other nodes. And then
kubelet --version Kubernetes v1.18.8-aliyun.1

Discussion:
PortOpenCheck{port: 10252}, HTTPProxyCheck{Proto: "https", Host: cfg.API.AdvertiseAddress, Port: int(cfg.API.BindPort)}, DirAvailableCheck{Path: filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ManifestsSubDirName)}, <del> DirAvailableCheck{Path: "/var/lib/kubelet"}, </del> FileContentCheck{Path: bridgenf, Content: []byte{'1'}}, SwapCheck{}, InPathCheck{executable: "ip", mandatory: true},

ServiceCheck{Service: "docker", CheckIfActive: true}, PortOpenCheck{port: 10250}, DirAvailableCheck{Path: filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ManifestsSubDirName)}, <del> DirAvailableCheck{Path: "/var/lib/kubelet"}, </del> FileAvailableCheck{Path: cfg.CACertPath}, FileAvailableCheck{Path: filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.KubeletKubeConfigFileName)}, FileContentCheck{Path: bridgenf, Content: []byte{'1'}},

RequireKubeConfig:       false, KubeConfig:              flag.NewStringFlag("/var/lib/kubelet/kubeconfig"), ContainerRuntimeOptions: *NewContainerRuntimeOptions(), <del> CertDirectory:           "/var/run/kubernetes", </del> <ins> CertDirectory:           "/var/lib/kubelet/pki", </ins> RootDirectory:           v1alpha1.DefaultRootDir, // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations