Change:
What happened: After upgrading to v1.18.0 not possible to join existing node into cluster: <details<summaryFull log</summary<preW0326 11:57:39. ] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. I0326 11:57:39. ] [preflight] found NodeName empty; using OS hostname as NodeName I0326 11:57:39. ] detected and using CRI socket: [preflight] Running pre-flight checks I0326 11:57:39. ] [preflight] Running general checks I0326 11:57:39. ] validating the existence and emptiness of directory /etc/kubernetes/manifests I0326 11:57:39. ] validating the existence of file I0326 11:57:39. ] validating the existence of file /etc/kubernetes/bootstrap- I0326 11:57:39. ] validating the container runtime I0326 11:57:39. ] validating if the service is enabled and active I0326 11:57:39. ] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables I0326 11:57:39. ] validating the contents of file /proc/sys/net/ipv4/ipforward I0326 11:57:39. ] validating whether swap is enabled or not I0326 11:57:39. ] validating the presence of executable conntrack I0326 11:57:39. ] validating the presence of executable ip I0326 11:57:39. ] validating the presence of executable iptables I0326 11:57:39. ] validating the presence of executable mount I0326 11:57:39. ] validating the presence of executable nsenter I0326 11:57:39. ] validating the presence of executable ebtables I0326 11:57:39. ] validating the presence of executable ethtool I0326 11:57:39. ] validating the presence of executable socat I0326 11:57:39. ] validating the presence of executable tc I0326 11:57:39. ] validating the presence of executable touch I0326 11:57:39. ] running all checks I0326 11:57:39. ] checking whether the given node name is reachable using net.LookupHost I0326 11:57:39. ] validating kubelet version I0326 11:57:39. ] validating if the service is enabled and active I0326 11:57:39. ] validating availability of port I0326 11:57:39. ] validating the existence of file I0326 11:57:39. ] validating if the connectivity type is via proxy or direct I0326 11:57:39. ] [preflight] Discovering cluster-info I0326 11:57:39. ] [discovery] Created cluster-info discovery client, requesting info from "10.28.36.157:8443" I0326 11:57:39. ] [discovery] Requesting info from "10.28.36.157:8443" again to validate TLS against the pinned public key I0326 11:57:39. ] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.28.36.157:8443" I0326 11:57:39. ] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process I0326 11:57:39. ] [preflight] Fetching init configuration I0326 11:57:39. ] [preflight] Retrieving KubeConfig objects [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' I0326 11:57:39. ] Looking for default routes with IPv4 addresses I0326 11:57:39. ] Default route transits interface "vmbr0" I0326 11:57:39. ] Interface vmbr0 is up I0326 11:57:39. ] Interface "vmbr0" has 3 addresses :[10.28.36.164/16 2a02:2b88:1:2:9657:a5ff:fed3:b6f2/64 fe80::9657:a5ff:fed3:b6f2/64]. I0326 11:57:39. ] Checking addr 10.28.36.164/16. I0326 11:57:39. ] IP found 10.28.36.164 I0326 11:57:39. ] Found valid IPv4 address 10.28.36.164 for interface "vmbr0". I0326 11:57:39. ] Found active IP 10.28.36.164 I0326 11:57:39. ] [preflight] Running configuration dependant checks I0326 11:57:39. ] [download-certs] Skipping certs download I0326 11:57:39. ] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap- I0326 11:57:39. ] [kubelet-start] writing CA certificate at I0326 11:57:39. ] [kubelet-start] Checking for an existing Node in the cluster with name "m1c4" and status "Ready" a Node with name "m1c4" and status "Ready" already exists in the cluster. You must delete the existing Node or change the name of this new joining Node /workspace/anago-v1.18.0- (Runner).Run.func1 /workspace/anago-v1.18.0- (Runner).visitAll /workspace/anago-v1.18.0- (Runner).Run /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- (Command).execute /workspace/anago-v1.18.0- (Command).ExecuteC /workspace/anago-v1.18.0- (Command).Execute /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- /usr/local/go/src/runtime/asmamd64.s:1357 error execution phase kubelet-start (Runner).Run.func1 /workspace/anago-v1.18.0- (Runner).visitAll /workspace/anago-v1.18.0- (Runner).Run /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- (Command).execute /workspace/anago-v1.18.0- (Command).ExecuteC /workspace/anago-v1.18.0- (Command).Execute /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- /usr/local/go/src/runtime/asmamd64.s:1357 </pre</detailsThe problem is that the check introduced in violates the existing diskless node approach. Eg. we are having cluster of hundreds servers which are booding over PXE and have no any local storage. After the boot they just running command to connect to the cluster, they can be rebooted ay any time, and after reboot they will run join procedure again. Thus node object in cluster may already have some settings, labels and annotation for this node. After the joining the node just reuses them, this is working as a charm. What you expected to happen: Existing node successful joined How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Currently if you try to join the node you will face with , but I created cluster role to check this chage. Environment: Kubernetes version: Cloud provider or hardware configuration: Bare Metal OS: Kernel: Install tools: Network plugin and version Others:
/sig cluster-lifecycle /area kubeadm
we have to be clear that this approach was never officially supported by kubeadm. simply restarting a node machine would make the kubelet and the CR bring that node back up. but resetting the node and then re-joining without deleting the node object is problematic. by preventing a node with the same name from joining the cluster we are solving a security issue, as otherwise an attacker knowing a bootstrap token would be able to bring your control-plane down. ideally, your steps should now be: /priority awaiting-more-evidence
/assign
OK, but this opportunity is supported by kubelet and api-server itself, it is working fine and I don't see the reason to stop using it. We are using this approach since v1.9 and never with the problems described in , every rejoin operation of old node is just reuse name and parameters of existing resource with no any problems. This is not fully truth, because: He can just use modified kubeadm binary, or other deployment tool. And now he can also list all the nodes in cluster to make a choose which one he want to hack Bootstrap token could be generated automatically every hour and for each new boot This precaution is useless for spheres where Kubernetes is used in closed perimeter just for convenient utilizing a compute power of physical servers, eg scientific where you need just convenient tool to perform calculating and no security.
We are using this approach since v1.9 and never with the problems described in , every rejoin operation of old node is just reuse name and parameters of existing resource with no any problems. i've seen in the wild a number of times and users are shooting them self in the foot causing outage of the control-plane creating tickets for us. this is absolutely true i should have said "partially" as it needs a server side fix as well. it only eliminates the issue for users that do not recompile the source code (for this particular version, for newer version they still need the RBAC) or users that do it by accident but the malicious bootstrap can only succeed if the user knows a token. they can still shoot themself in the foot by having two nodes with the same name. so i think it has some value. using kubernetes without security is not possible - e.g. disabling certificates completely.
OK, but I still do not understand how is it possible to create two nodes with the same name, is there any steps to reproduce? I was trying but Kuberentes API is not allowing that: The same as flag, however it is not breaking existing functionality, but still alerting the user about a possible problem.
creating the Node object directly will not work. the TLS bootstrap in the kubelet for matching node names (hostnames) can still trigger the race condition in etcd/apiserver
if we will restrict rejoining only for new nodes, say, whose age is 1 minute, would it solve the problem with the race condition in etcd?
Or even better: to not restrict the join if node having status.
not sure i understand. if you join a node named "foo" to a cluster that already has "foo", the new "foo" will remain NotReady but the existing "foo" will also become NotReady due to the crash loop of control-plane components. not sure how age matters here. are you suggesting the Node having status NodeReady but with condition Unknown: what would put it in this condition? my understanding is that you are trying to rejoin nodes while their status is still "Ready", so this line breaks this for you: would changing and possibly adjusting of the KCM to lower values and waiting for the nodes to become NotReady before joining solve the issue for you?
Unfortunately intentionally or accidentally, but this check does not work: In #diff-3c566a0c1e3b213a34b39503abf4fc11R150, you're checking just existence of condition with but you also have to check the of this condition
This variant is working fine:
it did work in my tests. i guess, it would not work for the case where the condition is still present but not True, so your PR makes sense.
but that does not help you with your use case, right?
This change is fine for me, because I need to rejoin nodes only after reboot. Usually it takes more than 30 seconds (it's a time when node is coming from to state.)
so by the time you reboot the NodeReady condition now has a status instead of ? does affect this? it's default is 40s.
you might want to lower that value. also see my comment about "release note" on the PR.
You're right it is 40 seconds:
ok, i will send a cherry pick for the PR to 1.18

Discussion:
return errors.Wrapf(err, "cannot get Node %q", nodeName) } for _, cond := range node.Status.Conditions { <del> if cond.Type == v1.NodeReady { </del> <ins> if cond.Type == v1.NodeReady && cond.Status == v1.ConditionTrue { </ins> return errors.Errorf("a Node with name %q and status %q already exists in the cluster. "+ "You must delete the existing Node or change the name of this new joining Node", nodeName, v1.NodeReady) }