Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have k8s cluster with version . And I noticed that if I deleted worker node VM from IAAS, then node stuck in state, I created statefulset before and the pod scheduled on that node was stuck in state, and no new pod was created to replace that. Node should be deleted if no instance is found Related log When I check the log of I could see following log keeps popping up. Looks like node_controller noticed that the VM is gone but it doesn't delete the node. Related code here will return if no instance is found. But it should check if it's VM is deleted not shutdown, then return And it should cherry pick this fix avoiding deleting shutdown node What you expected to happen: After VM is deleted, the node should be deleted, pods assigned to the node should be rescheduled to other available nodes How to reproduce it (as minimally and precisely as possible): k8s cluster. In our case we use PKS. workloads on node worker node VM from IAAS Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig cloud-provider /sig vmware
Same issue as this one The issue and is cluster API repo. We should fix it inside node_controller in k/k
Hi this issue was found when testing windows cluster deployed using PKS. We do have a workaround by manually deleting node using kubectl. Still we wish the fix could go in ASAP. Could you ask someone from upstream to to review the PR that fix this issue? Thank you
, we will try to push this through for v1.20. 1.19 is approaching code thaw, so we can't push this through
Thanks Michael. I'll let my team know.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Discussion:
return false, nil } <ins> // Exists checks if VM exists and is not terminated func (vm *VirtualMachine) Exists(ctx context.Context) (bool, error) { vmMoList, err := vm.Datacenter.GetVMMoList(ctx, []*VirtualMachine{vm}, []string{"summary.runtime.powerState"}) if err != nil { klog.Errorf("Failed to get VM Managed object with property summary. err: +%v", err) return false, err } // We check for VMs which are still available in vcenter and has not been terminated/removed from // disk and hence we consider PoweredOn,PoweredOff and Suspended as alive states. aliveStates := []types.VirtualMachinePowerState{ types.VirtualMachinePowerStatePoweredOff, types.VirtualMachinePowerStatePoweredOn, types.VirtualMachinePowerStateSuspended, } currentState := vmMoList[0].Summary.Runtime.PowerState for _, state := range aliveStates { if state == currentState { return true, nil } } return false, nil } </ins> // GetAllAccessibleDatastores gets the list of accessible Datastores for the given Virtual Machine func (vm *VirtualMachine) GetAllAccessibleDatastores(ctx context.Context) ([]*DatastoreInfo, error) { host, err := vm.HostSystem(ctx)

} } <ins> for _, turnOff := range []bool{true, false} { // Turn off for checking if exist return true if turnOff { _, _ = vm.PowerOff(ctx) } exist, err := vm.Exists(ctx) if err != nil { t.Error(err) } if !exist { t.Errorf("exist=%t, expected=%t", exist, true) } // Turn back on if turnOff { _, _ = vm.PowerOn(ctx) } } </ins> for _, expect := range []bool{true, false} { active, err := vm.IsActive(ctx) if err != nil {

t.Error(err) } } <ins> // Expecting Exists func to throw error if VM deleted _, err = vm.Exists(ctx) if err == nil { t.Error("expected error") } </ins> } }

return false, err } _, err = vs.InstanceID(ctx, convertToK8sType(nodeName)) <del> if err == nil { return true, nil </del> <ins> if err != nil { if err == cloudprovider.InstanceNotFound { return false, nil } return false, err </ins> } <del> return false, err </del> <ins> return true, nil </ins> } // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes

klog.Errorf("Failed to get VM object for node: %q. err: +%v", convertToString(nodeName), err) return "", err } <del> isActive, err := vm.IsActive(ctx) </del> <ins> exists, err := vm.Exists(ctx) </ins> if err != nil { <del> klog.Errorf("Failed to check whether node %q is active. err: %+v.", convertToString(nodeName), err) </del> <ins> klog.Errorf("Failed to check whether node %q still exists. err: %+v.", convertToString(nodeName), err) </ins> return "", err } <del> if isActive { </del> <ins> if exists { </ins> return vs.vmUUID, nil } <del> klog.Warningf("The VM: %s is not in %s state", convertToString(nodeName), vclib.ActivePowerState) </del> <ins> klog.Warningf("The VM: %s doesn't exist", convertToString(nodeName)) </ins> return "", cloudprovider.InstanceNotFound }