Change:
If a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster. Looking at the various cloud providers, some of them filter nodes based on the state whereas others do not. The node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed. Is there a reason why the cloud providers only return instances/vms that are "ready" instead of returning any instance/vm that matches the requested name? It seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.
If that's really the case it's unfortunate - we probably can fix logic to list everything (I guess there's a flag that you can pass to CP to give you everything, not only working machines). I have no idea who deals with that ? OTOH I think the only problem with this would be if the machine would flap, as this would be annoying and cluster autoscaler might get crazy ( If it's really unready/unavailable/whatever then it's fair to assume that nothing's working on it (if StatefulSet can live with that assumption
do you think the NodeController should/should not delete a Node from storage in this situation: Node is configured to use a cloud provider, VM is powered off but not deleted? I would argue for "should not delete".
A trivial answer is that PrivateDNSNames are not unique on AWS. A running instance and a stopped instance could have the same IP, hence same PrivateDNSName. Rob / Marek: Can you clarify with a scenario where something unexpected happens: what that is and what you expect to happen?
can we work around that somehow for ec2?
I don't know what we are working around! The concern is AWS instances that are Stopped but not Terminated? Separately, we've been working since before 1.0 to get us off PrivateDNSName on AWS though...
the issue is that, depending on cloud provider, the NodeController may delete a Node from etcd if the VM is not Running. If you power off a VM, wait longer than the NodeController's eviction threshold (5 minutes by default), and then power the VM back on later, the NodeController will have deleted the Node from etcd in the interim. Some cloud providers in the kube code base expect an instance to be Running, and if it's not, they don't include that instance in the list of instances, or a "get instance" call returns "not found" (I'm not sure which, has the full details), and so the NodeController doesn't see the powered off VM and deletes the Node.
That is the current behaviour, but you've skipped the bit where something goes wrong :-) Even if we fix this problem though do we close this issue? I would vote to close this issue now on the basis that there's no objective measure for when we can mark this issue resolved, and that we start again with a problem statement like the one you're describing, which feels actionable.
the intent of this issue was what I described. It's possible that it wasn't clear in the initial description.
At least on AWS: The motivation for the filtering is that Terminated instances still have the old PrivateDNSName listed (IIRC), and so the names aren't unique unless you exclude Terminated. But we don't have to exclude Stopped instances (power-off), and the nodes will just get marked NotReady. I don't think that will cause problems, but it may be that the correct place to fix is actually somewhere else - e.g. on kubelet rejoin. What goes wrong when the node tries to rejoin?
At least 1 thing: if you have manually labels to the node post-registration, those labels are gone when it rejoins.
That sounds like it would be correct for some classes of labels.
- small clarification: 5 min threshold has nothing to do here. It only says that if Node is unresponsive for 5 or more minutes we'll start evicting Pods from that Node (modulo rate limiting). Deleting non-existing Nodes is a separate thing - when Node skips few heartbeats (currently 4) NC checks if cloud provider still returns the instance corresponding to that Node, and if not then delete the Node instance. IIUC what wrote the current behavior is valid for AWS, as Node won't come back (it's DNS name might be used by something else).
This isn't the case for only aws. Looking at the code it appears at least openstack, aws, vsphere, ovirt, rackspace cloud providers also filter on available node. I wasn't able to verify whether azure or others were doing the filtering. If they weren't, then at best we have inconsistency with different cloud providers. IMO we shouldn't be deleting a node from kubernetes if it is just stopped or some state other than available. If the instance exists in the cloud provider then imo it should exist in kubernetes, but possibly marked unscheduable if needed.
ouch, that's even worse! Isn't that only 40-50 seconds? and I chatted yesterday and I think there are a couple of possible ways to look at this: 'm used to doing things the "old way" where I treat my servers as pets, and I want to be the one who decides when to retire/delete a server. 'm willing to treat servers as cattle, and if I ever need to take a server down so I can upgrade the kernel/packages/etc, or if a server goes offline for whatever reason, I'll just throw it away and replace it with a newer one. I'd be willing to bet that most IT organizations are still operating under 1). Justin recommended that we need to do a better job at education, and while I agree with that, there is still going to be a transitional period where sysadmins want to be making these kinds of decisions, and they're not yet willing to let the cluster automatically delete nodes. In summary, I think at the very least we need to: this behavior, if it's not already documented an option to the node controller to disable this behavior (and possibly disable it by default) more on evangelizing how we envision server management to happen in a Kubernetes world
Yes, I believe it's 40s. I'm curious why we don't want to just change the implementation of cloud provider, so that we'll use 'full list' (i.e. one including not running Nodes, so only delete ones that are really not present) in NC?
That also works for me, assuming it doesn't cause any issues.
It's strictly better than disabling it altogether, right? The drawback would be that if you create a VM which will get the same name (I guess it's a DNS name), than the suspended/whatever one, then it'll be treated by k8s as the old one, which may cause a issues when running StatefulSets (especially if old machine didn't release PV or sth, as IIUC k8s will believe that it's attached to the new machine But if you're willing to do that, then I don't see anything that will prevent you from doing so, assuming it will be flag gated.
I think stateful sets are unaffected by this unless both new and old machines are booted at the same time AND either kubelet fails to terminate the pods. I can't think of how that would happen. Not being able to unmount a volume could be caused by other admin actions and is something we can warn about.
For a change of this significance, we'll want an e2e showing the problem, and then a PR that lets the e2e pass.
I'm not sure an e2e is the best way to show this. An e2e would incur costs to run on a cloud provider so likely wouldn't be part of any standard e2e run. I think a unit test could be done though.
The PR I pushed only changes AWS. w/o this change the node is removed from kubernetes shortly after it is no longer running. With this PR the node transitions to unschedulable and details show it hasn't reported its health check.
PTAL and let me know if you see issues with this approach
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale

Discussion:
return false, fmt.Errorf("multiple instances found for instance: %s", instanceID) } <ins> state := instances[0].State.Name if *state == ec2.InstanceStateNameTerminated { glog.Warningf("the instance %s is terminated", instanceID) return false, nil } </ins> return true, nil }

return types.NodeName(aws.StringValue(i.PrivateDnsName)) } <ins> var aliveFilter = []string{ ec2.InstanceStateNamePending, ec2.InstanceStateNameRunning, ec2.InstanceStateNameShuttingDown, ec2.InstanceStateNameStopping, ec2.InstanceStateNameStopped, } </ins> // Returns the instance with the specified node name // Returns nil if it does not exist func (c *Cloud) findInstanceByNodeName(nodeName types.NodeName) (*ec2.Instance, error) { privateDNSName := mapNodeNameToPrivateDNSName(nodeName) filters := []*ec2.Filter{ newEc2Filter("private-dns-name", privateDNSName), <del> newEc2Filter("instance-state-name", "running"), </del> <ins> // exclude instances in "terminated" state newEc2Filter("instance-state-name", aliveFilter...), </ins> } instances, err := c.describeInstances(filters)

} func TestFindInstanceByNodeNameExcludesTerminatedInstances(t *testing.T) { <ins> awsStates := []struct { id       int64 state    string expected bool }{ {0, ec2.InstanceStateNamePending, true}, {16, ec2.InstanceStateNameRunning, true}, {32, ec2.InstanceStateNameShuttingDown, true}, {48, ec2.InstanceStateNameTerminated, false}, {64, ec2.InstanceStateNameStopping, true}, {80, ec2.InstanceStateNameStopped, true}, } </ins> awsServices := newMockedFakeAWSServices(TestClusterId) nodeName := types.NodeName("my-dns.internal")

tag.Value = aws.String(TestClusterId) tags := []*ec2.Tag{&tag} <del> var runningInstance ec2.Instance runningInstance.InstanceId = aws.String("i-running") runningInstance.PrivateDnsName = aws.String(string(nodeName)) runningInstance.State = &ec2.InstanceState{Code: aws.Int64(16), Name: aws.String("running")} runningInstance.Tags = tags var terminatedInstance ec2.Instance terminatedInstance.InstanceId = aws.String("i-terminated") terminatedInstance.PrivateDnsName = aws.String(string(nodeName)) terminatedInstance.State = &ec2.InstanceState{Code: aws.Int64(48), Name: aws.String("terminated")} terminatedInstance.Tags = tags </del> <ins> var testInstance ec2.Instance testInstance.PrivateDnsName = aws.String(string(nodeName)) testInstance.Tags = tags </ins> <del> instances := []*ec2.Instance{&terminatedInstance, &runningInstance} awsServices.instances = append(awsServices.instances, instances...) </del> <ins> awsDefaultInstances := awsServices.instances for _, awsState := range awsStates { id := "i-" + awsState.state testInstance.InstanceId = aws.String(id) testInstance.State = &ec2.InstanceState{Code: aws.Int64(awsState.id), Name: aws.String(awsState.state)} </ins> <del> c, err := newAWSCloud(CloudConfig{}, awsServices) if err != nil { t.Errorf("Error building aws cloud: %v", err) return } </del> <ins> awsServices.instances = append(awsDefaultInstances, &testInstance) </ins> <del> instance, err := c.findInstanceByNodeName(nodeName) </del> <ins> c, err := newAWSCloud(CloudConfig{}, awsServices) if err != nil { t.Errorf("Error building aws cloud: %v", err) return } </ins> <del> if err != nil { t.Errorf("Failed to find instance: %v", err) return } </del> <ins> resultInstance, err := c.findInstanceByNodeName(nodeName) </ins> <del> if *instance.InstanceId != "i-running" { t.Errorf("Expected running instance but got %v", *instance.InstanceId) </del> <ins> if awsState.expected { if err != nil || resultInstance == nil { t.Errorf("Expected to find instance %v", *testInstance.InstanceId) return } if *resultInstance.InstanceId != *testInstance.InstanceId { t.Errorf("Wrong instance returned by findInstanceByNodeName() expected: %v, actual: %v", *testInstance.InstanceId, *resultInstance.InstanceId) return } } else { if err == nil && resultInstance != nil { t.Errorf("Did not expect to find instance %v", *resultInstance.InstanceId) return } } </ins> } }