Change:
nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, err := nsu.nodeLister.Get(string(nodeName)) <del> if nodeObj == nil || err != nil { // If node does not exist, its status cannot be updated, log error and // reset flag statusUpdateNeeded back to true to indicate this node status // needs to be updated again </del> <ins> if errors.IsNotFound(err) { // If node does not exist, its status cannot be updated. // Remove the node entry from the collection of attach updates, preventing the // status updater from unnecessarily updating the node. </ins> glog.V(2).Infof( <del> "Could not update node status. Failed to find node %q in NodeInformer cache. %v", </del> <ins> "Could not update node status. Failed to find node %q in NodeInformer cache. Error: '%v'", </ins> nodeName, err) <ins> nsu.actualStateOfWorld.RemoveNodeFromAttachUpdates(nodeName) continue } else if err != nil { // For all other errors, log error and reset flag statusUpdateNeeded // back to true to indicate this node status needs to be updated again. glog.V(2).Infof("Error retrieving nodes from node lister. Error: %v", err) </ins> nsu.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) continue }

Discussion:
kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that