Change:
kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state. The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that

Discussion:
// GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor <ins> // Removes the given node from the record of attach updates. The node's entire // volumesToReportAsAttached list is removed. RemoveNodeFromAttachUpdates(nodeName types.NodeName) error </ins> } // AttachedVolume represents a volume that is attached to a node.

asw.addVolumeToReportAsAttached(volumeName, nodeName) } <ins> func (asw *actualStateOfWorld) RemoveNodeFromAttachUpdates(nodeName types.NodeName) error { asw.Lock() defer asw.Unlock() _, nodeToUpdateExists := asw.nodesToUpdateStatusFor[nodeName] if nodeToUpdateExists { delete(asw.nodesToUpdateStatusFor, nodeName) return nil } return fmt.Errorf("node %q does not exist in volumesToReportAsAttached list", nodeName) } </ins> func (asw *actualStateOfWorld) AddVolumeNode( uniqueName v1.UniqueVolumeName, volumeSpec *volume.Spec, nodeName types.NodeName, devicePath string) (v1.UniqueVolumeName, error) { asw.Lock()

} } <ins> // Test_RemoveNodeFromAttachUpdates_Positive expects an entire node entry to be removed // from nodesToUpdateStatusFor func Test_RemoveNodeFromAttachUpdates_Positive(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates(nodeName) // Assert if err != nil { t.Fatalf("RemoveNodeFromAttachUpdates should not return error, but got: %v", err) } if len(asw.nodesToUpdateStatusFor) > 0 { t.Fatal("nodesToUpdateStatusFor should be empty as its only entry has been deleted.") } } // Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist expects an error to be thrown // when nodeName is not in nodesToUpdateStatusFor. func Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates("node-2") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as the nodeName doesn't exist.") } if len(asw.nodesToUpdateStatusFor) != 1 { t.Fatal("The length of nodesToUpdateStatusFor should not change because no operation was performed.") } } // Test_RemoveNodeFromAttachUpdates_Negative_Empty expects an error to be thrown // when a nodesToUpdateStatusFor is empty. func Test_RemoveNodeFromAttachUpdates_Negative_Empty(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } // Act err := asw.RemoveNodeFromAttachUpdates("node-1") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as nodeToUpdateStatusFor is empty.") } if len(asw.nodesToUpdateStatusFor) != 0 { t.Fatal("The length of nodesToUpdateStatusFor should be 0.") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

"//pkg/client/listers/core/v1:go_default_library", "//pkg/controller/volume/attachdetach/cache:go_default_library", "//vendor/github.com/golang/glog:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/strategicpatch:go_default_library", ], )

"github.com/golang/glog" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/v1"

nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, err := nsu.nodeLister.Get(string(nodeName)) <del> if nodeObj == nil || err != nil { // If node does not exist, its status cannot be updated, log error and // reset flag statusUpdateNeeded back to true to indicate this node status // needs to be updated again </del> <ins> if errors.IsNotFound(err) { // If node does not exist, its status cannot be updated. // Remove the node entry from the collection of attach updates, preventing the // status updater from unnecessarily updating the node. </ins> glog.V(2).Infof( <del> "Could not update node status. Failed to find node %q in NodeInformer cache. %v", </del> <ins> "Could not update node status. Failed to find node %q in NodeInformer cache. Error: '%v'", </ins> nodeName, err) <ins> nsu.actualStateOfWorld.RemoveNodeFromAttachUpdates(nodeName) continue } else if err != nil { // For all other errors, log error and reset flag statusUpdateNeeded // back to true to indicate this node status needs to be updated again. glog.V(2).Infof("Error retrieving nodes from node lister. Error: %v", err) </ins> nsu.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) continue }

MarkVolumeAsDetached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) // Marks desire to detach the specified volume (remove the volume from the node's <del> // volumesToReportedAsAttached list) </del> <ins> // volumesToReportAsAttached list) </ins> RemoveVolumeFromReportAsAttached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) error // Unmarks the desire to detach for the specified volume (add the volume back to <del> // the node's volumesToReportedAsAttached list) </del> <ins> // the node's volumesToReportAsAttached list) </ins> AddVolumeToReportAsAttached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) }