Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a cluster with a fairly large number of nodes (~50), some nodes take up to 30 minutes after being to the kubernetes node list to be fully initialized. I am using the openstack external cloud provider. Each call of the AddCloudNode method (in ) takes around 2-4 seconds, though this occasionally spikes up to ~10 seconds. I also see that in the openstack-cloud-controller-manager logs, nodes are being initialized multiple times. This is particularly prominent for the nodes that enter the node list the latest. ! The cloud provider receives an Update event from each node every 10 seconds, but they are supposed to be ignored after the node has been initialized. The problem is that when the event rate (50 per 10 seconds ~ 5 per second) is higher than the processing rate (0.5 per second at best), there must be a queue of events that gets blocked up. The first nodes to be registered in kubernetes start filling up this queue with Update events, which is why it takes 30 minutes for the nodes which are last to get their first Add event processed. The reason that the Update events are not ignored is that the Update event checks the taints on the node object that is provided by the event, which I assume is a snapshot of the node state from the time when the event was triggered. Therefore any Update events that are triggered before the node has its initial Add event processed are treated again as Add events, taking up even more processing time. I tried modifying UpdateCloudNode to get and check the current state of the node, and when running the openstack-cloud-controller-manager with that change all nodes are successfully initialized within 5-20 seconds of first appearing in the node list. What you expected to happen: Each node should only be initialized once, and should be initialized very quickly after appearing in the kubernetes node list. How to reproduce it (as minimally and precisely as possible): Use a cloud where the AddCloudNode method takes at least a few seconds and create a cluster with a large amount of nodes, or just add a time.Sleep for a few seconds to AddCloudNode. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /area controller-manager
/sig scalability Doing so might overload the apiserver, as for each update event we request a node object. cc
thanks for reporting this issue! Can you share what version of the OpenStack external cloud controller manager you are using? cc
/assign
/area cloudprovider
I think in our production kubernetes deployments it's version 0.2, but for my own testing I was replacing with openstack-cloud-controller-manager built from the master branch (and modifying the vendored , which was 1.13). Both were affected. if getting the current state of the node is not feasible at scale, is there any way to determine the age of the Update event when it gets around to being processed? Either by some property of the node that is captured by the event, or by setting up some internal tracking? We know that the Update event snapshot of the node object must be out of date if it was taken before the time when the Add event was processed (assuming that the Add successfully modifies the node), so it would just be a case of discarding the Update event if we can determine that it was from before the Add. I wasn't able to track down exactly where the events originate from, could someone point me in the right direction?
This isn't ideal, but it's not completely unreasonable given how many calls to the cloud provider is done as part of node registration. Yeah, this is a bit unfortunate and I would be happy to discuss ways to improve this. The reason this is hard to work around is that node update events happen frequently due to updates that are triggered from node heartbeats so it's very likely that we receive an update event while the node's original add event is being processed. This may be better starting in v1.14 as we use for node heartbeats but I need to do some testing to validate this. One solution I think this of is to have a map that stores the current list of nodes under registration (protected by a mutex lock) and only try registering it if it's not in the list. Thoughts?
I opened a tracking issue for this here , this will probably be p0 or p1 (high priority) for v1.15. Thanks for reporting it! Let me know if you'd like to work on this!
before I made the change checking the current state of the node, I did try a quick change which was to store the node names that have already been processed in a map, and check that to skip nodes in the Update event (which also worked). I thought that might cause issues later on, though. Is it definitely the case that a node should never be reprocessed by an Update event -AddCloudNode method once it has been successfully initialized? There's no reason that a node would need to be initialized again, even if the VM is shut down and restarted with a different IP or something like that? I'll start working on this myself, trying the map idea again.
This is enough to fix the problem. The machineIDs are tracked as well since a resize down and then back up can bring back nodes with the same names as old ones. Those nodes would skip the check in Update and go to the Add event first which would replace the name-ID in the map, so it shouldn't really matter, but since mentioned wanting to merge the Add/Update methods this would already support that. If this looks good to you I can open a pull request for it.
To be clear, I think the map should only contain nodes that are undergoing registration, not nodes that are already registered. That way if a node wants to register again we can do so. If a node is shut down & its addresses change, we have a controller to catch that which is separate from node registration. See
Add & Update functions are intentionally different due to logging (we want to log on Add events, but not on Update events)
I agree, the map should contain contain only nodes going through registration. As for deduplication, we can log when initializing the nodes, this might help us refactor this. But this is not a blocker for this issue.
At what point should a node be removed from the map? When the Update method no longer has the uninitialized taint on it's node object?
Yes, I think here would be fine Would be even better if you do it in a line after you add the node to the map.
The problem isn't that the Add and Update events are being processed at the same time, not at all. For example in these logs the initializations of a single node are spread far apart. If my understanding is correct, these would have been the first Add and first 8 Update events triggered after the node appeared in kubernetes. It's just that with the huge volume of Updates from all the other nodes to get through, it never got around to processing these until many minutes after they should have been processed. By which time node has been initialized by the first Add event but the Update events don't know that. So the node gets initialized again for each Update event, blocking the Add events of other nodes, and contributing to why it took 30 minutes for the slowest node to be initialized. The map needs to retain the name/ID of the node it initialized after the initialization is finished because there could be any number of outdated Update events left to process. My suggestion is that once an Update event is observed where it sees the node as initialized, the name/ID can be removed and the following Updates can be rejected the same as they are now.
Seems to me like some API calls to the provider is taking a really long time there which leads to multiple update events piling up before the node was successfully registered. Especially if the open stack provider has backoff delay for API calls. The map needs to retain the name/ID of the node it initialized after the initialization is finished because there could be any number of outdated Update events left to process. My suggestion is that once an Update event is observed where it sees the node as initialized, the name/ID can be removed and the following Updates can be rejected the same as they are now. If a node exists in the map (i.e it's currently being initialized) then just drop the update event. I think it achieves the same behavior but it's easier to understand when you read the controller. You also wouldn't check the map anyways if the node doesn't have the taint so you wouldn't be missing anything from subsequent update events. Either way, I think both approaches would work, we can discuss further in the PR :)
Sorry, I should clarify, in the logs in my previous comment I was grepping for that one particular node - there were other nodes being initialized in between those lines. The rate of initializations was fairly constant, looking to be around 2 seconds each. After 30 minutes all nodes are properly initialized and there is a big queue of Update events from just the last couple of nodes left over - I'm not sure why these are handled quicker than the ones before, perhaps since it's only for a few nodes the results of the queries to openstack are able to be cached somewhere. ! Taking the logs as an example, the node actually gets initialized at exactly as it says on the first line. The event on the last line is processed finishing at , but the event was triggered before the first event was processed, so the apiv1.Node that the handler handles represents the state of the node before it was initialized. Since it sees the node as uninitialized, it does not discard the event and does the initialization again.
Sorry if I'm missing anything here. For any node with a cloud taint, if we record (via a map) that the node is being initialized (i.e. before we call ), then all following update events where the node still has a taint can be ignored (we can check that by looking at the map). When returns without error, we can remove the node from the map. That should solve ^ this case I think. What do you think?
My thought was that the node should be to the map at the (successful) end of the Add event initializeNode. If the Add fails then the Update should be able to try again, yes? Which call of initializeNode? The initial one from the Add event? Because that should be the only time initializeNode actually get called as long as there are no errors. If you add to the map at the start of the Add initialize and remove from the map at the end of the Add initialize, no Update event processed after that will know that the initialize even happened.
For this particular case, we have to treat Add & Update events in the same way because Add events can be missed (for various reasons). The only thing that matters when either event is being processed is whether or not the taint exists. So we would add a node into the map on the first event (Add or Update) where we see that it needs to be initialized. This is fine, the update event should be a no-op if the node is initialized anyways.
Yes but the problem here is that an update event created before the initialization but only processed after the initialization is cared about, and it shouldn't be.
I think this is true for most controllers, but this controller is slightly different IMO. We don't care about subsequent updates (unless it's an update from the cloud-controller-manager itself) since any relevant information pertaining to the cloud provider will get overwritten anyways. The provider is also very unlikely to change any information about the node if you asked it again. Even after accounting for that, the controller always gets the before starting the initialization logic to avoid this exact scenario.
In other words, the update event handler is only there to trigger the initialization process but the initialization logic will run independently of the node object passed into the event. Whether this is the correct behavior or not is likely for a different issue :)
Then at the very least we should add another one of these checks after the latest version of the node object is retrieved. Otherwise the whole initialization is being done again on nodes that do not need it. The decision of whether or not to do the initialization at all on receiving an Update event is being done based on the properties of the node object passed in to the event. The latest version of the node object is not retrieved until after the decision has already been made.
If we record what nodes are currently being initialized, we can prevent this IMO. Yeah, the taint determines whether or not to trigger the initialization, but the logic is independent of that & is also idempotent.
Storing the name in the map of nodes being initialized "currently" won't work. In the logs in my previous comment the node was initialized both at and . There's nothing current about that, that's 11 minutes apart. The record in the map needs to persist after the initialization is complete. As long as the events are processed in order of creation, it would be possible to add some lines to my change like this, in UpdateCloudNode on line 207 That would ensure that the node is only removed from the map once it is guaranteed that the next Update events will be discarded based on the nil cloud taint.
^ hard to gauge what this really means. Both initialization loops could be starting seconds away but finishing 11 minutes apart for various reasons (most likely a call to the provider is hanging?). Can you also log the timestamp at the start of ? As I mentioned in a previous comment, I think both approaches would work but semantically it would be cleaner to remove the node in the map as part of . ^ doing this seems a bit awkward IMO because it's not exactly clear why the map is being updated here and why we are removing a node from a map called if it was just initialized. We're also trying to delete an item in the map on the most common code path. You can achieve similar with: Yes, there can be a small window of time where we receive an update event and the cloud taint still exists but given is already idempotent & fetches the latest node object I think the tradeoff is worth it. Let's continue this discussion in the PR though, I think we can articulate these discussions better when we see the code :)
There are no initializeNode calls that take more than a few seconds. I the time taken from the start of initializeNode to the end of initializeNode to its log line to check this. The raw logs (grepped for this line) are here The red line in the second plot shows the time at which the last new name was observed. So the cluster resize was fully complete with all nodes initialized after 12 minutes, but there were still events being processed almost at 1 hour after starting. I split the distribution of durations for initializeNode into before and after that time when all nodes are properly initialized, but there doesn't seem to be any difference. Most calls take ~1 second, but there are up to 5 of them per second that need to be processed. ! I'll make the PR with what I have now and we can discuss further there.

Discussion:
return err } <ins> cloudTaint := getCloudTaint(curNode.Spec.Taints) if cloudTaint == nil { // Node object received from event had the cloud taint but was outdated, // the node has actually already been initialized. return nil } </ins> if curNode.Spec.ProviderID == "" { providerID, err := cloudprovider.GetInstanceProviderID(context.TODO(), cnc.cloud, types.NodeName(curNode.Name)) if err == nil {

// After adding, call UpdateNodeAddress to set the CloudProvider provided IPAddresses // So that users do not see any significant delay in IP addresses being filled into the node cnc.updateNodeAddress(curNode, instances) <ins> klog.Infof("Successfully initialized node %s with cloud provider", node.Name) </ins> return nil }) if err != nil { utilruntime.HandleError(err) return } <del> klog.Infof("Successfully initialized node %s with cloud provider", node.Name) </del> } func getCloudTaint(taints []v1.Taint) *v1.Taint {