Change:
I am reproducing this in OpenShift using the OpenStack cloud provider, but nothing here is specific to either OpenShift or OpenStack. When migrating a cluster which uses in-tree cloud provider to use external cloud-provider, we use the following process: cloud controllers in KCM CCM kubelets with --cloud-provider=external Between steps 1&2 the node-lifecycle, routes, and service controllers are not running anywhere, but this situation isn't unsafe for the cluster and we can keep it short. Between steps 2&3 is more interesting. The node controller is running in CCM, and the kubelets are running with --cloud-provider=openstack (or whatever, just not external). The main loop of node-controller will ignore a newly created node because . However, the loop does not have a similar exclusion. . Note that we shortcut if the kubelet is using an external cloud provider, but not if we don't. So between steps 2&3 both kubelet and CCM's node controller are managing Node.Addresses. If the legacy cloud provider and external CCM don't return the same list of NodeAddresses this causes flapping. Running OpenShift on OpenStack we have 2 problems. Firstly, our legacy cloud provider and external CCM produce different lists. We don't think this would be a problem in practise, except for the second problem. Kubelet adds the annotation to the node containing the node IP. This allows the CCM node controller to apply the same filtering based on node IP as kubelet does for the legacy cloud provider. This means that, although the lists are different, the node IP is still listed first which is the only thing that matters in practise. The problem is that kubelet only adds the annotation when --cloud-provider=external which, between steps 2&3, it is not. This means that Node.Addresses flaps between a correct and a broken configuration, as the NodeAddresses returned by the external CCM cannot be filtered by node IP without the annotation. The ideal solution here would be to ensure that only 1 controller is managing Node.Addresses during this period in the same way we do for Node creation with the external CCM taint. This would fix the issue entirely. A fix with a much smaller scope would be to have kubelet add the node IP annotation unconditionally so, although Node.Address can potentially flap, it will at least be between 2 valid configurations. Given that this should only ever be a problem during the upgrade window, the smaller fix may be more expedient. Node.Addresses should remain valid at all times when kubelet is configured with --cloud-provider=openstack but the CCM is also running. Good luck No response 24 OpenStack (but as described, not OpenStack-specific) No response No response No response No response
/sig cloud-provider
we've an agenda item for next week's (11 May) SIG cloud provider meeting about this topic
/triage accepted
Would it work to swap steps 2 and 3?
Good question. I don't think so. That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean: No cluster scale-up or scale-down No creation, deletion, or modification to LB Services I think we could handle this in OpenShift, although it's a bit fragile and I suspect updating all the kubelets can take a really long time on a big cluster. However, I think this would break cluster-api or any deployment tooling which rolls out updates by creating new machines. I think this would never complete: the new nodes would never come up because the node controller isn't running.
I am still trying to understand the issues here while reviewing your pr: , but I agreed with your comments above: That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean:

Discussion:
klog.V(4).InfoS("Using secondary node IP", "IP", secondaryNodeIP.String()) } <del> if externalCloudProvider { </del> <ins> if externalCloudProvider || cloud != nil { // Annotate the Node object with nodeIP for external cloud provider. // // We do this even when external CCM is not configured to cover a situation // during migration from legacy to external CCM: when CCM is running the // node controller in the cluster but kubelet is still running the in-tree // provider. Adding this annotation in all cases ensures that while // Addresses flap between the competing controllers, they at least flap // consistently. // // We do not add the annotation in the case where there is no cloud // controller at all, as we don't expect to migrate these clusters to use an // external CCM. </ins> if nodeIPSpecified { if node.ObjectMeta.Annotations == nil { node.ObjectMeta.Annotations = make(map[string]string) } node.ObjectMeta.Annotations[cloudproviderapi.AnnotationAlphaProvidedIPAddr] = nodeIP.String() } <ins> } if externalCloudProvider { </ins> // If --cloud-provider=external and node address is already set, // then we return early because provider set addresses should take precedence. // Otherwise, we try to look up the node IP and let the cloud provider override it later

// TODO(mtaufen): below is ported from the old kubelet_node_status_test.go code, potentially add more test coverage for NodeAddress setter in future func TestNodeAddress(t *testing.T) { <ins> type cloudProviderType int const ( cloudProviderLegacy cloudProviderType = iota cloudProviderExternal cloudProviderNone ) </ins> cases := []struct { <del> name                  string hostnameOverride      bool nodeIP                net.IP externalCloudProvider bool nodeAddresses         []v1.NodeAddress expectedAddresses     []v1.NodeAddress shouldError           bool </del> <ins> name                string hostnameOverride    bool nodeIP              net.IP cloudProviderType   cloudProviderType nodeAddresses       []v1.NodeAddress expectedAddresses   []v1.NodeAddress expectedAnnotations map[string]string shouldError         bool </ins> }{ { name:   "A single InternalIP",

shouldError:      false, }, { <del> name:                  "cloud provider is external", nodeIP:                netutils.ParseIPSloppy("10.0.0.1"), nodeAddresses:         []v1.NodeAddress{}, externalCloudProvider: true, </del> <ins> name:              "cloud provider is external", nodeIP:            netutils.ParseIPSloppy("10.0.0.1"), nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, </ins> expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname},

}, shouldError: false, }, <ins> { name:              "Legacy cloud provider gets nodeIP annotation", nodeIP:            netutils.ParseIPSloppy("10.1.1.1"), cloudProviderType: cloudProviderLegacy, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1", }, shouldError: false, }, { name:              "External cloud provider gets nodeIP annotation", nodeIP:            netutils.ParseIPSloppy("10.1.1.1"), cloudProviderType: cloudProviderExternal, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1", }, shouldError: false, }, { name:              "No cloud provider does not get nodeIP annotation", nodeIP:            netutils.ParseIPSloppy("10.1.1.1"), cloudProviderType: cloudProviderNone, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAnnotations: map[string]string{}, shouldError:         false, }, </ins> } for _, testCase := range cases { t.Run(testCase.name, func(t *testing.T) {

return testCase.nodeAddresses, nil } <del> // cloud provider is expected to be nil if external provider is set </del> <ins> // cloud provider is expected to be nil if external provider is set or there is no cloud provider </ins> var cloud cloudprovider.Interface <del> if testCase.externalCloudProvider { cloud = nil } else { </del> <ins> if testCase.cloudProviderType == cloudProviderLegacy { </ins> cloud = &fakecloud.Cloud{ Addresses: testCase.nodeAddresses, Err:       nil, } } // construct setter

nodeIPValidator, hostname, testCase.hostnameOverride, <del> testCase.externalCloudProvider, </del> <ins> testCase.cloudProviderType == cloudProviderExternal, </ins> cloud, nodeAddressesFunc)

assert.True(t, apiequality.Semantic.DeepEqual(testCase.expectedAddresses, existingNode.Status.Addresses), "Diff: %s", diff.ObjectDiff(testCase.expectedAddresses, existingNode.Status.Addresses)) <ins> if testCase.expectedAnnotations != nil { assert.True(t, apiequality.Semantic.DeepEqual(testCase.expectedAnnotations, existingNode.Annotations), "Diff: %s", diff.ObjectDiff(testCase.expectedAnnotations, existingNode.Annotations)) } </ins> }) } }