Change:
I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this. If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).

Discussion:
if err != nil { return err } <ins> matches, err = s.PopulateIPs(matches) if err != nil { return err } </ins> nodes, err := s.kubeClient.Nodes().List() if err != nil { return err } nodeMap := make(map[string]*api.Node) <del> for _, node := range nodes.Items { </del> <ins> for i := range nodes.Items { node := nodes.Items[i] </ins> nodeMap[node.Name] = &node }

fakeCloud            *fake_cloud.FakeCloud matchRE              string expectedRequestCount int <del> expectedCreated      []string </del> <ins> expectedNameCreated  []string expectedExtIDCreated []string </ins> expectedDeleted      []string }{ {

}, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", }, </ins> }, matchRE:              ".*", expectedRequestCount: 1, // List <del> expectedCreated:      []string{}, </del> <ins> expectedNameCreated:  []string{}, expectedExtIDCreated: []string{}, </ins> expectedDeleted:      []string{}, }, {

}, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0", "node1"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", }, </ins> }, matchRE:              ".*", expectedRequestCount: 2, // List + Create <del> expectedCreated:      []string{"node1"}, </del> <ins> expectedNameCreated:  []string{"node1"}, expectedExtIDCreated: []string{"ext-node1"}, </ins> expectedDeleted:      []string{}, }, {

}, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", }, </ins> }, matchRE:              ".*", expectedRequestCount: 2, // List + Delete <del> expectedCreated:      []string{}, </del> <ins> expectedNameCreated:  []string{}, expectedExtIDCreated: []string{}, </ins> expectedDeleted:      []string{"node1"}, }, {

}, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0", "node1", "fake"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", "fake":  "ext-fake", }, </ins> }, matchRE:              "node[0-9]+", expectedRequestCount: 2, // List + Create <del> expectedCreated:      []string{"node1"}, </del> <ins> expectedNameCreated:  []string{"node1"}, expectedExtIDCreated: []string{"ext-node1"}, </ins> expectedDeleted:      []string{}, }, }

t.Errorf("expected %v call, but got %v.", item.expectedRequestCount, item.fakeNodeHandler.RequestCount) } nodes := sortedNodeNames(item.fakeNodeHandler.CreatedNodes) <del> if !reflect.DeepEqual(item.expectedCreated, nodes) { t.Errorf("expected node list %+v, got %+v", item.expectedCreated, nodes) </del> <ins> if !reflect.DeepEqual(item.expectedNameCreated, nodes) { t.Errorf("expected node list %+v, got %+v", item.expectedNameCreated, nodes) } nodeExtIDs := sortedNodeExternalIDs(item.fakeNodeHandler.CreatedNodes) if !reflect.DeepEqual(item.expectedExtIDCreated, nodeExtIDs) { t.Errorf("expected node external id list %+v, got %+v", item.expectedExtIDCreated, nodeExtIDs) </ins> } nodes = sortedNodeNames(item.fakeNodeHandler.DeletedNodes) if !reflect.DeepEqual(item.expectedDeleted, nodes) {

return nodeNames } <ins> func sortedNodeExternalIDs(nodes []*api.Node) []string { nodeExternalIDs := []string{} for _, node := range nodes { nodeExternalIDs = append(nodeExternalIDs, node.Spec.ExternalID) } sort.Strings(nodeExternalIDs) return nodeExternalIDs } </ins> func contains(node *api.Node, nodes []*api.Node) bool { for i := 0; i < len(nodes); i++ { if node.Name == nodes[i].Name {

Err           error Calls         []string IP            net.IP <del> ExtID         string </del> <ins> ExtID         map[string]string </ins> Machines      []string NodeResources *api.NodeResources ClusterList   []string

// ExternalID is a test-spy implementation of Instances.ExternalID. // It adds an entry "external-id" into the internal method call record. <ins> // It returns an external id to the mapped instance name, if not found, it will return "ext-{instance}" </ins> func (f *FakeCloud) ExternalID(instance string) (string, error) { f.addCall("external-id") <del> return f.ExtID, f.Err </del> <ins> return f.ExtID[instance], f.Err </ins> } // List is a test-spy implementation of Instances.List.