Change:
// The comparison loops are O(N^2), but we don't expect N to be huge // (there's a hard-limit at 2^16, because they're ports; and even 4 ports would be a lot) for _, oldNodePort := range oldNodePorts { <del> if !contains(newNodePorts, oldNodePort) { </del> <ins> if contains(newNodePorts, oldNodePort) { </ins> continue } nodePortOp.ReleaseDeferred(oldNodePort)

Discussion:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): K8S 1.5.3, and kubectl 1.5.3 Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): xenial Kernel (e.g. ): latest xenial Install tools: Others: ACS-Engine cluster, 3 masters What happened: Nodeport is assigned to dashboard and to the nginx service that was started within a minute of starting the dashboard. The dashboard is being started as part of the addon manager. What you expected to happen: Each service should receive a separate node port. How to reproduce it (as minimally and precisely as possible): Run 3 master setup, with 3 API servers, 3 addon managers, 3 controllers (but one controller is elected leader). Run dashboard as an add-on. Create service shortly after dashboard is started an assigned nodeport. We run a Jenkin's job, and this repros about 30-50% of the time. Anything else we need to know: We instrumented the build and found the following: is initially created by the addon manager by issuing a POST to/api/v1/namespaces/kube-system/services. This is where port is , and snapshotted to etcd Immediately after the POST, another addon manager issues PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard.

The post calls a release to , and this is snapshotted to etcd. API server receives the request for nginx and assigns to nginx service. all api servers start reporting message the node port for service nginx/namespace- was assigned to multiple services. Workaround: delete the nginx service, re-create the service. have you seen this before? This build has your latest changelist around , but from best we can observe is unrelated to the changes.
The attached files are the API Server Logs: Here is a timeline of events: 16:07:02.862 - - Start POST /api/v1/namespaces/kube-system/services 16:07:02.863 - - [Create() 16:07:03.173 - - allocated - in memory 16:07:03.255 - - Create] 16:07:03.107 - - bitmap.Snapshot (177), 16:07:03.256 - - Finish POST /api/v1/namespaces/kube-system/services 16:07:08.474 - - start patch k8s-dashboard 16:07:08.832 - - released from allocator bitmap, and snapshotted to etcd
because type=NodePort ? Why is this releasing the port? On purpose or bug? Ports are allocated randomly from 2767 values, and you're reliably reproducing this collision? Why did they not get the update to the service that released it? Nope. I'll try to repro.
type=NodePort ? Correct, the dashboard addon specifies is this releasing the port? On purpose or bug? Discussed with and we figured it is because the addon manager does not see dynamically created , in the addon yaml (because it is dynamically assigned). This points at a potential bug in the add-on manager that it should ignore the dynamically assigned values, ignoring the need for the patch. are allocated randomly from 2767 values, and you're reliably reproducing this collision? We are running Jenkins, and this repro's 30-50% of the time in the multi-master case. The times it does not repro is when the same API server handles both the PUT of the dashboard and the nginx services, because AllocateNext() chooses the next random value for Nginx. did they not get the update to the service that released it?

It is being released from the allocator bitmap in the registry, but not the service definition. My guess is that it is successfully being removed from the allocator bitmap, but then when is attempted to be removed from the service it fails because the service is of type NodePort. I need to instrument to confirm this. If once confirmed, suggest I look to see if the removal from service can happen first, before removing from the allocator bitmap (nodeport should be gone from service definition, before being removed from allocator registry). That way if removal from service fails, then allocator bitmap still keeps other services from locking it.
for 3) it is the same across multiple machines, I assume it's because the random number generator has the same constant seed. for 4) my guess is that on Service update we're doing the following: M1: Update to service-1 comes in without NodePort (this is an update to an existing service) M1: Allocator releases the port , writes to etcd. M1: Decides to re-allocate to the service-1, but hasn't yet written to etcd M2: New service service-2 is created without NodePort M2: Allocator also decides to allocate to service-2, writes it to etcd. M1: Allocator fails to update etcd At this point, both Service-1 and Service-2 have the same NodePort assigned to them.
And a clarification on the Addon bits: Addon manager has a service that looks like: But when there is a for it, it has in the response: so add-on manager sees a difference, and posts back something missing the which clears and resets and starts this cycle.
OK, I think I have found 1 bug in this path, but I can't convince myself it would cause this failure mode, still working on it.
The addon manager (the one we support, anyway - did you write your own?) uses , so this should be fine. The nodePort is the same as clusterIP in this regard. You can see it by repeatedly running on this and seeing the nodePort not change.
The dashboard service object does not lose port , it is only removed from etcd path /registry/ranges/servicenodeports. Also, I see the same thing using , in that the release is happening, but when I view the dashboard service object, it still shows holding .

Here's the instrumented logs: 21:50:20.7778 ] BEGIN PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard 21:50:20.8228 ] [bitmap.Release(177), 21:50:20.8228 ] bitmap.Release(177)] 21:50:20.8381 ] [bitmap.Snapshot, 21:50:20.8381 ] bitmap.Snapshot] 21:50:20.8389 ] END PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard
Can you share this patch?
ohh, maybe the same bug is triggering this! God this code is twisty. Are you game to try a patch? in , around line 435, look for change the to - just remove the , then rebuild and re-run? Tim
Yeah, its the standard addon manager. We see it continuously doing a to the API Server. (has moved to the server?)
apply generates a PATCH , Brendan Burns <:
On reflection all afternoon, I am pretty confident this is the bug. I'll send a PR, sans test probably, shortly , Tim Hockin <: We see it continuously doing a PATCH to the API Server. (has apply moved to the server?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
, Tim Hockin <:
Thanks I'm trying the patch and testing now.
Jenkins is now passing 100% on 3 master HA. Also, logs no longer show the removal of the dashboard nodeport from registry. I confirmed this for kubectl apply as well on the dashboard file. Thanks for the fix!
Thanks for the challenging bug. It reminded me how much I hate this code :) , anhowe <: