Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. The calls to create the load balancer were failing with a 400 What you expected to happen: controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was. How to reproduce it (as minimally and precisely as possible): Just created a cluster with an add on for the dashboard. Anything else we need to know?: logs from controller manager: I0423 12:50:28. ] Starting range CIDR allocator I0423 12:50:28. ] Waiting for caches to sync for cidrallocator controller I0423 12:50:28. ] Starting NoExecuteTaintManager I0423 12:50:28. ] Initializing eviction metric for zone: westeurope: :0 W0423 12:50:28. ] Missing timestamp for Node aks-nodepool1--0. Assuming now as a timestamp. I0423 12:50:28. ] Controller detected that zone westeurope: :0 is now in state Normal. I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"aks-nodepool1--0", UID:"-4476-11e8-9b1c-", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node aks-nodepool1--0 event: Registered Node aks-nodepool1--0 in Controller I0423 12:50:28. ] Caches are synced for ClusterRoleAggregator controller I0423 12:50:28. ] Caches are synced for endpoint controller I0423 12:50:28. ] Caches are synced for attach detach controller E0423 12:50:28. ] Error creating spec for volume "mssqldb", pod "feature-test3"/"mssqlserver--r5r7l": error processing PVC "feature-test3"/"mssql-data": PVC feature-test3/mssql-data has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("") I0423 12:50:28. ] Caches are synced for deployment controller I0423 12:50:28. ] Caches are synced for persistent volume controller I0423 12:50:28. ] Caches are synced for ReplicationController controller I0423 12:50:28. ] Caches are synced for TTL controller I0423 12:50:28. ] Caches are synced for cidrallocator controller E0423 12:50:28. ] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"-4476-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"193", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"feature-initialproject", Name:"identityserver-service", UID:"-4485-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"7879", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer I0423 12:50:28. ] Caches are synced for namespace controller I0423 12:50:28. ] Caches are synced for service account controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Caches are synced for stateful set controller I0423 12:50:28. ] Caches are synced for disruption controller I0423 12:50:28. ] Sending events to api server. I0423 12:50:29. ] selectLoadBalancer: cluster(kubernetes) service(feature-initialproject/identityserver-service) isInternal(true) - availabilitysetsnames [nodepool1-availabilitySet-] E0423 12:50:30. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:509 /usr/local/go/src/runtime/asmamd64.s:2337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d] goroutine 1338 [running]: (0x0, 0x0, 0x0) +0x111 panic(0x40b2e00, 0xa603590) +0x283 (Cloud).getServiceLoadBalancerStatus(0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1) +0x30d (Cloud).EnsureLoadBalancer(0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40) +0x2ea (ServiceController).ensureLoadBalancer(0xc420f56000, 0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6) +0xcc (ServiceController).createLoadBalancerIfNeeded(0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492) +0x20e (ServiceController).processServiceUpdate(0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0) +0xeb (ServiceController).syncService(0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0) +0x3aa (ServiceController).worker.func1(0xc420f56000) +0xd9 (ServiceController).worker(0xc420f56000) +0x2b (ServiceController).()-fm() +0x2a (0xc420a04570) +0x5e (0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0) +0xbd (0xc420a04570, 0x3b9aca00, 0xc4208fa0c0) +0x4d created by (ServiceController).Run +0x20c Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Azure, AKS OS (e.g. from /etc/os-release): Ubunutu Kernel (e.g. ): xenial Install tools:AKS Others:
not sure what sig should be labeled in on this. Seems like a bug in sig-azure and whoever owns the piece getting the nil pointer/panic
what's the k8s version?
1.9.1
- I suspect that the error is here. This can be either one of two things A bug in SDK serialization (LB does have a private ip) but it does not get serialized correctly. A bug in our assumption that that each internal LB ip fe config will have a valid internal IP. Can you dump the model for this internal LB here? Also -- did this error eventually recover? (it could be a race condition between setting ip and getting ip)
this was actually a customer cluster and they deleted it before I got a chance to get things like the model/what error code on the 400 was. And it didn't ever recover.
according to the evente table it was an internal loadbalancer here is the link to it --https://jarvis- maybe it the Details logged Can help you further
I was able to get more details from logs, {"error":{"code":"LoadBalancerFrontendIPConfigurationCountLimitReached","message":"Basic sku load balancer /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/loadBalancers/kubernetes-internal cannot have more than 10 FrontendIPConfigurations.","details":[]}}
Thanks - So there are two errors here. One the user can not use more than 10 IPs. The fact that we panic on this. i believe the fix is to log and fail the operation. /assign
The issue has been fixed in and has been included in v1.9.2. Could you upgrade your cluster (e.g. to latest v1.9.7) and check whether it fixes the issue?
I'm closing this issue since it's been resolved. /close
I am seeing this on 1.10.x and 1.11.x clusters: 10.6 11.2
The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that /reopen
Reopening this issue. <detailsIn response to : The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Added this to v1.12 since this is a critical urgent bug.

Discussion:
} } <del> glog.V(2).Infof("getServiceLoadBalancerStatus gets ingress IP %q from frontendIPConfiguration %q for service %q", *lbIP, lbFrontendIPConfigName, serviceName) return &v1.LoadBalancerStatus{Ingress: []v1.LoadBalancerIngress{{IP: *lbIP}}}, nil </del> <ins> glog.V(2).Infof("getServiceLoadBalancerStatus gets ingress IP %q from frontendIPConfiguration %q for service %q", to.String(lbIP), lbFrontendIPConfigName, serviceName) return &v1.LoadBalancerStatus{Ingress: []v1.LoadBalancerIngress{{IP: to.String(lbIP)}}}, nil </ins> } }

} func allowsConsolidation(rule network.SecurityRule) bool { <del> return strings.HasPrefix(*rule.Name, "shared") </del> <ins> return strings.HasPrefix(to.String(rule.Name), "shared") </ins> } func findConsolidationCandidate(rules []network.SecurityRule, rule network.SecurityRule) (int, bool) { for index, r := range rules { if allowsConsolidation(r) { <del> if strings.EqualFold(*r.Name, *rule.Name) { </del> <ins> if strings.EqualFold(to.String(r.Name), to.String(rule.Name)) { </ins> return index, true } }

func findProbe(probes []network.Probe, probe network.Probe) bool { for _, existingProbe := range probes { <del> if strings.EqualFold(*existingProbe.Name, *probe.Name) && *existingProbe.Port == *probe.Port { </del> <ins> if strings.EqualFold(to.String(existingProbe.Name), to.String(probe.Name)) && to.Int32(existingProbe.Port) == to.Int32(probe.Port) { </ins> return true } }

func findRule(rules []network.LoadBalancingRule, rule network.LoadBalancingRule) bool { for _, existingRule := range rules { <del> if strings.EqualFold(*existingRule.Name, *rule.Name) && </del> <ins> if strings.EqualFold(to.String(existingRule.Name), to.String(rule.Name)) && </ins> equalLoadBalancingRulePropertiesFormat(existingRule.LoadBalancingRulePropertiesFormat, rule.LoadBalancingRulePropertiesFormat) { return true }

// despite different DestinationAddressPrefixes, in order to give it a chance to consolidate the two rules. func findSecurityRule(rules []network.SecurityRule, rule network.SecurityRule) bool { for _, existingRule := range rules { <del> if !strings.EqualFold(*existingRule.Name, *rule.Name) { </del> <ins> if !strings.EqualFold(to.String(existingRule.Name), to.String(rule.Name)) { </ins> continue } if existingRule.Protocol != rule.Protocol { continue } <del> if !strings.EqualFold(*existingRule.SourcePortRange, *rule.SourcePortRange) { </del> <ins> if !strings.EqualFold(to.String(existingRule.SourcePortRange), to.String(rule.SourcePortRange)) { </ins> continue } <del> if !strings.EqualFold(*existingRule.DestinationPortRange, *rule.DestinationPortRange) { </del> <ins> if !strings.EqualFold(to.String(existingRule.DestinationPortRange), to.String(rule.DestinationPortRange)) { </ins> continue } <del> if !strings.EqualFold(*existingRule.SourceAddressPrefix, *rule.SourceAddressPrefix) { </del> <ins> if !strings.EqualFold(to.String(existingRule.SourceAddressPrefix), to.String(rule.SourceAddressPrefix)) { </ins> continue } if !allowsConsolidation(existingRule) && !allowsConsolidation(rule) { <del> if !strings.EqualFold(*existingRule.DestinationAddressPrefix, *rule.DestinationAddressPrefix) { </del> <ins> if !strings.EqualFold(to.String(existingRule.DestinationAddressPrefix), to.String(rule.DestinationAddressPrefix)) { </ins> continue } }