Repository: tailscale/tailscale (Go)

Issue:
Kubernetes operator produces insanely long status.loadBalancer.ingress list
### What is the issue?

When using a basic LoadBalancer service with the Kubernetes operator, the service has a rather insance number of items added to the `status.loadBalancer.ingress` list. `2608` entries when I grabbed it, but judging by the resource usage of my kube-apiserver I'm betting it just keeps going if I leave it long enough.

I dug into this when my control plane nodes started to fall over and OOM kube-apiserver because of constant requests.

For the time being, I've scaled the operator itself down to zero replicas unless I need to change the service itself.

If I was to guess, it's around this logic here https://github.com/tailscale/tailscale/blob/a1dcf12b671e8668b1bd3eedc7cfcb4381b9d29c/cmd/k8s-operator/svc.go#L351-L370 - the list is appended to on every reconcilliation loop, but it's never *cleared*, so grows indefinetly.

### Steps to reproduce

Kubernetes version:
```
> kubectl version                                                           [admin@homelab|default]
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.34.1
```

Image:

```
docker.io/tailscale/tailscale@sha256:a9a4d20b6eb020e17ea62e872719624bcd0f56d44701b5f2675cac22f
d29884d
```

Kubernetes service:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: tailscale-contour-envoy
  namespace: contour
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: envoy
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerClass: tailscale
  ipFamilyPolicy: PreferDualStack
```

### Are there any recent changes that introduced the issue?

_No response_

### OS

Linux

### OS version

Talos Linux v1.11.1

### Tailscale version

1.87.158

### Other software

_No response_

### Bug report

_No response_
k8s-operator reconciler fight
### What is the issue?

I was troubleshooting some severe stability issues on my homelab cluster that began a few days ago. The issue began when I rebooted the node that was running the unstable k8s operator (deployed via the static YAML), causing it to pull the latest unstable image (1.87.134, I believe). Eventually I narrowed the cause of the stability problems down to k8s-operator (unstable) sending an excessive and increasing amount of traffic to etcd, which was eventually exhausting the etcd server memory. Once I deployed k8s-operator stable, the cluster rapidly stabilised. 

Chart of client traffic inbound to etcd, by server, with annotations:

<img width="1013" height="463" alt="Image" src="https://github.com/user-attachments/assets/2c602082-339f-4681-808c-f10c161e4d13" />

Today I ran unstable for a few minutes again (with debug logs enabled) to replicate. Since I switched to Helm to recover, I tested with the unstable Helm charts. It is replicable on my cluster with versions 1.87.134 and 1.87.158, but not replicated with 1.87.85.

Below is a sample of the debug logs on the broken versions, which is enough to demonstrate a pattern that did not occur on the working version. It appears that immediately after "service-reconciler" reconciles a service, "service-pg-reconciler" considers the same service "is being deleted or is (no longer) referring to Tailscale ingress/egress, ensuring any created resources are cleaned up".

```json
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"secret tailscale/ts-atwarrior-tailscale-frontend-cmg6d-0 already exists","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"Hostname\":\"default-atwarrior-tailscale-frontend\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-atwarrior-tailscale-frontend-cmg6d","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"setting Service LoadBalancer status to \"atwarrior.orca-dragon.ts.net\", 100.124.14.121, fd7a:115c:a1e0::c801:e79","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"default","service-name":"atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"starting reconcile","Service":"default/atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"Service is being deleted or is (no longer) referring to Tailscale ingress/egress, ensuring any created resources are cleaned up","Service":"default/atwarrior-tailscale-frontend","hostname":"default-atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"Ensuring any resources for Service are cleaned up","Service":"default/atwarrior-tailscale-frontend","hostname":"default-atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"no finalizer, nothing to do","Service":"default/atwarrior-tailscale-frontend","hostname":"default-atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"reconcile finished","Service":"default/atwarrior-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"secret tailscale/ts-forgejo-tailscale-frontend-s7bnl-0 already exists","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"Hostname\":\"default-forgejo-tailscale-frontend\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-forgejo-tailscale-frontend-s7bnl","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"setting Service LoadBalancer status to \"gitea.orca-dragon.ts.net\", 100.64.233.49, fd7a:115c:a1e0::b201:e936","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"default","service-name":"forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"default","service-name":"pihole-0"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"default","service-name":"pihole-0"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"starting reconcile","Service":"default/forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"Service is being deleted or is (no longer) referring to Tailscale ingress/egress, ensuring any created resources are cleaned up","Service":"default/forgejo-tailscale-frontend","hostname":"default-forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"Ensuring any resources for Service are cleaned up","Service":"default/forgejo-tailscale-frontend","hostname":"default-forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"no finalizer, nothing to do","Service":"default/forgejo-tailscale-frontend","hostname":"default-forgejo-tailscale-frontend"}
{"level":"debug","ts":"2025-09-12T07:04:16Z","logger":"service-pg-reconciler","msg":"reconcile finished","Service":"default/forgejo-tailscale-frontend"}
```

I could attach more logs, but it's really just pages and pages of that.

### Steps to reproduce

Create 10 or so Services, each morally equivalent to the following example:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: prom-tailscale-frontend
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  selector:
    prometheus: prom
  ports:
    - name: web
      port: 80
      targetPort: web
```

Deploy unstable k8s-operator version 1.87.134 or 1.87.158 using Helm or the static manifest, e.g.

```shell
helm repo add tailscale-unstable https://pkgs.tailscale.com/unstable/helmcharts
helm repo update
helm upgrade \
  --install \
  tailscale-operator \
  tailscale-unstable/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId="[REDACTED]" \
  --set-string oauth.clientSecret="[REDACTED]" \
  --set operatorConfig.logging=debug \
  --version 1.87.134 \
  --wait
```

### Are there any recent changes that introduced the issue?

_No response_

### OS

Linux

### OS version

Debian 12 "Bookworm" and 13 "Trixie"

### Tailscale version

Unstable

### Other software

k3s 1.33.4+k3s1 in HA mode (etcd) with three servers and three other nodes


### Bug report

_No response_


Discussion:
Also affected by this. This is causing issues on our API server side. Also on Talos linux.
Hey @gmemstr, thanks for raising this! I've confirmed the problem and raised #17120 to fix it. The fix was as you described. Previously we did reset that slice but from a recent refactor it was missed, leading to a never ending reconciliation loop updating the status.

We're due a release of the tailscale container early next week and I'll look to have this included.

Hey @DrJosh9000, thanks for bringing this from the discord to here. This appears to be the same problem reported in #17105, which has a fix that will go in today #17120 



Patch:
diff --git a/cmd/k8s-operator/svc.go b/cmd/k8s-operator/svc.go
index 51ad1aea3c808..eec1924e7902c 100644
--- a/cmd/k8s-operator/svc.go
+++ b/cmd/k8s-operator/svc.go
@@ -348,9 +348,10 @@ func (a *ServiceReconciler) maybeProvision(ctx context.Context, logger *zap.Suga
 
 	dev := devices[0]
 	logger.Debugf("setting Service LoadBalancer status to %q, %s", dev.hostname, strings.Join(dev.ips, ", "))
-	svc.Status.LoadBalancer.Ingress = append(svc.Status.LoadBalancer.Ingress, corev1.LoadBalancerIngress{
-		Hostname: dev.hostname,
-	})
+
+	ingress := []corev1.LoadBalancerIngress{
+		{Hostname: dev.hostname},
+	}
 
 	clusterIPAddr, err := netip.ParseAddr(svc.Spec.ClusterIP)
 	if err != nil {
@@ -365,10 +366,11 @@ func (a *ServiceReconciler) maybeProvision(ctx context.Context, logger *zap.Suga
 			continue
 		}
 		if addr.Is4() == clusterIPAddr.Is4() { // only add addresses of the same family
-			svc.Status.LoadBalancer.Ingress = append(svc.Status.LoadBalancer.Ingress, corev1.LoadBalancerIngress{IP: ip})
+			ingress = append(ingress, corev1.LoadBalancerIngress{IP: ip})
 		}
 	}
 
+	svc.Status.LoadBalancer.Ingress = ingress
 	tsoperator.SetServiceCondition(svc, tsapi.ProxyReady, metav1.ConditionTrue, reasonProxyCreated, reasonProxyCreated, a.clock, logger)
 	return nil
 }


Test patch:
diff --git a/cmd/k8s-operator/operator_test.go b/cmd/k8s-operator/operator_test.go
index 50f8738cefc39..5af237342e8cd 100644
--- a/cmd/k8s-operator/operator_test.go
+++ b/cmd/k8s-operator/operator_test.go
@@ -173,6 +173,10 @@ func TestLoadBalancerClass(t *testing.T) {
 			},
 		},
 	}
+
+	// Perform an additional reconciliation loop here to ensure resources don't change through side effects. Mainly
+	// to prevent infinite reconciliation
+	expectReconciled(t, sr, "default", "test")
 	expectEqual(t, fc, want)
 
 	// Turn the service back into a ClusterIP service, which should make the