Repository: metallb/metallb

Issue:
Updating status - Reconciler error - Value is immutable
### MetalLB Version

0.14.8

### Deployment method

Charts

### Main CNI

calico

### Kubernetes Version

v1.31.2+k3s1

### Cluster Distribution

k3s

### Describe the bug

I am getting warning messages regarding being unable to update status. I reported to slack first, and was requested to open an upstream issue here so I am. https://kubernetes.slack.com/archives/C8FSTGU56/p1733390261743249

The error I see on three my deployments is this:

```
{"level":"error","ts":"2024-12-05T08:44:57Z","msg":"Reconciler error","controller":"servicel2status","namespace":"kopia","name":"kopia-b2","reconcileID":"ea81fe75-b0fe-4b97-bb70-09096bed9319","error":"ServiceL2Status.metallb.io \"l2-7br4p\" is invalid: status.node: Invalid value: \"string\": Value is immutable","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
```

I see these approximately every 5 to 10 minutes.

### To Reproduce

Apply the following service:

```
---
apiVersion: v1
kind: Service
metadata:
  name: kopia-ssd
  namespace: kopia
  annotations:
    metallb.universe.tf/loadBalancerIPs: 10.1.11.54
spec:
  ports:
  - port: 51515
    targetPort: 51515
    name: grpc
    protocol: TCP
  selector:
    app: kopia-ssd
  type: LoadBalancer
```

### Expected Behavior

No warnings or errors.

### Additional Context

Nothing else really, happy to test / update as needed for debugging.

### I've read and agree with the following

- [X] I've checked all open and closed issues and my request is not there.
- [X] I've checked all open and closed pull requests and my request is not there.

### I've read and agree with the following

- [X] I've checked all open and closed issues and my issue is not there.
- [X] This bug is reproducible when deploying MetalLB from the main branch
- [X] I have read the [troubleshooting guide](https://metallb.universe.tf/troubleshooting/) and I am still not able to make it work
- [X] I checked the logs and MetalLB is not discarding the configuration as not valid
- [X] I enabled the debug logs, collected the information required from the cluster using the [collect script](https://raw.githubusercontent.com/metallb/metallb/main/troubleshooting/collect.sh) and will attach them to the issue
- [X] I will provide the definition of my service and the related endpoint slices and attach them to this issue

Hints from issue comments:
@danielgoepp: [metallb_report.tgz](https://github.com/user-attachments/files/18021014/metallb_report.tgz)
@markmaker: Same here.@oribon: thanks for submitting this! for some reason the pod logs weren't collected properly, in any case I opened a PR that handles some problems around duplicate ServiceL2Status resources, hopefully solving this.@danielgoepp: @oribon - I just checked my logs - this is still happening. It doesn't cause me any issues, but in case you thought it was fixed, it doesn't appear to be.

```
{"level":"error","ts":"2025-08-01T08:21:45Z","msg":"Reconciler error","controller":"servicel2status","namespace":"kopia","name":"kopia-ssd","reconcileID":"8435da5d-523b-4429-8eba-482f7a114ccf","error":"ServiceL2Status.metallb.io \"l2-fr24x\" is invalid: status.node: Invalid value: \"string\": Value is immutable","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
```
@oribon: @danielgoepp thanks 😅 I can think of another scenario that is not covered in the code - the leader for the service bounces and the new node attempts to modify the resource belonging to the former node (before the former node deletes its state). as you said it is not harmful (the state would eventually converge), but does it seem that this error is logged less frequently than the last time? can you provide the logs please?
@danielgoepp: Interestingly, it is just one of my deployments now that it is logging for now, kopia, my backups services. I don't know if this helps or not. Seems to be at about the same interval. Let me know if you need me to run debug or anything.

[All-Messages-search-result.csv](https://github.com/user-attachments/files/21564867/All-Messages-search-result.csv)

Solution patch:
diff --git a/internal/k8s/controllers/layer2_status_controller.go b/internal/k8s/controllers/layer2_status_controller.go
index 1ef426f2..1a53b7d3 100644
--- a/internal/k8s/controllers/layer2_status_controller.go
+++ b/internal/k8s/controllers/layer2_status_controller.go
@@ -14,7 +14,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
-	utilErrors "k8s.io/apimachinery/pkg/util/errors"
+	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
@@ -81,17 +81,31 @@ func (r *Layer2StatusReconciler) Reconcile(ctx context.Context, req ctrl.Request
 	var errs []error
 	if len(ipAdvS) == 0 {
 		for key, item := range serviceL2statuses.Items {
-			if item.Status.Node != r.NodeName {
+			if item.Labels[LabelAnnounceNode] != r.NodeName {
 				continue
 			}
 			if err := r.Client.Delete(ctx, &serviceL2statuses.Items[key]); err != nil && !errors.IsNotFound(err) {
 				errs = append(errs, err)
 			}
 		}
-		if len(errs) > 0 {
-			return ctrl.Result{}, utilErrors.NewAggregate(errs)
+		return ctrl.Result{}, utilerrors.NewAggregate(errs)
+	}
+
+	deleteRedundantErrs := []error{}
+	if len(serviceL2statuses.Items) > 1 {
+		// We shouldn't get here, just in case the controller created redundant resources
+		for i := range serviceL2statuses.Items[1:] {
+			if serviceL2statuses.Items[i+1].Labels[LabelAnnounceNode] != r.NodeName {
+				continue
+			}
+			if err := r.Client.Delete(ctx, &serviceL2statuses.Items[i+1]); err != nil && !errors.IsNotFound(err) {
+				deleteRedundantErrs = append(deleteRedundantErrs, err)
+			}
 		}
-		return ctrl.Result{}, nil
+	}
+
+	if len(deleteRedundantErrs) > 0 {
+		return ctrl.Result{}, utilerrors.NewAggregate(deleteRedundantErrs)
 	}
 
 	// creating a brand new cr
@@ -131,8 +145,10 @@ func (r *Layer2StatusReconciler) Reconcile(ctx context.Context, req ctrl.Request
 	}
 	if result == controllerutil.OperationResultCreated {
 		// According to https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#CreateOrPatch
-		// If the object is created for the first time, we have to requeue it to ensure that the status is updated.
-		return ctrl.Result{Requeue: true}, nil
+		// If the object is created, we have to patch it again to ensure the status is created.
+		// This will happen when we reconcile the creation event.
+		level.Debug(r.Logger).Log("controller", "Layer2StatusReconciler", "created state", dumpResource(state))
+		return ctrl.Result{}, nil
 	}
 
 	level.Debug(r.Logger).Log("controller", "Layer2StatusReconciler", "updated state", dumpResource(state))
diff --git a/troubleshooting/collect.sh b/troubleshooting/collect.sh
index 7c494e57..ce05ae40 100755
--- a/troubleshooting/collect.sh
+++ b/troubleshooting/collect.sh
@@ -9,7 +9,8 @@ METALLB_NS="${METALLB_NS:-metallb-system}"
 OUTPUT="./report"
 
 function get_metallb_crs() {
-    declare -a METALLB_CRDS=("bgppeers" "bfdprofiles" "bgpAdvertisements" "ipaddresspools" "l2advertisements" "communities")
+    declare -a METALLB_CRDS=("bgppeers" "bfdprofiles" "bgpAdvertisements" "ipaddresspools" 
+    "l2advertisements" "communities" "servicel2statuses" "servicebgpstatuses" "frrconfigurations" "frrnodestates")
     mkdir -p ${OUTPUT}/crds/
 
     for CRD in "${METALLB_CRDS[@]}"; do



Test patch:
diff --git a/e2etest/pkg/status/l2status.go b/e2etest/pkg/status/l2status.go
index 81ca9ce5..7a666d37 100644
--- a/e2etest/pkg/status/l2status.go
+++ b/e2etest/pkg/status/l2status.go
@@ -34,5 +34,11 @@ func L2ForService(cs client.Client, svc *v1.Service) (*v1beta1.ServiceL2Status,
 	if len(statuses) > 1 {
 		return nil, fmt.Errorf("got more than 1 serviceL2Status object: %d", len(statuses))
 	}
-	return &(statuses[0]), nil
+
+	s := statusList.Items[0]
+	if s.Status.Node == "" {
+		return nil, fmt.Errorf(".Status.Node not populated for service %s/%s, resource: %v", svc.Name, svc.Namespace, s)
+	}
+
+	return &s, nil
 }