Repository: envoyproxy/gateway

Issue:
Race conditions between controller and gateway service
*Description*:

When deploying for the first time, the Envoy Gateway controller appears to experience a race condition when trying to reference the Envoy service before it's fully created. One has to restart the controller pod for it to resolve. 

*Repro steps*:

v1.5.0 of the install.yaml with minor annotation changes

`envoy-proxy.yaml`

```yaml
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: public-proxy-config
  namespace: envoy-gateway-system
spec:
  # Bootstrap configuration to fix admin interface IPv6 binding
  bootstrap:
    type: JSONPatch
    jsonPatches:
    - {"op": "replace", "path": "/admin/address/socket_address/port_value", "value": 19003}
    - {"op": "replace", "path": "/admin/address/socket_address/address", "value": "::"}

  ipFamily: IPv6
  logging:
    level:
      default: debug
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          # EKS Auto Mode compatible annotations
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
          service.beta.kubernetes.io/aws-load-balancer-subnets: "example-1, example-2"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
        type: LoadBalancer
        loadBalancerClass: eks.amazonaws.com/nlb
        externalTrafficPolicy: Cluster
```

`gateway-class.yaml`

```yaml
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: some-public
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: public-proxy-config
    namespace: envoy-gateway-system
```

`gateway.yaml`

```yaml
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: some-public
  namespace: envoy-gateway-system
  labels:
    gateway: public
spec:
  gatewayClassName: some-public
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: <some secret>
  # HTTP listener for redirects
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All

```

*Environment*:

Envoy installed through Helm chart at `v1.5.0`.
ArgoCD > 3.0 applies manifests. 
AWS EKS 1.33 Kubernetes w/ EKS Automode (karpenter enabled).

*Logs*:

Failure when envoy gateway is being deployed. Doesn't automatically resolve once service is online.

```
2025-08-16T02:27:27.124Z ERROR provider kubernetes/controller.go:618 failed to get Service {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-envoy-gateway-system-public-gateway-b3b5c242", "error": "Service \"envoy-envoy-gateway-system-public-gateway-b3b5c242\" not found"}
```


Discussion:
@arkodg: this is expected / eventual consistent state, relates to the new zone aware routing feature which relies on proxy fleet endpoint data
https://github.com/envoyproxy/gateway/pull/6597
we need to find a way to make this less alarming
cc @jukie @jukie: Yeah it shows up as an error if the service hasn't been created yet but it should resolve itself afterwards, albeit silently, without a restart.

If it's not an interesting error at normal runtime perhaps we should move this line to debug level.@adrielp: Eventual consistency makes sense, unfortunately, it seemed to only resolve once the pod was restarted. I left it running for an entire night, the service was online, but errors were still propagating. I can try and replicate it on the next cluster I build.@arkodg: +1 to moving this to `debug` since its not a error, this can also show up if a invalid/non existant backendRef is linked to a HTTPRoute@Kazaen: I have a very similar setup as @adrielp, except that in the EnvoyProxy, the service name is overwritten:
```
      envoyService:
        externalTrafficPolicy: Local
        name: envoy-envoy-gateway-system-eg
        type: ClusterIP
```

in order to be used by a TargetGroupBinding (aws-load-balancer-controller), it was easier to hardcode the name. However, since the upgrade to 1.5.0:

`2025-09-02T12:46:26.773Z	ERROR	provider	kubernetes/controller.go:618	failed to get Service	{"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-envoy-gateway-system-eg-5391c79d", "error": "Service \"envoy-envoy-gateway-system-eg-5391c79d\" not found"}`

Which means that the controller is expecting this name instead of the one that I override. Did i miss something ?@arkodg: @Kazaen was the envoyproxy attached to the gateway class ? if so its fixed with https://github.com/envoyproxy/gateway/pull/6838@Kazaen: > [@Kazaen](https://github.com/Kazaen) was the envoyproxy attached to the gateway class ? if so its fixed with [#6838](https://github.com/envoyproxy/gateway/pull/6838)

Indeed, it was ! 

Thanks for confirming.

@zirain: @arkodg  is this fixed by #6838?@arkodg: the work left here is to convert the error log into an info log@jukie: @adrielp this should be resolved now and I tested a few scenarios in a live cluster to confirm.

Please re-open if you're still experiencing this once v1.6.0 is out later this week or from an image built with the above commit.

Solution patch:
diff --git a/internal/provider/kubernetes/controller.go b/internal/provider/kubernetes/controller.go
index b471261f1..fac9d8d8b 100644
--- a/internal/provider/kubernetes/controller.go
+++ b/internal/provider/kubernetes/controller.go
@@ -1605,11 +1605,7 @@ func (r *gatewayAPIReconciler) processServiceClusterForGatewayClass(ep *egv1a1.E
 		}
 	}
 
-	resourceMap.insertBackendRef(gwapiv1.BackendObjectReference{
-		Kind:      ptr.To(gwapiv1.Kind("Service")),
-		Namespace: gatewayapi.NamespacePtr(proxySvcNamespace),
-		Name:      gwapiv1.ObjectName(proxySvcName),
-	})
+	r.insertProxyServiceIfExists(proxySvcName, proxySvcNamespace, resourceMap)
 }
 
 // Called on a Gateway when merged gateways mode is not enabled for its parent GatewayClass.
@@ -1634,10 +1630,24 @@ func (r *gatewayAPIReconciler) processServiceClusterForGateway(ep *egv1a1.EnvoyP
 		}
 	}
 
+	r.insertProxyServiceIfExists(proxySvcName, proxySvcNamespace, resourceMap)
+}
+
+func (r *gatewayAPIReconciler) insertProxyServiceIfExists(name, namespace string, resourceMap *resourceMappings) {
+	svcNN := types.NamespacedName{Name: name, Namespace: namespace}
+	svc := new(corev1.Service)
+	err := r.client.Get(context.Background(), svcNN, svc)
+	// Only insert if service exists
+	if err != nil {
+		if !kerrors.IsNotFound(err) {
+			r.log.Error(err, "failed to get proxy service", "namespace", namespace, "name", name)
+		}
+		return
+	}
 	resourceMap.insertBackendRef(gwapiv1.BackendObjectReference{
-		Kind:      ptr.To(gwapiv1.Kind("Service")),
-		Namespace: gatewayapi.NamespacePtr(proxySvcNamespace),
-		Name:      gwapiv1.ObjectName(proxySvcName),
+		Kind:      ptr.To(gwapiv1.Kind(resource.KindService)),
+		Namespace: gatewayapi.NamespacePtr(svc.Namespace),
+		Name:      gwapiv1.ObjectName(svc.Name),
 	})
 }
 



Test patch:
diff --git a/internal/provider/kubernetes/controller_test.go b/internal/provider/kubernetes/controller_test.go
index 9a978495a..c3d29519b 100644
--- a/internal/provider/kubernetes/controller_test.go
+++ b/internal/provider/kubernetes/controller_test.go
@@ -1249,32 +1249,43 @@ func TestProcessSecurityPolicyObjectRefs(t *testing.T) {
 }
 
 func TestProcessServiceClusterForGatewayClass(t *testing.T) {
+	gcName := "merged-gc"
+	nsName := "envoy-gateway-system"
 	testCases := []struct {
 		name            string
 		gatewayClass    *gwapiv1.GatewayClass
 		envoyProxy      *egv1a1.EnvoyProxy
 		expectedSvcName string
+		serviceCluster  []client.Object
 	}{
 		{
 			name: "when merged gateways and no hardcoded svc name is used",
 			gatewayClass: &gwapiv1.GatewayClass{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "merged-gc",
+					Name: gcName,
 				},
 			},
 			envoyProxy:      nil,
-			expectedSvcName: proxy.ExpectedResourceHashedName("merged-gc"),
+			expectedSvcName: proxy.ExpectedResourceHashedName(gcName),
+			serviceCluster: []client.Object{
+				&corev1.Service{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      proxy.ExpectedResourceHashedName(gcName),
+						Namespace: nsName,
+					},
+				},
+			},
 		},
 		{
 			name: "when merged gateways and a hardcoded svc name is used",
 			gatewayClass: &gwapiv1.GatewayClass{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "merged-gc",
+					Name: gcName,
 				},
 			},
 			envoyProxy: &egv1a1.EnvoyProxy{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "merged-gc",
+					Name: gcName,
 				},
 				Spec: egv1a1.EnvoyProxySpec{
 					Provider: &egv1a1.EnvoyProxyProvider{
@@ -1288,6 +1299,25 @@ func TestProcessServiceClusterForGatewayClass(t *testing.T) {
 				},
 			},
 			expectedSvcName: "merged-gc-svc",
+			serviceCluster: []client.Object{
+				&corev1.Service{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "merged-gc-svc",
+						Namespace: nsName,
+					},
+				},
+			},
+		},
+		{
+			name: "non-existent proxy service",
+			gatewayClass: &gwapiv1.GatewayClass{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: gcName,
+				},
+			},
+			envoyProxy:      nil,
+			expectedSvcName: proxy.ExpectedResourceHashedName(gcName),
+			serviceCluster:  nil,
 		},
 	}
 
@@ -1299,6 +1329,10 @@ func TestProcessServiceClusterForGatewayClass(t *testing.T) {
 			resourceMap := newResourceMapping()
 
 			r := newGatewayAPIReconciler(logger)
+			r.client = fakeclient.NewClientBuilder().
+				WithScheme(envoygateway.GetScheme()).
+				WithObjects(tc.serviceCluster...).
+				Build()
 			r.namespace = "envoy-gateway-system"
 
 			r.processServiceClusterForGatewayClass(tc.envoyProxy, tc.gatewayClass, resourceMap)
@@ -1309,8 +1343,12 @@ func TestProcessServiceClusterForGatewayClass(t *testing.T) {
 				Name:      gwapiv1.ObjectName(tc.expectedSvcName),
 			}
 			key := backendRefKey(&expectedRef)
-			require.Contains(t, resourceMap.allAssociatedBackendRefs, key)
-			require.Equal(t, expectedRef, resourceMap.allAssociatedBackendRefs[key])
+			if tc.serviceCluster != nil {
+				require.Contains(t, resourceMap.allAssociatedBackendRefs, key)
+				require.Equal(t, expectedRef, resourceMap.allAssociatedBackendRefs[key])
+			} else {
+				require.NotContains(t, resourceMap.allAssociatedBackendRefs, key)
+			}
 		})
 	}
 }
@@ -1324,6 +1362,7 @@ func TestProcessServiceClusterForGateway(t *testing.T) {
 		gatewayNamespacedMode  bool
 		expectedSvcName        string
 		expectedSvcNamespace   string
+		serviceCluster         []client.Object
 	}{
 		{
 			name: "no gateway namespaced mode with no hardcoded service name",
@@ -1377,6 +1416,14 @@ func TestProcessServiceClusterForGateway(t *testing.T) {
 			gatewayNamespacedMode: true,
 			expectedSvcName:       "my-gateway",
 			expectedSvcNamespace:  "app-namespace",
+			serviceCluster: []client.Object{
+				&corev1.Service{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "my-gateway",
+						Namespace: "app-namespace",
+					},
+				},
+			},
 		},
 		{
 			name: "gateway namespaced mode with hardcoded service name",
@@ -1404,6 +1451,14 @@ func TestProcessServiceClusterForGateway(t *testing.T) {
 			gatewayNamespacedMode: true,
 			expectedSvcName:       "my-gateway-svc",
 			expectedSvcNamespace:  "app-namespace",
+			serviceCluster: []client.Object{
+				&corev1.Service{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "my-gateway-svc",
+						Namespace: "app-namespace",
+					},
+				},
+			},
 		},
 		{
 			name: "no gateway namespaced mode with no hardcoded service name attached gatewayclass",
@@ -1443,6 +1498,10 @@ func TestProcessServiceClusterForGateway(t *testing.T) {
 			resourceMap := newResourceMapping()
 
 			r := newGatewayAPIReconciler(logger)
+			r.client = fakeclient.NewClientBuilder().
+				WithScheme(envoygateway.GetScheme()).
+				WithObjects(tc.serviceCluster...).
+				Build()
 			r.namespace = "envoy-gateway-system"
 			r.gatewayNamespaceMode = tc.gatewayNamespacedMode
 
@@ -1466,8 +1525,12 @@ func TestProcessServiceClusterForGateway(t *testing.T) {
 				Name:      gwapiv1.ObjectName(tc.expectedSvcName),
 			}
 			key := backendRefKey(&expectedRef)
-			require.Contains(t, resourceMap.allAssociatedBackendRefs, key)
-			require.Equal(t, expectedRef, resourceMap.allAssociatedBackendRefs[key])
+			if tc.serviceCluster != nil {
+				require.Contains(t, resourceMap.allAssociatedBackendRefs, key)
+				require.Equal(t, expectedRef, resourceMap.allAssociatedBackendRefs[key])
+			} else {
+				require.NotContains(t, resourceMap.allAssociatedBackendRefs, key)
+			}
 		})
 	}
 }