Repository: envoyproxy/gateway

Issue:
All HTTPRoutes fail to resolve when a single route's SecurityPolicy fails to load
*Description*:

We encountered an odd issue where user had accidentally defined an invalid htpasswd secret, which lead to all the HTTPRoutes in the cluster being unresponsive. Not sure if HTTPRoute registration is working as intended or if it's a bug in case of misconfigurations.

Setup was as follows:

- Cluster has multiple HTTPRoutes and one proxy deployment with 2 replicas
- A single route has a SecurityPolicy with basic auth enabled
- SecurityPolicy has an invalid Basic Auth with other than `SHA` format in the htpasswd secret

After this, all the HTTPRoutes fail to resolve properly. Gateway status in the kube resource seem to indicate loaded routes though.

Logs showed this as:

```
[2025-03-21 14:01:51.985][1][warning][config] [source/extensions/config_subscription/grpc/delta_subscription_state.cc:296] delta config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) envoy-gateway-system/internal/https: basic auth: unsupported htpasswd format: please use {SHA}
[2025-03-21 14:01:51.985][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) envoy-gateway-system/internal/https: basic auth: unsupported htpasswd format: please use {SHA}
```

Gateway showed that `N` listeners had been registered, but none of the routes worked with following CLI logs:

```
❯ curl https://... -v
* Host ...:443 was resolved.
* IPv6: (none)
* IPv4: ...
*   Trying ...:443...
* Connected to ... (...) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to ...:443
* Closing connection
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to ...:443

❯ openssl s_client -connect ...:443
Connecting to ...
CONNECTED(00000005)
40C842FE01000000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:693:
```

After fixing the htpasswd secret to correct SHA format (as it's the supported value), all the routes started working again. So a misconfiguration made the cluster's endpoints unavailable.

*Repro steps*:

- Define `HTTPRoutes` A and B for a cluster. Whether they are in the same namespace or not doesn't matter.
- Include a `SecurityPolicy` to route A with Basic Auth for user `foo` with e.g. SHA-512 htpasswd hash as long as it's not the correct supported `SHA` format.
- Neither of the routes are working

*Environment*:
Envoy Gateway v1.3.0 installed via Helm chart.

*Logs*:
Included in the description.


Hints from issue comments:
@arkodg: we'll need to add a helper function in the gateway api layer to validate the basic auth secret to make sure its in the right format

Solution patch:
diff --git a/.github/workflows/latest_release.yaml b/.github/workflows/latest_release.yaml
index 50c6c546b..88bac8ffd 100644
--- a/.github/workflows/latest_release.yaml
+++ b/.github/workflows/latest_release.yaml
@@ -107,7 +107,7 @@ jobs:
         GITHUB_REPOSITORY: ${{ github.repository_owner }}/${{ github.event.repository.name }}
 
     - name: Recreate the Latest Release and Tag
-      uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631  # v2.2.2
+      uses: softprops/action-gh-release@72f2c25fcb47643c292f7107632f7a47c1df5cd8  # v2.3.2
       with:
         draft: false
         prerelease: true
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
index a79633f76..2cb42adf1 100644
--- a/.github/workflows/release.yaml
+++ b/.github/workflows/release.yaml
@@ -97,7 +97,7 @@ jobs:
           zip -r egctl_${{ env.release_tag }}_windows_amd64.zip bin/windows/amd64/egctl
 
       - name: Upload Release Manifests
-        uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631  # v2.2.2
+        uses: softprops/action-gh-release@72f2c25fcb47643c292f7107632f7a47c1df5cd8  # v2.3.2
         with:
           files: |
             release-artifacts/install.yaml
diff --git a/api/v1alpha1/ext_proc_types.go b/api/v1alpha1/ext_proc_types.go
index 95893d3c6..9e821f68c 100644
--- a/api/v1alpha1/ext_proc_types.go
+++ b/api/v1alpha1/ext_proc_types.go
@@ -17,7 +17,8 @@ const (
 	StreamedExtProcBodyProcessingMode ExtProcBodyProcessingMode = "Streamed"
 	// BufferedExtProcBodyProcessingMode will buffer the message body in memory and send the entire body at once. If the body exceeds the configured buffer limit, then the downstream system will receive an error.
 	BufferedExtProcBodyProcessingMode ExtProcBodyProcessingMode = "Buffered"
-	// FullDuplexStreamedExtBodyProcessingMode will send the body in pieces, to be read in a stream. Full details here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/processing_mode.proto.html#enum-extensions-filters-http-ext-proc-v3-processingmode-bodysendmode
+	// FullDuplexStreamedExtBodyProcessingMode will send the body in pieces, to be read in a stream. When enabled, trailers are also sent, and failOpen must be false.
+	// Full details here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/processing_mode.proto.html#enum-extensions-filters-http-ext-proc-v3-processingmode-bodysendmode
 	FullDuplexStreamedExtBodyProcessingMode ExtProcBodyProcessingMode = "FullDuplexStreamed"
 	// BufferedPartialExtBodyHeaderProcessingMode will buffer the message body in memory and send the entire body in one chunk. If the body exceeds the configured buffer limit, then the body contents up to the buffer limit will be sent.
 	BufferedPartialExtBodyHeaderProcessingMode ExtProcBodyProcessingMode = "BufferedPartial"
@@ -67,6 +68,7 @@ type ExtProcProcessingMode struct {
 // +kubebuilder:validation:XValidation:message="BackendRefs must be used, backendRef is not supported.",rule="!has(self.backendRef)"
 // +kubebuilder:validation:XValidation:message="BackendRefs only supports Service and Backend kind.",rule="has(self.backendRefs) ? self.backendRefs.all(f, f.kind == 'Service' || f.kind == 'Backend') : true"
 // +kubebuilder:validation:XValidation:message="BackendRefs only supports Core and gateway.envoyproxy.io group.",rule="has(self.backendRefs) ? (self.backendRefs.all(f, f.group == \"\" || f.group == 'gateway.envoyproxy.io')) : true"
+// +kubebuilder:validation:XValidation:message="If FullDuplexStreamed body processing mode is used, FailOpen must be false.",rule="!(has(self.failOpen) && self.failOpen == true && ((has(self.processingMode.request.body) && self.processingMode.request.body == 'FullDuplexStreamed') || (has(self.processingMode.response.body) && self.processingMode.response.body == 'FullDuplexStreamed')))"
 type ExtProc struct {
 	BackendCluster `json:",inline"`
 
diff --git a/charts/gateway-addons-helm/README.md b/charts/gateway-addons-helm/README.md
index bd1c5adce..ad9bd969a 100644
--- a/charts/gateway-addons-helm/README.md
+++ b/charts/gateway-addons-helm/README.md
@@ -149,6 +149,11 @@ To uninstall the chart:
 | opentelemetry-collector.image.repository | string | `"otel/opentelemetry-collector-contrib"` |  |
 | opentelemetry-collector.image.tag | string | `"0.121.0"` |  |
 | opentelemetry-collector.mode | string | `"deployment"` |  |
+| opentelemetry-collector.ports.datadog.containerPort | int | `8126` |  |
+| opentelemetry-collector.ports.datadog.enabled | bool | `true` |  |
+| opentelemetry-collector.ports.datadog.hostPort | int | `8126` |  |
+| opentelemetry-collector.ports.datadog.protocol | string | `"TCP"` |  |
+| opentelemetry-collector.ports.datadog.servicePort | int | `8126` |  |
 | opentelemetry-collector.ports.envoy-als.appProtocol | string | `"grpc"` |  |
 | opentelemetry-collector.ports.envoy-als.containerPort | int | `9000` |  |
 | opentelemetry-collector.ports.envoy-als.enabled | bool | `true` |  |
diff --git a/charts/gateway-addons-helm/values.yaml b/charts/gateway-addons-helm/values.yaml
index b491b506a..309b6ee92 100644
--- a/charts/gateway-addons-helm/values.yaml
+++ b/charts/gateway-addons-helm/values.yaml
@@ -814,6 +814,12 @@ opentelemetry-collector:
       hostPort: 9000
       protocol: TCP
       appProtocol: grpc
+    datadog:
+      enabled: true
+      containerPort: 8126
+      servicePort: 8126
+      hostPort: 8126
+      protocol: TCP
   fullnameOverride: otel-collector
   mode: deployment
   image:
diff --git a/charts/gateway-crds-helm/templates/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml b/charts/gateway-crds-helm/templates/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
index 6896bb956..351359131 100644
--- a/charts/gateway-crds-helm/templates/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
+++ b/charts/gateway-crds-helm/templates/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
@@ -1057,6 +1057,12 @@ spec:
                       group.
                     rule: 'has(self.backendRefs) ? (self.backendRefs.all(f, f.group
                       == "" || f.group == ''gateway.envoyproxy.io'')) : true'
+                  - message: If FullDuplexStreamed body processing mode is used, FailOpen
+                      must be false.
+                    rule: '!(has(self.failOpen) && self.failOpen == true && ((has(self.processingMode.request.body)
+                      && self.processingMode.request.body == ''FullDuplexStreamed'')
+                      || (has(self.processingMode.response.body) && self.processingMode.response.body
+                      == ''FullDuplexStreamed'')))'
                 maxItems: 16
                 type: array
               lua:
diff --git a/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml b/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
index 9e2e94109..e825e8284 100644
--- a/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
+++ b/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyextensionpolicies.yaml
@@ -1056,6 +1056,12 @@ spec:
                       group.
                     rule: 'has(self.backendRefs) ? (self.backendRefs.all(f, f.group
                       == "" || f.group == ''gateway.envoyproxy.io'')) : true'
+                  - message: If FullDuplexStreamed body processing mode is used, FailOpen
+                      must be false.
+                    rule: '!(has(self.failOpen) && self.failOpen == true && ((has(self.processingMode.request.body)
+                      && self.processingMode.request.body == ''FullDuplexStreamed'')
+                      || (has(self.processingMode.response.body) && self.processingMode.response.body
+                      == ''FullDuplexStreamed'')))'
                 maxItems: 16
                 type: array
               lua:
diff --git a/internal/gatewayapi/securitypolicy.go b/internal/gatewayapi/securitypolicy.go
index 6c136941f..b31dd073a 100644
--- a/internal/gatewayapi/securitypolicy.go
+++ b/internal/gatewayapi/securitypolicy.go
@@ -363,6 +363,12 @@ func validateSecurityPolicy(p *egv1a1.SecurityPolicy) error {
 		}
 	}
 
+	basicAuth := p.Spec.BasicAuth
+	if basicAuth != nil {
+		if err := validateBasicAuth(basicAuth); err != nil {
+			return err
+		}
+	}
 	return nil
 }
 
@@ -378,6 +384,16 @@ func validateAPIKeyAuth(apiKeyAuth *egv1a1.APIKeyAuth) error {
 	return nil
 }
 
+// validateBasicAuth validates the BasicAuth configuration.
+// Currently, we only validate that the secret exists, but we don't validate
+// the content of the secret. This function will be called when the security policy
+// is being processed, but before the secret is actually read.
+func validateBasicAuth(basicAuth *egv1a1.BasicAuth) error {
+	// The actual validation of the htpasswd format will happen when the secret is read
+	// in the buildBasicAuth function.
+	return nil
+}
+
 func resolveSecurityPolicyGatewayTargetRef(
 	policy *egv1a1.SecurityPolicy,
 	target gwapiv1a2.LocalPolicyTargetReferenceWithSectionName,
@@ -1309,6 +1325,11 @@ func (t *Translator) buildBasicAuth(
 			usersSecret.Namespace, usersSecret.Name)
 	}
 
+	// Validate the htpasswd format
+	if err := validateHtpasswdFormat(usersSecretBytes); err != nil {
+		return nil, err
+	}
+
 	return &ir.BasicAuth{
 		Name:                  irConfigName(policy),
 		Users:                 usersSecretBytes,
@@ -1316,6 +1337,29 @@ func (t *Translator) buildBasicAuth(
 	}, nil
 }
 
+// validateHtpasswdFormat validates that the htpasswd data is in the correct format.
+// Currently, only the SHA format is supported by Envoy.
+func validateHtpasswdFormat(data []byte) error {
+	lines := strings.Split(string(data), "\n")
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+
+		parts := strings.SplitN(line, ":", 2)
+		if len(parts) != 2 {
+			return fmt.Errorf("invalid htpasswd format: each line must be in the format 'username:password'")
+		}
+
+		password := parts[1]
+		if !strings.HasPrefix(password, "{SHA}") {
+			return fmt.Errorf("unsupported htpasswd format: please use {SHA}")
+		}
+	}
+	return nil
+}
+
 func (t *Translator) buildExtAuth(
 	policy *egv1a1.SecurityPolicy,
 	resources *resource.Resources,
diff --git a/internal/xds/translator/extproc.go b/internal/xds/translator/extproc.go
index 1aca83fdb..a1039575f 100644
--- a/internal/xds/translator/extproc.go
+++ b/internal/xds/translator/extproc.go
@@ -99,16 +99,10 @@ func extProcConfig(extProc ir.ExtProc) *extprocv3.ExternalProcessor {
 				Seconds: defaultExtServiceRequestTimeout,
 			},
 		},
-		ProcessingMode: &extprocv3.ProcessingMode{
-			RequestHeaderMode:   extprocv3.ProcessingMode_SKIP,
-			ResponseHeaderMode:  extprocv3.ProcessingMode_SKIP,
-			RequestBodyMode:     extprocv3.ProcessingMode_NONE,
-			ResponseBodyMode:    extprocv3.ProcessingMode_NONE,
-			RequestTrailerMode:  extprocv3.ProcessingMode_SKIP,
-			ResponseTrailerMode: extprocv3.ProcessingMode_SKIP,
-		},
 	}
 
+	config.ProcessingMode = buildProcessingMode(extProc)
+
 	if extProc.FailOpen != nil {
 		config.FailureModeAllow = *extProc.FailOpen
 	}
@@ -117,22 +111,6 @@ func extProcConfig(extProc ir.ExtProc) *extprocv3.ExternalProcessor {
 		config.MessageTimeout = durationpb.New(extProc.MessageTimeout.Duration)
 	}
 
-	if extProc.RequestBodyProcessingMode != nil {
-		config.ProcessingMode.RequestBodyMode = buildExtProcBodyProcessingMode(extProc.RequestBodyProcessingMode)
-	}
-
-	if extProc.RequestHeaderProcessing {
-		config.ProcessingMode.RequestHeaderMode = extprocv3.ProcessingMode_SEND
-	}
-
-	if extProc.ResponseBodyProcessingMode != nil {
-		config.ProcessingMode.ResponseBodyMode = buildExtProcBodyProcessingMode(extProc.ResponseBodyProcessingMode)
-	}
-
-	if extProc.ResponseHeaderProcessing {
-		config.ProcessingMode.ResponseHeaderMode = extprocv3.ProcessingMode_SEND
-	}
-
 	if extProc.RequestAttributes != nil {
 		var attrs []string
 		attrs = append(attrs, extProc.RequestAttributes...)
@@ -232,7 +210,43 @@ func (*extProc) patchRoute(route *routev3.Route, irRoute *ir.HTTPRoute, _ *ir.HT
 	return nil
 }
 
-func buildExtProcBodyProcessingMode(mode *ir.ExtProcBodyProcessingMode) extprocv3.ProcessingMode_BodySendMode {
+func buildProcessingMode(extProc ir.ExtProc) *extprocv3.ProcessingMode {
+	processingMode := &extprocv3.ProcessingMode{
+		RequestHeaderMode:   extprocv3.ProcessingMode_SKIP,
+		ResponseHeaderMode:  extprocv3.ProcessingMode_SKIP,
+		RequestBodyMode:     extprocv3.ProcessingMode_NONE,
+		ResponseBodyMode:    extprocv3.ProcessingMode_NONE,
+		RequestTrailerMode:  extprocv3.ProcessingMode_SKIP,
+		ResponseTrailerMode: extprocv3.ProcessingMode_SKIP,
+	}
+
+	if extProc.RequestBodyProcessingMode != nil {
+		processingMode.RequestBodyMode = translateExtProcBodyProcessingMode(extProc.RequestBodyProcessingMode)
+		//
+		if processingMode.RequestBodyMode == extprocv3.ProcessingMode_FULL_DUPLEX_STREAMED {
+			processingMode.RequestTrailerMode = extprocv3.ProcessingMode_SEND
+		}
+	}
+
+	if extProc.RequestHeaderProcessing {
+		processingMode.RequestHeaderMode = extprocv3.ProcessingMode_SEND
+	}
+
+	if extProc.ResponseBodyProcessingMode != nil {
+		processingMode.ResponseBodyMode = translateExtProcBodyProcessingMode(extProc.ResponseBodyProcessingMode)
+		if processingMode.ResponseBodyMode == extprocv3.ProcessingMode_FULL_DUPLEX_STREAMED {
+			processingMode.ResponseTrailerMode = extprocv3.ProcessingMode_SEND
+		}
+	}
+
+	if extProc.ResponseHeaderProcessing {
+		processingMode.ResponseHeaderMode = extprocv3.ProcessingMode_SEND
+	}
+
+	return processingMode
+}
+
+func translateExtProcBodyProcessingMode(mode *ir.ExtProcBodyProcessingMode) extprocv3.ProcessingMode_BodySendMode {
 	lookup := map[ir.ExtProcBodyProcessingMode]extprocv3.ProcessingMode_BodySendMode{
 		ir.ExtProcBodyBuffered:           extprocv3.ProcessingMode_BUFFERED,
 		ir.ExtProcBodyBufferedPartial:    extprocv3.ProcessingMode_BUFFERED_PARTIAL,
diff --git a/release-notes/current.yaml b/release-notes/current.yaml
index df7b4c760..aa805e8c8 100644
--- a/release-notes/current.yaml
+++ b/release-notes/current.yaml
@@ -36,6 +36,9 @@ bug fixes: |
   Fixed issue that switch on wrong SubjectAltNameType enum value in BackendTLSPolicy.
   Fixed issue that BackendTLSPolicy should not reference ConfigMap or Secret across namespace.
   Fixed bug in certificate SANs overlap detection in listeners.
+  Fixed issue where EnvoyExtensionPolicy ExtProc body processing mode is set to FullDuplexStreamed, but trailers were not sent.
+  Fixed validation issue where EnvoyExtensionPolicy ExtProc failOpen is true, and body processing mode FullDuplexStreamed is not rejected.
+
 
 # Enhancements that improve performance.
 performance improvements: |



Test patch:
diff --git a/internal/gatewayapi/securitypolicy_test.go b/internal/gatewayapi/securitypolicy_test.go
index 2d1308dc8..0685b044a 100644
--- a/internal/gatewayapi/securitypolicy_test.go
+++ b/internal/gatewayapi/securitypolicy_test.go
@@ -660,3 +660,57 @@ func Test_OIDC_PassThroughAuthHeader(t *testing.T) {
 func ToPointer[T any](v T) *T {
 	return &v
 }
+
+func Test_validateHtpasswdFormat(t *testing.T) {
+	tests := []struct {
+		name      string
+		htpasswd  string
+		wantError bool
+	}{
+		{
+			name:      "valid htpasswd with SHA format",
+			htpasswd:  "user1:{SHA}hashed_user1_password\nuser2:{SHA}hashed_user2_password",
+			wantError: false,
+		},
+		{
+			name:      "valid htpasswd with SHA format and empty lines",
+			htpasswd:  "user1:{SHA}hashed_user1_password\n\nuser2:{SHA}hashed_user2_password\n",
+			wantError: false,
+		},
+		{
+			name:      "invalid htpasswd with missing SHA prefix",
+			htpasswd:  "user1:hashed_user1_password",
+			wantError: true,
+		},
+		{
+			name:      "invalid htpasswd with MD5 format",
+			htpasswd:  "user1:$apr1$hashed_user1_password",
+			wantError: true,
+		},
+		{
+			name:      "invalid htpasswd with bcrypt format",
+			htpasswd:  "user1:$2y$hashed_user1_password",
+			wantError: true,
+		},
+		{
+			name:      "invalid htpasswd with missing colon",
+			htpasswd:  "user1{SHA}hashed_user1_password",
+			wantError: true,
+		},
+		{
+			name:      "mixed valid and invalid formats",
+			htpasswd:  "user1:{SHA}hashed_user1_password\nuser2:$apr1$hashed_user2_password",
+			wantError: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := validateHtpasswdFormat([]byte(tt.htpasswd))
+			if (err != nil) != tt.wantError {
+				t.Errorf("validateHtpasswdFormat() error = %v, wantErr %v", err, tt.wantError)
+				return
+			}
+		})
+	}
+}
diff --git a/internal/xds/translator/testdata/in/xds-ir/ext-proc.yaml b/internal/xds/translator/testdata/in/xds-ir/ext-proc.yaml
index 6f39befb8..bc4b11ed0 100644
--- a/internal/xds/translator/testdata/in/xds-ir/ext-proc.yaml
+++ b/internal/xds/translator/testdata/in/xds-ir/ext-proc.yaml
@@ -102,6 +102,28 @@ http:
                   - protocol: GRPC
                     weight: 1
                     name: envoyextensionpolicy/envoy-gateway/policy-for-gateway-1/0/grpc-backend/backend/0
+      - destination:
+          name: httproute/default/httproute-3/rule/0
+          settings:
+            - addressType: IP
+              endpoints:
+                - host: 7.7.7.7
+                  port: 8080
+              protocol: HTTP
+              weight: 1
+              name: httproute/default/httproute-3/rule/0/backend/0
+        envoyExtensions:
+          extProcs:
+            - name: envoyextensionpolicy/envoy-gateway/policy-for-route-3/extproc/0
+              authority: grpc-backend-3.envoy-gateway:3000
+              requestBodyProcessingMode: FullDuplexStreamed
+              responseBodyProcessingMode: FullDuplexStreamed
+              destination:
+                name: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3
+                settings:
+                  - protocol: GRPC
+                    weight: 1
+                    name: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3/backend/0
         hostname: gateway.envoyproxy.io
         isHTTP2: false
         name: httproute/default/httproute-2/rule/0/match/0/gateway_envoyproxy_io
diff --git a/internal/xds/translator/testdata/out/xds-ir/ext-proc.clusters.yaml b/internal/xds/translator/testdata/out/xds-ir/ext-proc.clusters.yaml
index 14ff9d61d..6e72ef509 100755
--- a/internal/xds/translator/testdata/out/xds-ir/ext-proc.clusters.yaml
+++ b/internal/xds/translator/testdata/out/xds-ir/ext-proc.clusters.yaml
@@ -32,6 +32,23 @@
   name: httproute/default/httproute-2/rule/0
   perConnectionBufferLimitBytes: 32768
   type: EDS
+- circuitBreakers:
+    thresholds:
+    - maxRetries: 1024
+  commonLbConfig:
+    localityWeightedLbConfig: {}
+  connectTimeout: 10s
+  dnsLookupFamily: V4_PREFERRED
+  edsClusterConfig:
+    edsConfig:
+      ads: {}
+      resourceApiVersion: V3
+    serviceName: httproute/default/httproute-3/rule/0
+  ignoreHealthOnHostRemoval: true
+  lbPolicy: LEAST_REQUEST
+  name: httproute/default/httproute-3/rule/0
+  perConnectionBufferLimitBytes: 32768
+  type: EDS
 - circuitBreakers:
     thresholds:
     - maxRetries: 1024
@@ -128,3 +145,27 @@
         http2ProtocolOptions:
           initialConnectionWindowSize: 1048576
           initialStreamWindowSize: 65536
+- circuitBreakers:
+    thresholds:
+    - maxRetries: 1024
+  commonLbConfig:
+    localityWeightedLbConfig: {}
+  connectTimeout: 10s
+  dnsLookupFamily: V4_PREFERRED
+  edsClusterConfig:
+    edsConfig:
+      ads: {}
+      resourceApiVersion: V3
+    serviceName: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3
+  ignoreHealthOnHostRemoval: true
+  lbPolicy: LEAST_REQUEST
+  name: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3
+  perConnectionBufferLimitBytes: 32768
+  type: EDS
+  typedExtensionProtocolOptions:
+    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
+      '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
+      explicitHttpConfig:
+        http2ProtocolOptions:
+          initialConnectionWindowSize: 1048576
+          initialStreamWindowSize: 65536
diff --git a/internal/xds/translator/testdata/out/xds-ir/ext-proc.endpoints.yaml b/internal/xds/translator/testdata/out/xds-ir/ext-proc.endpoints.yaml
index 4ec680ce7..b6f842679 100755
--- a/internal/xds/translator/testdata/out/xds-ir/ext-proc.endpoints.yaml
+++ b/internal/xds/translator/testdata/out/xds-ir/ext-proc.endpoints.yaml
@@ -22,6 +22,18 @@
     loadBalancingWeight: 1
     locality:
       region: httproute/default/httproute-2/rule/0/backend/0
+- clusterName: httproute/default/httproute-3/rule/0
+  endpoints:
+  - lbEndpoints:
+    - endpoint:
+        address:
+          socketAddress:
+            address: 7.7.7.7
+            portValue: 8080
+      loadBalancingWeight: 1
+    loadBalancingWeight: 1
+    locality:
+      region: httproute/default/httproute-3/rule/0/backend/0
 - clusterName: envoyextensionpolicy/default/policy-for-route-2/0/grpc-backend-4
   endpoints:
   - loadBalancingWeight: 1
@@ -42,3 +54,8 @@
   - loadBalancingWeight: 1
     locality:
       region: envoyextensionpolicy/envoy-gateway/policy-for-gateway-1/0/grpc-backend/backend/0
+- clusterName: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3
+  endpoints:
+  - loadBalancingWeight: 1
+    locality:
+      region: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3/backend/0
diff --git a/internal/xds/translator/testdata/out/xds-ir/ext-proc.listeners.yaml b/internal/xds/translator/testdata/out/xds-ir/ext-proc.listeners.yaml
index 16f313934..7b5e8bbb7 100755
--- a/internal/xds/translator/testdata/out/xds-ir/ext-proc.listeners.yaml
+++ b/internal/xds/translator/testdata/out/xds-ir/ext-proc.listeners.yaml
@@ -103,6 +103,22 @@
             - connection.requested_server_name
             responseAttributes:
             - request.path
+        - disabled: true
+          name: envoy.filters.http.ext_proc/envoyextensionpolicy/envoy-gateway/policy-for-route-3/extproc/0
+          typedConfig:
+            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
+            grpcService:
+              envoyGrpc:
+                authority: grpc-backend-3.envoy-gateway:3000
+                clusterName: envoyextensionpolicy/envoy-gateway/policy-for-route-3/0/grpc-backend-3
+              timeout: 10s
+            processingMode:
+              requestBodyMode: FULL_DUPLEX_STREAMED
+              requestHeaderMode: SKIP
+              requestTrailerMode: SEND
+              responseBodyMode: FULL_DUPLEX_STREAMED
+              responseHeaderMode: SKIP
+              responseTrailerMode: SEND
         - name: envoy.filters.http.router
           typedConfig:
             '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
diff --git a/internal/xds/translator/testdata/out/xds-ir/ext-proc.routes.yaml b/internal/xds/translator/testdata/out/xds-ir/ext-proc.routes.yaml
index c17e6456d..8cef38ab6 100755
--- a/internal/xds/translator/testdata/out/xds-ir/ext-proc.routes.yaml
+++ b/internal/xds/translator/testdata/out/xds-ir/ext-proc.routes.yaml
@@ -22,6 +22,20 @@
     - match:
         pathSeparatedPrefix: /bar
       name: httproute/default/httproute-2/rule/0/match/0/gateway_envoyproxy_io
+      route:
+        cluster: httproute/default/httproute-3/rule/0
+        upgradeConfigs:
+        - upgradeType: websocket
+      typedPerFilterConfig:
+        envoy.filters.http.ext_proc/envoyextensionpolicy/envoy-gateway/policy-for-route-3/extproc/0:
+          '@type': type.googleapis.com/envoy.config.route.v3.FilterConfig
+          config: {}
+  - domains:
+    - ""
+    name: envoy-gateway/gateway-1/http/
+    routes:
+    - match:
+        prefix: /
       route:
         cluster: httproute/default/httproute-2/rule/0
         upgradeConfigs:
diff --git a/site/content/en/latest/api/extension_types.md b/site/content/en/latest/api/extension_types.md
index db10b6eb9..855ddb417 100644
--- a/site/content/en/latest/api/extension_types.md
+++ b/site/content/en/latest/api/extension_types.md
@@ -1715,7 +1715,7 @@ _Appears in:_
 | ----- | ----------- |
 | `Streamed` | StreamedExtProcBodyProcessingMode will stream the body to the server in pieces as they arrive at the proxy.<br /> | 
 | `Buffered` | BufferedExtProcBodyProcessingMode will buffer the message body in memory and send the entire body at once. If the body exceeds the configured buffer limit, then the downstream system will receive an error.<br /> | 
-| `FullDuplexStreamed` | FullDuplexStreamedExtBodyProcessingMode will send the body in pieces, to be read in a stream. Full details here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/processing_mode.proto.html#enum-extensions-filters-http-ext-proc-v3-processingmode-bodysendmode<br /> | 
+| `FullDuplexStreamed` | FullDuplexStreamedExtBodyProcessingMode will send the body in pieces, to be read in a stream. When enabled, trailers are also sent, and failOpen must be false.<br />Full details here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/processing_mode.proto.html#enum-extensions-filters-http-ext-proc-v3-processingmode-bodysendmode<br /> | 
 | `BufferedPartial` | BufferedPartialExtBodyHeaderProcessingMode will buffer the message body in memory and send the entire body in one chunk. If the body exceeds the configured buffer limit, then the body contents up to the buffer limit will be sent.<br /> | 
 
 
diff --git a/site/content/en/latest/install/gateway-addons-helm-api.md b/site/content/en/latest/install/gateway-addons-helm-api.md
index 75e2be730..df002772c 100644
--- a/site/content/en/latest/install/gateway-addons-helm-api.md
+++ b/site/content/en/latest/install/gateway-addons-helm-api.md
@@ -128,6 +128,11 @@ An Add-ons Helm chart for Envoy Gateway
 | opentelemetry-collector.image.repository | string | `"otel/opentelemetry-collector-contrib"` |  |
 | opentelemetry-collector.image.tag | string | `"0.121.0"` |  |
 | opentelemetry-collector.mode | string | `"deployment"` |  |
+| opentelemetry-collector.ports.datadog.containerPort | int | `8126` |  |
+| opentelemetry-collector.ports.datadog.enabled | bool | `true` |  |
+| opentelemetry-collector.ports.datadog.hostPort | int | `8126` |  |
+| opentelemetry-collector.ports.datadog.protocol | string | `"TCP"` |  |
+| opentelemetry-collector.ports.datadog.servicePort | int | `8126` |  |
 | opentelemetry-collector.ports.envoy-als.appProtocol | string | `"grpc"` |  |
 | opentelemetry-collector.ports.envoy-als.containerPort | int | `9000` |  |
 | opentelemetry-collector.ports.envoy-als.enabled | bool | `true` |  |
diff --git a/test/cel-validation/envoyextensionpolicy_test.go b/test/cel-validation/envoyextensionpolicy_test.go
index c8f3fe776..195d45dd0 100644
--- a/test/cel-validation/envoyextensionpolicy_test.go
+++ b/test/cel-validation/envoyextensionpolicy_test.go
@@ -414,6 +414,209 @@ func TestEnvoyExtensionPolicyTarget(t *testing.T) {
 				"spec.extProc[0].processingMode.request.body: Unsupported value: \"not-a-body-mode\": supported values: \"Streamed\", \"Buffered\", \"BufferedPartial\"",
 			},
 		},
+		{
+			desc: "valid ExtProc without FullDuplexStreamed body and failOpen true",
+			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
+				sp.Spec = egv1a1.EnvoyExtensionPolicySpec{
+					ExtProc: []egv1a1.ExtProc{
+						{
+							BackendCluster: egv1a1.BackendCluster{
+								BackendRefs: []egv1a1.BackendRef{
+									{
+										BackendObjectReference: gwapiv1.BackendObjectReference{
+											Name: "grpc-proc-service",
+											Port: ptr.To(gwapiv1.PortNumber(80)),
+										},
+									},
+								},
+							},
+							ProcessingMode: &egv1a1.ExtProcProcessingMode{
+								Request: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("Buffered")),
+								},
+								Response: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("Buffered")),
+								},
+							},
+							FailOpen: ptr.To(true),
+						},
+					},
+					PolicyTargetReferences: egv1a1.PolicyTargetReferences{
+						TargetRef: &gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+							LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{
+								Group: "gateway.networking.k8s.io",
+								Kind:  "Gateway",
+								Name:  "eg",
+							},
+						},
+					},
+				}
+			},
+			wantErrors: []string{},
+		},
+		{
+			desc: "valid ExtProc with FullDuplexStreamed body and failOpen false",
+			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
+				sp.Spec = egv1a1.EnvoyExtensionPolicySpec{
+					ExtProc: []egv1a1.ExtProc{
+						{
+							BackendCluster: egv1a1.BackendCluster{
+								BackendRefs: []egv1a1.BackendRef{
+									{
+										BackendObjectReference: gwapiv1.BackendObjectReference{
+											Name: "grpc-proc-service",
+											Port: ptr.To(gwapiv1.PortNumber(80)),
+										},
+									},
+								},
+							},
+							ProcessingMode: &egv1a1.ExtProcProcessingMode{
+								Request: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+								Response: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+							},
+							FailOpen: ptr.To(false),
+						},
+					},
+					PolicyTargetReferences: egv1a1.PolicyTargetReferences{
+						TargetRef: &gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+							LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{
+								Group: "gateway.networking.k8s.io",
+								Kind:  "Gateway",
+								Name:  "eg",
+							},
+						},
+					},
+				}
+			},
+			wantErrors: []string{},
+		},
+		{
+			desc: "valid ExtProc with FullDuplexStreamed body and failOpen nil",
+			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
+				sp.Spec = egv1a1.EnvoyExtensionPolicySpec{
+					ExtProc: []egv1a1.ExtProc{
+						{
+							BackendCluster: egv1a1.BackendCluster{
+								BackendRefs: []egv1a1.BackendRef{
+									{
+										BackendObjectReference: gwapiv1.BackendObjectReference{
+											Name: "grpc-proc-service",
+											Port: ptr.To(gwapiv1.PortNumber(80)),
+										},
+									},
+								},
+							},
+							ProcessingMode: &egv1a1.ExtProcProcessingMode{
+								Request: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+								Response: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+							},
+						},
+					},
+					PolicyTargetReferences: egv1a1.PolicyTargetReferences{
+						TargetRef: &gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+							LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{
+								Group: "gateway.networking.k8s.io",
+								Kind:  "Gateway",
+								Name:  "eg",
+							},
+						},
+					},
+				}
+			},
+			wantErrors: []string{},
+		},
+		{
+			desc: "invalid ExtProc with FullDuplexStreamed request body and failOpen",
+			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
+				sp.Spec = egv1a1.EnvoyExtensionPolicySpec{
+					ExtProc: []egv1a1.ExtProc{
+						{
+							BackendCluster: egv1a1.BackendCluster{
+								BackendRefs: []egv1a1.BackendRef{
+									{
+										BackendObjectReference: gwapiv1.BackendObjectReference{
+											Name: "grpc-proc-service",
+											Port: ptr.To(gwapiv1.PortNumber(80)),
+										},
+									},
+								},
+							},
+							ProcessingMode: &egv1a1.ExtProcProcessingMode{
+								Request: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+								Response: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("Buffered")),
+								},
+							},
+							FailOpen: ptr.To(true),
+						},
+					},
+					PolicyTargetReferences: egv1a1.PolicyTargetReferences{
+						TargetRef: &gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+							LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{
+								Group: "gateway.networking.k8s.io",
+								Kind:  "Gateway",
+								Name:  "eg",
+							},
+						},
+					},
+				}
+			},
+			wantErrors: []string{
+				"spec.extProc[0]: Invalid value: \"object\": If FullDuplexStreamed body processing mode is used, FailOpen must be false.",
+			},
+		},
+		{
+			desc: "invalid ExtProc with FullDuplexStreamed response body and failOpen",
+			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
+				sp.Spec = egv1a1.EnvoyExtensionPolicySpec{
+					ExtProc: []egv1a1.ExtProc{
+						{
+							BackendCluster: egv1a1.BackendCluster{
+								BackendRefs: []egv1a1.BackendRef{
+									{
+										BackendObjectReference: gwapiv1.BackendObjectReference{
+											Name: "grpc-proc-service",
+											Port: ptr.To(gwapiv1.PortNumber(80)),
+										},
+									},
+								},
+							},
+							ProcessingMode: &egv1a1.ExtProcProcessingMode{
+								Request: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("Buffered")),
+								},
+								Response: &egv1a1.ProcessingModeOptions{
+									Body: ptr.To(egv1a1.ExtProcBodyProcessingMode("FullDuplexStreamed")),
+								},
+							},
+							FailOpen: ptr.To(true),
+						},
+					},
+					PolicyTargetReferences: egv1a1.PolicyTargetReferences{
+						TargetRef: &gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+							LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{
+								Group: "gateway.networking.k8s.io",
+								Kind:  "Gateway",
+								Name:  "eg",
+							},
+						},
+					},
+				}
+			},
+			wantErrors: []string{
+				"spec.extProc[0]: Invalid value: \"object\": If FullDuplexStreamed body processing mode is used, FailOpen must be false.",
+			},
+		},
 		{
 			desc: "Valid Lua filter (inline)",
 			mutate: func(sp *egv1a1.EnvoyExtensionPolicy) {
diff --git a/test/e2e/testdata/tracing-datadog.yaml b/test/e2e/testdata/tracing-datadog.yaml
index e3c2a1b6b..d39f5e1fa 100644
--- a/test/e2e/testdata/tracing-datadog.yaml
+++ b/test/e2e/testdata/tracing-datadog.yaml
@@ -18,21 +18,6 @@ spec:
       kind: EnvoyProxy
       name: datadog-tracing
 ---
-apiVersion: v1
-kind: Service
-metadata:
-  name: datadog-agent
-  namespace: monitoring
-spec:
-  selector:
-    app.kubernetes.io/instance: eg-addons
-    app.kubernetes.io/name: opentelemetry-collector
-    component: standalone-collector
-  ports:
-    - protocol: TCP
-      port: 8126
-      targetPort: 8126
----
 apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: EnvoyProxy
 metadata:
@@ -48,9 +33,8 @@ spec:
       provider:
         type: Datadog
         backendRefs:
-          - name: backend-fqdn
-            kind: Backend
-            group: gateway.envoyproxy.io
+          - name: otel-collector
+            namespace: monitoring
             port: 8126
       customTags:
         "provider":
@@ -75,17 +59,6 @@ spec:
     drainTimeout: 5s
     minDrainDuration: 1s
 ---
-apiVersion: gateway.envoyproxy.io/v1alpha1
-kind: Backend
-metadata:
-  name: backend-fqdn
-  namespace: gateway-conformance-infra
-spec:
-  endpoints:
-    - fqdn:
-        hostname: datadog-agent.monitoring.svc.cluster.local
-        port: 8126
----
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
 metadata:
diff --git a/test/helm/gateway-addons-helm/e2e.out.yaml b/test/helm/gateway-addons-helm/e2e.out.yaml
index 60c1ec512..6684b559a 100644
--- a/test/helm/gateway-addons-helm/e2e.out.yaml
+++ b/test/helm/gateway-addons-helm/e2e.out.yaml
@@ -9747,6 +9747,10 @@ spec:
   type: ClusterIP
   ports:
     
+    - name: datadog
+      port: 8126
+      targetPort: 8126
+      protocol: TCP
     - name: envoy-als
       port: 9000
       targetPort: 9000
@@ -9997,6 +10001,9 @@ spec:
           imagePullPolicy: IfNotPresent
           ports:
             
+            - name: datadog
+              containerPort: 8126
+              protocol: TCP
             - name: envoy-als
               containerPort: 9000
               protocol: TCP
diff --git a/test/helm/gateway-crds-helm/all.out.yaml b/test/helm/gateway-crds-helm/all.out.yaml
index a723ab566..9f093bfb1 100644
--- a/test/helm/gateway-crds-helm/all.out.yaml
+++ b/test/helm/gateway-crds-helm/all.out.yaml
@@ -22227,6 +22227,12 @@ spec:
                       group.
                     rule: 'has(self.backendRefs) ? (self.backendRefs.all(f, f.group
                       == "" || f.group == ''gateway.envoyproxy.io'')) : true'
+                  - message: If FullDuplexStreamed body processing mode is used, FailOpen
+                      must be false.
+                    rule: '!(has(self.failOpen) && self.failOpen == true && ((has(self.processingMode.request.body)
+                      && self.processingMode.request.body == ''FullDuplexStreamed'')
+                      || (has(self.processingMode.response.body) && self.processingMode.response.body
+                      == ''FullDuplexStreamed'')))'
                 maxItems: 16
                 type: array
               lua:
diff --git a/test/helm/gateway-crds-helm/envoy-gateway-crds.out.yaml b/test/helm/gateway-crds-helm/envoy-gateway-crds.out.yaml
index d4a47a424..352dcf6ed 100644
--- a/test/helm/gateway-crds-helm/envoy-gateway-crds.out.yaml
+++ b/test/helm/gateway-crds-helm/envoy-gateway-crds.out.yaml
@@ -4915,6 +4915,12 @@ spec:
                       group.
                     rule: 'has(self.backendRefs) ? (self.backendRefs.all(f, f.group
                       == "" || f.group == ''gateway.envoyproxy.io'')) : true'
+                  - message: If FullDuplexStreamed body processing mode is used, FailOpen
+                      must be false.
+                    rule: '!(has(self.failOpen) && self.failOpen == true && ((has(self.processingMode.request.body)
+                      && self.processingMode.request.body == ''FullDuplexStreamed'')
+                      || (has(self.processingMode.response.body) && self.processingMode.response.body
+                      == ''FullDuplexStreamed'')))'
                 maxItems: 16
                 type: array
               lua: