Repository: envoyproxy/gateway

Issue:
hostNetwork support
*Description*:
>I'd like to ask about hostNetwork support in Custom Resource EnvoyProxy

[optional *Relevant Links*:]
>https://gateway.envoyproxy.io/v0.5.0/api/config_types.html#kubernetesdeploymentspec


Discussion:
@arkodg: @yangchuansheng can you share your use case / why you need this feature ?@arkodg: copying ChatGPT's answer
```
In Kubernetes, the hostNetwork is a Pod-level setting that allows you to share the network namespace with the host machine on which the Pod is scheduled. When you configure a Pod to use the hostNetwork, it essentially means that the Pod directly uses the host machine's network stack, and it doesn't have its own isolated network namespace. This can be useful in certain scenarios, but it comes with some trade-offs and should be used with caution.

Here are some reasons why users might configure hostNetwork in Kubernetes:

Performance: By using the host's network stack, you can potentially achieve better network performance for applications that require low-latency communication. This is particularly relevant for applications that demand high network throughput.
Access to Host Networking Resources: Some applications might need direct access to network resources that are only available on the host machine, such as specific network devices or services. Configuring the Pod to use the host network allows access to these resources.
Port Conflicts: If your Pod needs to bind to a port that's already in use on the host, you might configure hostNetwork to avoid port conflicts. This is because, by default, Pods have their own IP address and network namespace, which can lead to port collisions if the host is already using the desired port.
However, there are several important considerations and drawbacks to using hostNetwork:

Isolation: Using hostNetwork sacrifices network isolation, which is one of the key benefits of containerization and Kubernetes. It can lead to security and stability issues, as Pods can potentially interfere with the host's network configuration.
Network Conflicts: If multiple Pods are configured to use the host's network, there can still be conflicts, as they might contend for network resources or ports. Proper coordination and planning are necessary in such scenarios.
Compatibility: Not all Kubernetes clusters or cloud environments support the use of hostNetwork, so it may limit the portability of your workloads.
Security: Running containers in the host's network namespace can pose security risks, as containers have a level of isolation by design to prevent them from directly impacting the host system. Using hostNetwork reduces this isolation.
In summary, configuring hostNetwork in Kubernetes can be a useful tool in specific cases where performance and direct access to host network resources are crucial. However, it should be used judiciously, considering the trade-offs in terms of network isolation, compatibility, and security. It's generally recommended to explore alternative solutions, such as using hostPort, NodePort services, or network policies, before resorting to hostNetwork.
```

The only issue I see with introducing `hostNetwork` is, EG cannot guarantee the listener port will be available on the host causing erratic issues in the field that will be hard to debug
@arkodg: ptal @envoyproxy/gateway-maintainers@Xunzhuo: I think we should hold to merge the PR, until figure out the use case. @zzjin: First, this is useful for bare-metal envs with or without `MetaLB` support.
Then, I think we should give the choice to the user, to decide whether to use `hostnetwork`, with just some documentation and discouragements, but not simply ban it at the project code level.
@zirain: First of all, EG cannot provide every single feature users request.
If the API was promoted, that means EG need support and maitain.@docapotamus: Sorry if this isn't directly related to this issue but I have a use case for `hostPort` which to me is related to `hostNetwork`.

We use Cilium on top of an EKS cluster with an Network Load Balancer in front. We currently use ingress-nginx and we have to set the HostPort there so that the NLB can communicate in to the Cilium network.

Not sure if there if there is another way to acheive this, if there is I haven't found it or it's not obvious to me

Update: Setting externalTrafficPolicy solved the issue for us.@fanux: With hostNetwork, users can expose ports 80 and 443 with almost no other dependencies. These are the ports that the gateway must expose. This is not a use case issue, but a rigid demand. In many cases, users do not want to use the loadbalancer mode, such as not wanting to introduce more dependencies, such as on their own servers, or to quickly use the gateway function. I also do not recommend other containers to use hostNetwork, but the gateway is an exception. Without this feature, more than 50% of users would be lost.@miklezzzz: Hello! 
Some of our ingress-nginx installations assume using host networks (bare metal w/o a load balancer, for example) and I'm currently think through switching from ingress-nginx controller to envoy gateway.

For the beginning, I tried to use envoy proxy's patch field to apply `hostNetwork: true` to the resulting envoy demonset, yet both envoy proxy and shutdown-manager use the same 19002 port for health checking, resulting in a collide. It actually would be quite nice to be able to define custom ports for health checking (shouldn't cost a lot of efforts).

But then I though I could use an additional `hostPort` type port in the envoy daemonset to pass traffic directly to the listener port, like the following:
an example gateway spec
```
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    name: http
    port: 1024
    protocol: HTTP
```
and the related envoy proxy spec
```
spec:
  logging:
    level:
      default: debug
  provider:
    kubernetes:
      envoyDaemonSet:
        container:
          image: envoyproxy/envoy:v1.33-latest
          resources:
            limits:
              cpu: 500m
              memory: 1Gi
            requests:
              cpu: 150m
              memory: 250Mi
        patch:
          value:
            spec:
              template:
                spec:
                  containers:
                  - name: envoy
                    ports:
                    - containerPort: 1024
                      hostPort: 1024
                      name: http
                      protocol: TCP
```
and it seems to be working, I haven't thoroughly tested it yet though.

P.s. in case the listener uses a port greater than 1024, the hostPort's value must equal to 10000 + listener port.

Solution patch:
diff --git a/api/v1alpha1/shared_types.go b/api/v1alpha1/shared_types.go
index b3126c42f..7fa593ef1 100644
--- a/api/v1alpha1/shared_types.go
+++ b/api/v1alpha1/shared_types.go
@@ -113,6 +113,10 @@ type KubernetesPodSpec struct {
 	//
 	// +optional
 	Volumes []corev1.Volume `json:"volumes,omitempty"`
+
+	// HostNetwork, If this is set to true, the pod will use host's network namespace.
+	// +optional
+	HostNetwork bool `json:"hostNetwork,omitempty"`
 }
 
 // KubernetesContainerSpec defines the desired state of the Kubernetes container resource.
diff --git a/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml b/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml
index 7bff9e414..3228d86cc 100644
--- a/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml
+++ b/charts/gateway-helm/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml
@@ -3024,6 +3024,10 @@ spec:
                                   should be appended to the pods. By default, no pod
                                   annotations are appended.
                                 type: object
+                              hostNetwork:
+                                description: HostNetwork, If this is set to true,
+                                  the pod will use host's network namespace.
+                                type: boolean
                               labels:
                                 additionalProperties:
                                   type: string
diff --git a/internal/infrastructure/kubernetes/proxy/resource_provider.go b/internal/infrastructure/kubernetes/proxy/resource_provider.go
index 6ab47c0b3..20ce9d4bf 100644
--- a/internal/infrastructure/kubernetes/proxy/resource_provider.go
+++ b/internal/infrastructure/kubernetes/proxy/resource_provider.go
@@ -216,6 +216,7 @@ func (r *ResourceRender) Deployment() (*appsv1.Deployment, error) {
 					RestartPolicy:                 corev1.RestartPolicyAlways,
 					SchedulerName:                 "default-scheduler",
 					SecurityContext:               deploymentConfig.Pod.SecurityContext,
+					HostNetwork:                   deploymentConfig.Pod.HostNetwork,
 					Affinity:                      deploymentConfig.Pod.Affinity,
 					Tolerations:                   deploymentConfig.Pod.Tolerations,
 					Volumes:                       expectedDeploymentVolumes(r.infra.Name, deploymentConfig),
diff --git a/internal/infrastructure/kubernetes/ratelimit/resource_provider.go b/internal/infrastructure/kubernetes/ratelimit/resource_provider.go
index bb7f2ee59..f1309d322 100644
--- a/internal/infrastructure/kubernetes/ratelimit/resource_provider.go
+++ b/internal/infrastructure/kubernetes/ratelimit/resource_provider.go
@@ -178,6 +178,7 @@ func (r *ResourceRender) Deployment() (*appsv1.Deployment, error) {
 					RestartPolicy:                 corev1.RestartPolicyAlways,
 					SchedulerName:                 "default-scheduler",
 					SecurityContext:               r.rateLimitDeployment.Pod.SecurityContext,
+					HostNetwork:                   r.rateLimitDeployment.Pod.HostNetwork,
 					Volumes:                       expectedDeploymentVolumes(r.rateLimit, r.rateLimitDeployment),
 					Affinity:                      r.rateLimitDeployment.Pod.Affinity,
 					Tolerations:                   r.rateLimitDeployment.Pod.Tolerations,



Test patch:
diff --git a/internal/infrastructure/kubernetes/proxy/resource_provider_test.go b/internal/infrastructure/kubernetes/proxy/resource_provider_test.go
index f4603fc37..6d2d6a610 100644
--- a/internal/infrastructure/kubernetes/proxy/resource_provider_test.go
+++ b/internal/infrastructure/kubernetes/proxy/resource_provider_test.go
@@ -93,6 +93,7 @@ func TestDeployment(t *testing.T) {
 					SecurityContext: &corev1.PodSecurityContext{
 						RunAsUser: pointer.Int64(1000),
 					},
+					HostNetwork: true,
 				},
 				Container: &egv1a1.KubernetesContainerSpec{
 					Image: pointer.String("envoyproxy/envoy:v1.2.3"),
diff --git a/internal/infrastructure/kubernetes/proxy/testdata/deployments/custom.yaml b/internal/infrastructure/kubernetes/proxy/testdata/deployments/custom.yaml
index 03c4ec7b9..2a1fe7549 100644
--- a/internal/infrastructure/kubernetes/proxy/testdata/deployments/custom.yaml
+++ b/internal/infrastructure/kubernetes/proxy/testdata/deployments/custom.yaml
@@ -34,6 +34,7 @@ spec:
         prometheus.io/scrape: "true"
     spec:
       automountServiceAccountToken: false
+      hostNetwork: true
       containers:
         - args:
             - --service-cluster default
diff --git a/internal/infrastructure/kubernetes/ratelimit/resource_provider_test.go b/internal/infrastructure/kubernetes/ratelimit/resource_provider_test.go
index 02d8df725..ab4d6b65a 100644
--- a/internal/infrastructure/kubernetes/ratelimit/resource_provider_test.go
+++ b/internal/infrastructure/kubernetes/ratelimit/resource_provider_test.go
@@ -158,6 +158,7 @@ func TestDeployment(t *testing.T) {
 					SecurityContext: &corev1.PodSecurityContext{
 						RunAsUser: pointer.Int64(1000),
 					},
+					HostNetwork: true,
 				},
 				Container: &egv1a1.KubernetesContainerSpec{
 					Image: pointer.String("custom-image"),
diff --git a/internal/infrastructure/kubernetes/ratelimit/testdata/deployments/custom.yaml b/internal/infrastructure/kubernetes/ratelimit/testdata/deployments/custom.yaml
index b34ab0fe2..c922b53f5 100644
--- a/internal/infrastructure/kubernetes/ratelimit/testdata/deployments/custom.yaml
+++ b/internal/infrastructure/kubernetes/ratelimit/testdata/deployments/custom.yaml
@@ -31,6 +31,7 @@ spec:
         prometheus.io/scrape: "true"
     spec:
       automountServiceAccountToken: false
+      hostNetwork: true
       containers:
       - command:
         - /bin/ratelimit
diff --git a/site/content/en/latest/api/extension_types.md b/site/content/en/latest/api/extension_types.md
index 650863ef0..62789bcf9 100644
--- a/site/content/en/latest/api/extension_types.md
+++ b/site/content/en/latest/api/extension_types.md
@@ -1011,6 +1011,7 @@ _Appears in:_
 | `affinity` _[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#affinity-v1-core)_ | If specified, the pod's scheduling constraints. |
 | `tolerations` _[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#toleration-v1-core) array_ | If specified, the pod's tolerations. |
 | `volumes` _[Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#volume-v1-core) array_ | Volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes |
+| `hostNetwork` _boolean_ | HostNetwork, If this is set to true, the pod will use host's network namespace. |
 
 
 #### KubernetesServiceSpec