Project: Antrea (Runtime / Cloud Native Network)
File: noencap-hybrid-modes.md

Besides the default `Encap` mode, in which Pod traffic across Nodes will be encapsulated and sent over tunnels, Antrea also supports `NoEncap` and `Hybrid` traffic modes. In `NoEncap` mode, Antrea does not encapsulate Pod traffic, but relies on the Node network to route the traffic across Nodes. In `Hybrid` mode, Antrea encapsulates Pod traffic when the source Node and the destination Node are in different subnets, but does not encapsulate when the source and the destination Nodes are in the same subnet. This document describes how to configure Antrea with the `NoEncap` and `Hybrid` modes. The NoEncap and Hybrid traffic modes require AntreaProxy to support correct NetworkPolicy enforcement, which is why trying to disable AntreaProxy in these modes will normally cause the Antrea Agent to fail. It is possible to override this behavior and force AntreaProxy to be disabled by setting the ALLOWNOENCAPWITHOUTANTREA_PROXY environment variable to true for the Antrea Agent in the . For example: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: antrea-agent labels: component: antrea-agent spec: template: spec: containers: name: antrea-agent env: name: ALLOWNOENCAPWITHOUTANTREA_PROXY value: "true" ``` Let us start from `Hybrid` mode which is simpler to configure. `Hybrid` mode does not encapsulate Pod traffic when the source and the destination Nodes are in the same subnet. Thus it requires the Node network to allow Pod IP addresses to be sent out from the Nodes' NICs. This requirement is not supported in all the networks and clouds, or in some cases it might require specific configuration of the Node network. For example: On AWS, the source/destination checks must be disabled on the EC2 instances of the Kubernetes Nodes, as described in the . On Google Compute Engine, IP forwarding must be enabled on the VM instances as described in the . On Azure, there is no way to let VNet forward unknown IPs, hence Antrea `Hybrid` mode cannot work on Azure. If the Node network does allow Pod IPs sent out from the Nodes, you can configure Antrea to run in the `Hybrid` mode by setting the `trafficEncapMode` config parameter of `antrea-agent` to `hybrid`. The `trafficEncapMode` config parameter is defined in `antrea-agent.conf` of the `antrea` ConfigMap in the . ```yaml antrea-agent.conf: | trafficEncapMode: hybrid ``` After changing the config parameter, you can deploy Antrea in `Hybrid` mode with the usual command: ```bash kubectl apply -f antrea.yml ``` In `NoEncap` mode, Antrea never encapsulates Pod traffic. Just like `Hybrid` mode, the Node network needs to allow Pod IP addresses sent out from Nodes. When the Nodes are not in the same subnet, `NoEncap` mode additionally requires the Node network be able to route the Pod traffic from the source Node to the destination

There are two possibilities to enable this routing by Node network: Leverage Route Controller of . The Kubernetes Cloud Providers that implement Route Controller can add routes to the cloud network routers for the Pod CIDRs of Nodes, and then the cloud network is able to route Pod traffic between Nodes. This Route Controller functionality is supported by the Cloud Provider implementations of the major clouds, including: , , , and . Run a routing protocol or even manually configure routers to add routes to the Node network routers. For example, Antrea can work with and leverage kube-router to advertise Pod CIDRs to routers using BGP. Section describes how to configure Antrea and kube-router to work together. When the Node network can support forwarding and routing of Pod traffic, Antrea can be configured to run in the `NoEncap` mode, by setting the `trafficEncapMode` config parameter of `antrea-agent` to `noEncap`. By default, Antrea performs SNAT (source network address translation) for the outbound connections from a Pod to outside of the Pod network, using the Node's IP address as the SNAT IP. In the `NoEncap` mode, as the Node network knows about Pod IP addresses, the SNAT by Antrea might be unnecessary. In this case, you can disable it by setting the `noSNAT` config parameter to `true`. The `trafficEncapMode` and `noSNAT` config parameters are defined in `antrea-agent.conf` of the `antrea` ConfigMap in the . ```yaml kind: ConfigMap apiVersion: v1 metadata: name: antrea-config namespace: kube-system data: antrea-agent.conf: | trafficEncapMode: noEncap noSNAT: false # Set to true to disable Antrea SNAT for external traffic ``` After changing the parameters, you can deploy Antrea in `noEncap` mode by applying the deployment yaml. We can run kube-router in advertisement-only mode to advertise Pod CIDRs to the peered routers, so the routers can know how to route Pod traffic to the Nodes. To deploy kube-router in advertisement-only mode, first download the : ```bash curl -LO https://raw.githubusercontent.com/cloudnativelabs/kube-router/v0.4.0/daemonset/generic-kuberouter-only-advertise-routes.yaml ``` Then edit the yaml file and set the following kube-router arguments: ```yaml "--run-router=true" "--run-firewall=false" "--run-service-proxy=false" "--enable-cni=false" "--enable-ibgp=false" "--enable-overlay=false" "--enable-pod-egress=false" "--peer-router-ips=<CHANGE ME>" "--peer-router-asns=<CHANGE ME>" ``` The BGP peers should be configured by specifying the `--peer-router-asns` and `--peer-router-ips` parameters. Note, the ASNs and IPs must match the configuration on the peered routers. For example: ```yaml "--peer-router-ips=192.168.1.99,192.168.1.100 "--peer-router-asns=65000,65000" ``` Then you can deploy the kube-router DaemonSet with: ```bash kubectl apply -f generic-kuberouter-only-advertise-routes.yaml ``` You can verify that the kube-router Pods are running on the Nodes of your Kubernetes cluster by (the cluster in the following example has only two Nodes): ```bash $ kubectl -n kube-system get pods -l k8s-app=kube-router NAME READY STATUS RESTARTS AGE kube-router-rn4xc 1/1 Running 0 1m kube-router-vhrf5 1/1 Running 0 1m ``` Antrea can be deployed either before or after kube-router, with the `NoEncap` mode.