Project: OpenYurt (Provisioning / Automation & Configuration)
File: 20230613-raven-l7-proxy.md

title: support raven-l7-proxy authors: "@River-sh" reviewers: "" creation-date: last-updated: status: - - - - - - - - - - - In the edge scenario, the cloud and the edge are usually on different network domain, so the host and container networks across the network domain are not interconnected. raven-l3 can solve cross-domain communication in the absence of IP address conflict, raven still needs to be enhanced to support cross-domain access to host services in the case of IP conflict. This proposal aims to enhance raven network capabilities and replace yurt-tunnel. Enhance raven network capability to support cross-domain access to host services in the case of IP conflict Enhanced Raven networking to replace Yurt-Tunnel, we want to achieve the following goals: Implement 7 layer proxy in the Raven project Maintain uniform architecture and design with Raven L3 Do not rely too heavily on the control plane Re-implement the reverse proxy tunnel packages into Raven repo instead of using ANP <img src = "../img/networking/img-4.png" width="800"> RavenAgent is deployed on each node of the cluster in DaemonSet mode, and the container is deployed in host network mode The container network within the network domain can communicate with each other, and cross-domain requests are forwarded to the Gateway node for forwarding RavenAgent on each node determines the routing configuration based on the CR configuration information of Gateway Raven Agent on the Gateway node establishes a VPN channel with the Gateway exposed on the public network. <img src = "../img/networking/img-5.png" width="800"> RavenAgent is deployed on each node of the cluster in DaemonSet mode, and the container is deployed in host network mode The container network within the network domain can communicate with each other, and cross-domain requests are forwarded to the gateway node for forwarding RavenAgent will start service according to its identity: ProxyClient will enable if it is not exposed to the public, ProxyServer will be started, if it is exposed to the public network ProxyClient obtains the address of the exposed ProxyServer through the Gateway CR and actively establishes link Requests across network domains are forwarded by ProxyServer proxy to ProxyClient in other network domains. The Endpoint property adds the port Spec add attributes: ProxyConfig and TunnelConfig ExposeType support two mode: LoadBalancerPublicIP Support elect multi active endpoints for l7 proxy Proxy server need listen port 10262, tunnel server need listen port 4500 ```go // Gateway is the Schema for the gateways API type Gateway struct { metav1.TypeMeta `json:",inline"`

`json:"metadata,omitempty"` Spec GatewaySpec `json:"spec,omitempty"` Status GatewayStatus `json:"status,omitempty"` } // ProxyConfiguration is the configuration for raven l7 proxy type ProxyConfiguration struct { // Replicas is the number of gateway active endpoints that enabled proxy Replicas int `json:"Replicas,omitempty"` // ProxyHTTPPort is the proxy http port of the cross-domain request ProxyHTTPPort string `json:"proxyHTTPPort,omitempty"` // ProxyHTTPSPort is the proxy https port of the cross-domain request ProxyHTTPSPort string `json:"proxyHTTPSPort,omitempty"` // ProxyLocalHostPort is the proxy localhost port of the cross-domain request ProxyLocalHostPort string `json:"proxyLocalHostPort,omitempty"` // ProxyServerPort is the proxy service port of the exposed gateway ProxyServerPort string `json:"proxyServerPort,omitempty"` } // TunnelConfiguration is the configuration for raven l3 tunnel type TunnelConfiguration struct { // Replicas is the number of gateway active endpoints that enabled tunnel Replicas int `json:"Replicas,omitempty"` // VPNServerPort is the tunnel service port of the exposed gateway VPNServerPort string `json:"VPNServerPort,omitempty"` } // GatewaySpec defines the desired state of Gateway type GatewaySpec struct { // NodeSelector is a label query over nodes that managed by the gateway. // The nodes in the same gateway should share same layer 3 network. NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty"` // ProxyConfig determine the l7 proxy configuration ProxyConfig ProxyConfiguration `json:"proxyConfig,omitempty"` // TunnelConfig determine the l3 tunnel configuration TunnelConfig TunnelConfiguration `json:"tunnelConfig,omitempty"` // Endpoints are a list of available Endpoint. Endpoints []Endpoint `json:"endpoints,omitempty"` // ExposeType determines how the Gateway is exposed. ExposeType string `json:"exposeType,omitempty"` } // Endpoint stores all essential data for establishing the VPN tunnel and Proxy type Endpoint struct { // NodeName is the Node hosting this endpoint. NodeName string `json:"nodeName"` // Type is the service type of the node, proxy or tunnel Type string `json:"type"` Port string `json:"port,omitempty"` PublicIP string `json:"publicIP,omitempty"` Config map[string]string `json:"config,omitempty"` UnderNAT bool `json:"underNAT,omitempty"` } // NodeInfo stores information of node managed by Gateway. type NodeInfo struct { NodeName string `json:"nodeName"` PrivateIP string `json:"privateIP"` Subnets []string `json:"subnets"` } // GatewayStatus defines the observed state of Gateway type GatewayStatus struct { // Nodes contains all information of nodes managed by Gateway. Nodes []NodeInfo `json:"nodes,omitempty"` // ActiveEndpoints is the reference of the active endpoint. ActiveEndpoints []*Endpoint `json:"activeEndpoints,omitempty"` } ``` Global Config ```yaml apiVersion: v1 kind: ConfigMap metadata: name: raven-cfg namespace: kube-system labels: app: raven data: enable-l7-proxy: true enable-l3-tunnel: false ``` Select nodes to provide tunnel or proxy service based on Spec.Endpoints Update the configuration of each ActiveEndpoint according to the global configmap raven-cfg, Spec.ProxyConfig, and Spec.TunnelConfig Traverse the nodes in the local network domain and update the node information to the Status of the Gateway If Gateway Spec.ExposedType == LoadBalancer and L3 Tunnel is enabled, manage an LB Service: x-raven-tunnel-svc -${GatewayName}$ If Gateway Spec.ExposedType == LoadBalancer and L7 Proxy is enabled, manage an LB Service: x-raven-proxy-svc -${GatewayName}$ Create/Update/Delete Endpoints by Gateway.Status Each service is labeled

Manage a Cluster Service : x-raven-internal-svc for all Gateway that Spec.ExposedType != "" Create/Update/Delete Endpoints for x-raven-internal-svc <img src = "../img/networking/img-7.png" width="800"> Update the configmap edge-tunnel-nodes to record dns for coredns All requests that using NodeName+Port are forwarded to x-raven-internal-svc A certificate is generated for each ProxyServer and approved by the csrapprover A certificate is generated for each ProxyClient and approved by the csrapprover A certificate is generated for each ProxyServer to establishes a tls link with kubelet and approved by csrapprover <img src = "../img/networking/img-6.png" width="800"> ProxyClient obtains the public IP address of the ProxyServer and initiates a gRPC link request through the watch gateway ProxyServer receive requests and registers these ProxyClients as backend The Http request that across the network domain will be hijacked by the Interceptor and the header will be modified The Interceptor establishes a socket link with the Proxy and forwards the request to the Proxy The Proxy sends a dial request to the destination services in other network domains to establish tunnels In this case, the cross-domain tunnel is smooth, start data transmission <img src = "../img/networking/img-5.png" width="800"> Gateway,DNS,CrossDomainProxyService,IntraDomainProxyService and webhook are unified management by yurt-manager Raven-cfg and raven-agent are created when raven agent is deployed Each node should be labeled ```raven.openyurt.io/gateway=${GatewayName}``` to divide network domains and create a gateway CR for each network domain. We expect each node pool to be a network domain, but currently there is no mandatory limit for node pools Gateway controller elect gateway node and update gateway status DNS Controller maintains the configmap edge-tunnel-nodes for coredns CrossDomainProxyService and IntraDomainProxyService manage service and endpoints Raven agent watch gateway and identify themselves to enable service The Raven Agent on the Gateway node exposed to the public network starts the ProxyServer The Raven Agent on the Gateway node not exposed to the public network starts the ProxyClient The Raven Agent on the node that is not belong to any gateway or has an exclusive Gateway starts the ProxyClient Cert manager request three certificates Re-implement a reverse tunnel scheme as a Raven project package to replace ANP Optimize network links and remove Interceptor Support raven l7 proxy for edge-edge commutation As an end user, I want to make some DevOps from Cloud to Edge, such as kubectl logs/exec. As an end user, I want to get the edge nodes metrics status through Prometheus/Metrics server from Cloud. As an end user, I want to access another business pod data from one NodePool to another NodePool. [ ] 06/12/2023: Draft proposal created [ ] 06/14/2023: Present proposal at the community meeting [ ] 07/4/2023: Update proposal