Project: KubeEdge (Provisioning / Automation & Configuration)
File: edgemesh-design.md

title: EdgeMesh Design status: implementable authors: "@qizha" "@sids-b" approvers: "@kevin-wangzefeng" "@CindyXing" creation-date: 2019-03-20 last-updated: 2019-03-27 To support service mesh capabilities on edge to support microservice communication cross cloud and edge. The service discovery, communication should be considered in this feature. Developers can define services on top of the deployments deployed on the cloud or edges in a unified manner without caring about their location. The service name is the only thing they need to know to do the communication. For example, the following address can be used for a REST request: `http://{service_name}/resources` The cloud native and microservice architecture is becoming more and more popular and the edge node is becoming more and more powerful. The users want to decompose their application into multiple microservices. Some of those microservices need to be deployed in edge node to get close to the data generated by devices. By simplifying the development, the users need to use the same mechanism to do the service discovery and communication with what they are doing on cloud. No DNS services available for edge. The identification and authentication should be handled by microservices. When service communication between edge to edge, they may work offline. With the purpose to deliver an "expected result" to the user system, the service definition should be pushed down to the edge right after the definition operation. That is to say, the istio model may not be suitable for edge, in istio model, the service definition is pulled from K8S master when the service is invoked. Only support HTTP communication in the first step All edge nodes are in the same subnetwork, and they have the network reachability. <img src="../images/proposals/service-design.png"> CS0.1: User create service(type ClusterIP) in K8S API Server CS0.2: ServiceController get the service definition and create router rules with the edge ServiceBus as source and service clusterIP as target. CS0.3: Send the service definition to edge CS1: User create service(headless service without ClusterIP) in K8S API Server CS2, CS3: ServiceController get the service definition and update the service ClusterIP to itself. Then create router rule with the EC(Router) address as the source and edge ServiceBus as target. CS4: Send the service definition to edge Application access service with the same mechanism no matter it located on cloud or edge Use iptables to redirect inbound and outbound traffic to the EdgeHub/ServiceBus. Then do service discovery in EdgeHub/ServiceBus by pick up the right POD from the service definition. The idea is similar with what istio is doing in the init_container: https://github.com/istio/istio/blob/master/tools/packaging/common/istio-iptables.sh ESD1: Application on cloud visit service with clusterIP and request goes to EC(refer to step CS2, CS3) ESD2: Router find out the PODs in this service and pick one with some kind of load balance mechanism. Then send the request to the chosen edge with the router rule set in CS3. ESD3: EdgeHub/ServiceBus get the request and redirect to the POD ESR1: Application send the request to ServiceBus(refer to the "Service discovery on edge") ESR2: ServiceBus pick one POD in this service with some kind of load balance mechanism. If the target is in the same node, just forward the request. If not, edgehub wrapper the request and send to the

ESR3: EdgeHub open the wrapped package and forward to the POD ESU1: Application send the request to ServiceBus(refer to the "Service discovery on edge") ESU2: ServiceBus pick one POD in this service with some kind of load balance mechanism. The target is on cloud, EdgeHub send the request to cloud. ESU3: Router send the request to the corresponding POD on cloud following the rule created in CS0.2 ServiceBus is in the same process with EdgeHub as a module Add two endpoints definition: | Endpoint | Type | Capability | ||--|--| | ClusterIP | Target | Request Forward | | ServiceBus | Source | Request Forward, Service Discovery, Load Balance | | ServiceBus | Target | Request Forward | | Router | Source | Request Forward, Service Discovery, Load Balance | <img src="../images/proposals/router-design.png"> R1: ServiceController(SC) watches on k8s apiserver for Services/Endpoints. R2: SC get the service definition and create router rule definition and save in DB(ETCD). R3: Router gets the rule definition from DB and initializes the listeners/handlers for source and target. R4: Users/Applications create a request. R5: Router receives the request and forwards it to the target based on rule. Since Router fetches rules from DB, in later versions it can be started as a different process than edge-controller. Also, it enables router to retrieve all the rules after process restarts. Providers are plugins to reach a service running on edge or on cloud. For example (ServiceBus, EventBus running in edgecore) Providers can be classified into two types 1) Source 2) Target In Alpha only ServiceBus, REST and EventBus providers will be supported. Providers in router work in following way. <img src="../images/proposals/router-flow.png"> 1) Router rule is added by SC say REST as source and ServiceBus at edge as target. 2) Router receives the rule and registers a listener in the source which is REST in this case. 3) REST provider starts listening on the path mentioned in the SourceResource of the rule. for eg: http://${restendpoint}:${restendpointport}/${nodeid}${path of rule.source_resource} 4) 4.1 User/App make a request to the endpoint in 3. 4.2 REST provider forwards the request to ServiceBus provider. 5) 5.1 ServiceBus provider then creates a beehive message and sends it to CloudHub. 5.2 CloudHub then forwards the message to edgeHub. 5.3 EdgeHub send the message to ServiceBus module. 6) 6.1 ServiceBus at edge receives the message, makes the http request to applications running at edge. 6.2 Receives the response back from the application. 7) 7.1 ServiceBus at Edge sends the response in beehive message back to EdgeHub. 7.2 EdgeHub forwards the message to CloudHub. 7.3 CloudHub sends the message back to router. 7.4 The message is then converted to http response and sent to the user/application which made the request in step 4.1 Each provider that wants to register as a Source should implement following interfaces. ```go type Source interface { // Name Returns name of the provider Name() string // RegisterListener is used to register listener for a rule with this source type RegisterListener(rule model.Rule, res model.Map, handler func(interface{})) error // UnregisterListener is used to unregister a listener when rule is deleted UnregisterListener(rule

res map[string]interface{}) // Callback is function for sending response/ACK if required for a request Callback(map[string]interface{}) } ``` ```go type SourceFactory interface { // Type returns type of the source Type() string // GetSource returns Source implementation GetSource(ep *model.Endpoint) Source } ``` Each provider that wants to register as a Target should implement following interfaces. ```go type Target interface { // Name returns name of the provider Name() string // Forward is used to forward the request to target Forward(map[string]interface{}, interface{}, func(map[string]interface{})) error } ``` ```go type TargetFactory interface { // Type returns type of the target Type() string // GetTarget returns Target implementation GetTarget(ep *model.Endpoint) Target } ``` ```go // Rule define message translate from Endpoint to Endpoint type Rule struct { // ID is unique id of rule ID string `orm:"column(id); size(36); pk" json:"id" yaml:"id"` // Name is the name of the Rule Name string `orm:"column(name); size(64)" json:"name" yaml:"name"` // ProjectID can be used to differentiate rules across projects ProjectID string `orm:"column(projectid); size(36); index" json:"projectid" yaml:"project_id"` // DomainID is the domain from where the source is accessed DomainID string `orm:"column(domainid); size(36); index" json:"domainid" yaml:"domain_id"` // Source is the source Endpoint Source *Endpoint `orm:"column(source); rel(fk)" json:"source" yaml:"source"` // SourceResource is a map used to store resources required for source SourceResource Map `orm:"column(sourceresource); size(1024)" json:"sourceresource" yaml:"source_resource"` // Target is target endpoint Target *Endpoint `orm:"column(target); rel(fk)" json:"target" yaml:"target"` // TargetResource is a map used to store resources required for target TargetResource Map `orm:"column(targetresource); size(1024)" json:"targetresource" yaml:"target_resource"` // InUse is flag to check if rule is in use InUse bool `orm:"column(inuse); default(true)" json:"inuse" yaml:"in_use"` // IsDeleted is flag to check if rule is deleted IsDeleted bool `orm:"column(isdeleted); default(false)" json:"isdeleted" ` // CreatedAt is rule creation timestamp CreatedAt time.Time `orm:"column(createdat); autonowadd; type(datetime)" json:"createdat" ` // UpdatedAt is last rule updation timestamp UpdatedAt time.Time `orm:"column(updatedat); autonow; type(datetime)" json:"updated_at" ` // SuccessMessages is the number of success messages for this rule SuccessMessages uint64 `orm:"column(successmessages); default(0)" json:"successmessages" ` // FailMessages in the number of fail messages for this rule FailMessages uint64 `orm:"column(failmessages); default(0)" json:"failmessages" ` // Error is reverse maping of error messages for this rule Error []*Error `orm:"reverse(many)" json:"-"` } ``` ```go // Endpoint define how to access external service type Endpoint struct { // ID is the ID of the endpoint ID string `orm:"column(id); size(36); pk" json:"id" yaml:"id"` // Type is type of the endpoint Type string `orm:"column(type); size(64)" json:"type" yaml:"type"` // Name is name of the endpoint Name string `orm:"column(name); size(64)" json:"name" yaml:"name"` // ProjectID can be used to differentiate endpoints across projects ProjectID string `orm:"column(projectid); size(36); index; null" json:"projectid" yaml:"project_id"` // IsShared is flag to check if endpoint is shared IsShared bool `orm:"column(isshared); default(false)" json:"isshared" ` // URI is unique resource identifier of the endpoint URI string `orm:"column(uri); size(256)" json:"uri" yaml:"uri"` // Properties is map used to store properties of an Endpoint Properties Map `orm:"column(properties); size(10240)" json:"properties" yaml:"properties"` // CreatedAt is endpoint creation timestamp CreatedAt time.Time `orm:"column(createdat); autonowadd; type(datetime)" json:"createdat" ` // UpdatedAt is last endpoint updation timestamp UpdatedAt time.Time `orm:"column(updatedat); autonow; type(datetime)" json:"updated_at" ` // Rule is reverse mapping of Rules of this endpoint Rule []*Rule `orm:"reverse(many)" json:"-"` } ``` Listen to the edge communication Package the message and send to the target Listen the service definition and create routing rule Listen the service definition to PODs deployed on edge and update the ClusterIP