Project: DeepFlow (Observability and Analysis / Observability)
File: data-flow.md

```mermaid flowchart LR subgraph deepflow-agent Dispatcher EbpfCollector IntegrationCollector XXX StatsdClient agent.queue.1(["queue"]) agent.queue.2(["queue"]) agent.queue.3(["queue"]) agent.queue.4(["queue"]) agent.queue.5(["queue"]) agent.queue.6(["queue"]) agent.queue.7(["queue"]) agent.queue.8(["queue"]) agent.queue.9(["queue"]) UniformSender.1["UniformSender"] UniformSender.2["UniformSender"] UniformSender.3["UniformSender"] UniformSender.4["UniformSender"] UniformSender.5["UniformSender"] UniformSender.6["UniformSender"] UniformSender.7["UniformSender"] UniformSender.8["UniformSender"] UniformSender.9["UniformSender"] end subgraph deepflow-server.ingester flow_metrics.decoder flowlog.decoder.1["flowlog.decoder"] flowlog.decoder.2["flowlog.decoder"] flowlog.decoder.3["flowlog.decoder"] flowlog.decoder.4["flowlog.decoder"] extmetrics.decoder.1["extmetrics.decoder"] extmetrics.decoder.2["extmetrics.decoder"] extmetrics.decoder.3["extmetrics.decoder"] event.decoder["event.decoder"] profile.decoder["profile.decoder"] ingester.queue.1(["queue"]) ingester.queue.2(["queue"]) ingester.queue.3(["queue"]) ingester.queue.4(["queue"]) ingester.queue.5(["queue"]) flow_metrics.dbwriter flow_log.dbwriter ext_metrics.dbwriter event.dbwriter profile.dbwriter end Kernel -->|cBPF/AF_PACKET| Dispatcher Dispatcher --> agent.queue.1 -->|Metrics| UniformSender.1 -->|"tcp(pb)"| flowmetrics.decoder -->|Document| ingester.queue.1 --> flowmetrics.dbwriter Dispatcher --> agent.queue.2 -->|L4FlowLog| UniformSender.2 -->|"tcp(pb)"| flowlog.decoder.1 -->|L4FlowLog| ingester.queue.2 --> flowlog.dbwriter Dispatcher --> agent.queue.3 -->|L7FlowLog| UniformSender.3 -->|"tcp(pb)"| flow_log.decoder.2 -->|L7FlowLog| ingester.queue.2 Kernel -->|eBPF| EbpfCollector EbpfCollector --> agent.queue.1 EbpfCollector --> agent.queue.3 EbpfCollector --> agent.queue.8 -->|ProcEvent| UniformSender.8 -->|"tcp(pb)"| event.decoder -->|EventStore| ingester.queue.4 --> event.dbwriter EbpfCollector --> agent.queue.9 otel-collector -->|OTLP| IntegrationCollector otel-javaagent/sdk -->|OTLP| IntegrationCollector IntegrationCollector -->|OTLP| agent.queue.1 IntegrationCollector --> agent.queue.4 -->|"zip(OTLP)"| UniformSender.4 -->|"tcp(zip(OTLP))"| flow_log.decoder.3 -->|L7FlowLog| ingester.queue.2 IntegrationCollector --> agent.queue.5 -->|OTLP| UniformSender.5 -->|"tcp(OTLP)"| flow_log.decoder.4 -->|L7FlowLog| ingester.queue.2 prometheus-server -->|prom-pb| IntegrationCollector telegraf -->|influxdb| IntegrationCollector pyroscope-remote-write -->|http| IntegrationCollector IntegrationCollector --> agent.queue.6 -->|prom-pb| UniformSender.6 -->|"tcp(prom-pb)"| extmetrics.decoder.1 -->|ExtMetrics| ingester.queue.3 --> extmetrics.dbwriter IntegrationCollector --> agent.queue.7 -->|influxdb| UniformSender.7 -->|"tcp(influxdb)"| ext_metrics.decoder.2 -->|ExtMetrics| ingester.queue.3 IntegrationCollector --> agent.queue.9 -->|profile-pb| UniformSender.9 -->|"tcp(pb)"| profile.decoder -->|InProcessProfile| ingester.queue.5 --> profile.dbwriter XXX -->|XXXCounter| StatsdClient -->|"tcp(pb)"| ext_metrics.decoder.3 -->|ExtMetrics| ingester.queue.3 flowmetrics.dbwriter -->|flowmetrics| ClickHouse flowlog.dbwriter -->|flowlog| ClickHouse extmetrics.dbwriter -->|extmetrics| ClickHouse event.dbwriter -->|perf_event| ClickHouse profile.dbwriter -->|profile| ClickHouse ``` ```mermaid flowchart LR Dispatcher -->|MetaPacket| FlowGenerator FlowGenerator -->|MetaPacket| FlowMap[(FlowMap)] FlowGenerator -->|"TaggedFlow (1s)"| queue.1([queue]) queue.1 --> QuadrupleGenerator QuadrupleGenerator --> SubQuadGen.1["SubQuadGen (1s)"] SubQuadGen.1 -->|"AccumulatedFlow (1s)"| QuadrupleStash.1[("QuadrupleStash (1s)")] SubQuadGen.1 -->|QuadrupleConnections| ConcurrentConnection.1[("ConcurrentConnection (1s)")] SubQuadGen.1 -->|"AccumulatedFlow (1s)"| queue.2([queue]) --> Collector.1[Collector] -->|"Metrics(Document)"| queue.3([queue]) --> UniformSender.1[UniformSender] QuadrupleGenerator --> SubQuadGen.2["SubQuadGen (1m)"] SubQuadGen.2 -->|"AccumulatedFlow (1m)"| QuadrupleStash.2[("QuadrupleStash (1m)")] SubQuadGen.2 -->|QuadrupleConnections| ConcurrentConnection.2[("ConcurrentConnection (1m)")] SubQuadGen.2 -->|"AccumulatedFlow (1m)"| queue.4([queue]) --> Collector.2[Collector] -->|"Metrics(Document)"| queue.5([queue]) --> UniformSender.1 QuadrupleGenerator --> queue.6([queue]) --> FlowAggr -->|"TaggedFlow (1m)"| throttler -->|"L4FlowLog(TaggedFlow)"| queue.7([queue]) --> UniformSender.2[UniformSender] queue.1 --> L7QuadrupleGenerator L7QuadrupleGenerator --> SubQuadGen.1["SubQuadGen (1s)"] SubQuadGen.1 -->|"AppMeterWithFlow (1s)"| queue.2([queue]) L7QuadrupleGenerator --> SubQuadGen.2["SubQuadGen (1m)"] SubQuadGen.2 -->|"AppMeterWithFlow (1m)"| queue.4([queue]) FlowGenerator -->|MetaAppProto| queue.8([queue]) --> AppProtoLogsParser -->|AppProtoLogsData| throttler.1[throttler] -->|"L7FlowLog(AppProtoLogsData)"| queue.9([queue]) --> UniformSender.3[UniformSender] EbpfCollector -->|MetaPacket| queue.10([queue]) --> EbpfRunner -->|AppProtoLogsData| SessionAggr --> throttler.2[throttler] -->|"L7FlowLog(AppProtoLogsData)"| queue.9 EbpfCollector -->|stackprofiledata| ebpfoncpu_callback -->|"Profile"| queue.9 ``` ```mermaid flowchart TD subgraph ingester.flow_metrics direction LR flowmetrics.receiver --> flowmetrics.queue([queue]) --> flow_metrics.decoder end subgraph ingester.flow_log direction LR flowlog.receiver --> flowlog.([queue]) --> flow_log.decoder --> throttler end subgraph ingester.ext_metrics direction LR extmetrics.receiver --> extmetrics.([queue]) --> ext_metrics.decoder end ``` ```mermaid flowchart TD subgraph K8s.Node.1 APIServer[(k8s-apiserver)] PodMAC.1[(PodMAC)] NodeMAC.1[(NodeMAC)] subgraph deepflow-agent.k8s.1 subgraph PlatformSynchronizer.1[PlatformSynchronizer] ActivePoller.1[ActivePoller] PassivePoller.1[PassivePoller] end subgraph ApiWatcher ResourceWatcher end end end subgraph K8s.Node.2 PodMAC.2[(PodMAC)] NodeMAC.2[(NodeMAC)] subgraph deepflow-agent.k8s.2 subgraph PlatformSynchronizer.2[PlatformSynchronizer] ActivePoller.2[ActivePoller] PassivePoller.2[PassivePoller] end end end subgraph Host.1 HostInfo.1[(HostInfo)] subgraph deepflow-agent.host.1 PlatformSynchronizer.3[PlatformSynchronizer] end end subgraph Host.2 HostInfo.2[(HostInfo)] subgraph deepflow-agent.host.2 PlatformSynchronizer.4[PlatformSynchronizer] end end subgraph deepflow-server.1 controller.genesis.1[controller.genesis] controller.cloud.1[controller.cloud] controller.recorder.1[controller.recorder] queue.1(queueu) ingester.event.1[ingester.event] controller.tagrecoreder controller.genesis.1 -->|"k8s & host meta"| controller.cloud.1 -->|meta| controller.recorder.1 -->|event| queue.1 --> ingester.event.1 end subgraph deepflow-server.2 controller.genesis.2[controller.genesis] controller.cloud.2[controller.cloud] controller.recorder.2[controller.recorder] queue.2(queueu) ingester.event.2[ingester.event] controller.genesis.2 -->|"k8s & host meta"| controller.cloud.2 -->|meta| controller.recorder.2 -->|event| queue.2 --> ingester.event.2 end APIServer -->|"list, watch"| ResourceWatcher ApiWatcher -->|gRPC.KubernetesAPISync| controller.genesis.1 NodeMAC.1 -->|"ip"| PlatformSynchronizer.1 -->|gRPC.GenesisSync| controller.genesis.1 PodMAC.1 -->|"setns, ip (w/ SYS_ADMIN)"| ActivePoller.1 PodMAC.1 -->|"AFPACKET (w/o SYSADMIN)"| PassivePoller.1 NodeMAC.2 -->|"ip"| PlatformSynchronizer.2 -->|gRPC.GenesisSync| controller.genesis.2 PodMAC.2 -->|"setns, ip (w/ SYS_ADMIN)"| ActivePoller.2 PodMAC.2 -->|"AFPACKET (w/o SYSADMIN)"| PassivePoller.2 HostInfo.1 -->|"hostname, ip"| PlatformSynchronizer.3 -->|gRPC.GenesisSync| controller.genesis.1 HostInfo.2 -->|"hostname, ip"| PlatformSynchronizer.4 -->|gRPC.GenesisSync| controller.genesis.2 CloudAPI[(CloudAPI)] -->|cloud meta| controller.cloud.2 controller.genesis.1 <-->|"exchange & merge"| controller.genesis.2 controller.recorder.1 -->|meta| MySQL.Meta[(MySQL Meta Tables)] controller.recorder.2 -->|meta| MySQL.Meta MySQL.Meta -->|meta|

-->|tag| MySQL.Tag[(MySQL Tag Tables)] MySQL.Tag -.->|sync| ClickHouse.1[(ClickHouse.1)] MySQL.Tag -.->|sync| ClickHouse.2[(ClickHouse.2)] ingester.event.1 -->|event| ClickHouse.1 ingester.event.2 -->|event| ClickHouse.2 ``` ```mermaid sequenceDiagram participant apiserver participant agent participant server.controller.trisolaris participant server.controller.genesis participant server.controller.cloud participant server.controller.recorder participant mysql Note right of agent: 1. get kubernetes-cluster-id activate agent agent ->> agent: load k8s-cluster-id from agent.yaml opt k8s-cluster-id is None agent ->> server.controller.trisolaris: gRPC.GetKubernetesClusterID(ca-md5) server.controller.trisolaris ->> server.controller.trisolaris: lookup k8s-cluster-id by ca-md5 in cache opt k8s-cluster-id is None server.controller.trisolaris ->> mysql: lookup k8s-cluster-id by ca-md5, if it does not exist, generate it. end server.controller.trisolaris ->> agent: k8s-cluster-id end deactivate agent Note right of agent: 2. sync K8s resources activate agent agent ->> server.controller.trisolaris: gRPC.Sync(k8s-cluster-id, node-ip, node-mac) server.controller.trisolaris ->> server.controller.trisolaris: Elect one K8s watcher agent for each K8s cluster alt is the K8s watcher server.controller.trisolaris ->> agent: kubernetesapienabled=true agent ->> apiserver: watch resource agent ->> server.controller.genesis: gRPC.KubernetesAPISync(k8s-cluster-id, k8s-resource-list) server.controller.genesis ->> server.controller.cloud: Node Info server.controller.cloud ->> server.controller.recorder: Node Info server.controller.recorder ->> mysql: insert pod_node table else is not the K8s watcher server.controller.trisolaris ->> agent: kubernetesapienabled=false end deactivate agent Note right of agent: 3. sync IP/MAC addresses activate agent agent ->> server.controller.genesis: gRPC.GenesisSync(k8s-cluster-id, local-ip/mac-list) server.controller.genesis ->> server.controller.cloud: Node IP/MAC server.controller.cloud ->> server.controller.recorder: Node IP/MAC server.controller.recorder ->> mysql: insert node mac into vinterface table server.controller.recorder ->> mysql: insert node ip into vinterface_ip table (LAN IP) server.controller.recorder ->> mysql: insert node ip into ip_resource table (WAN IP) deactivate agent Note right of agent: 4. agent (vtap) registration activate agent agent ->> server.controller.trisolaris: gRPC.Sync(k8s-cluster-id, node-ip, node-mac) server.controller.trisolaris ->> mysql: lookup node-mac in vinterface table server.controller.trisolaris ->> mysql: lookup node-ip in vinterfaceip/ipresource table server.controller.trisolaris ->> server.controller.trisolaris: match vtap and k8s-node server.controller.trisolaris ->> mysql: inesrt vtap table, return vtap_id server.controller.trisolaris ->> mysql: lookup agent_config by vtap-group-id server.controller.trisolaris ->> agent: vtapid, agentconfig deactivate agent Note right of agent: 5. subsequent agent requests activate agent agent ->> server.controller.trisolaris: gRPC.Sync(vtap_id) server.controller.trisolaris ->> agent: agent_config deactivate agent ``` ```mermaid sequenceDiagram participant agent participant server.controller.trisolaris participant server.controller.genesis participant server.controller.cloud participant server.controller.recorder participant mysql Note right of agent: 1. agent (vtap) first request activate agent agent ->> server.controller.trisolaris: gRPC.Sync(host-ip, host-mac) server.controller.trisolaris ->> mysql: lookup host-mac in vinterface table server.controller.trisolaris ->> mysql: lookup host-ip in vinterfaceip/ipresource table server.controller.trisolaris ->> server.controller.trisolaris: unknown agent, without k8s-cluster-id server.controller.trisolaris ->> agent: platform_enabled=true deactivate agent Note right of agent: 2. sync IP/MAC addresses activate agent agent ->> server.controller.genesis: gRPC.GenesisSync(hostname, local-ip/mac-list) server.controller.genesis ->> server.controller.cloud: Host Info server.controller.cloud ->> server.controller.recorder: Host Info server.controller.recorder ->> mysql: insert host into vm table server.controller.recorder ->> mysql: insert host mac into vinterface table server.controller.recorder ->> mysql: insert host ip into vinterface_ip table (LAN IP) server.controller.recorder ->> mysql: insert host ip into ip_resource table (WAN IP) deactivate agent Note right of agent: 3. agent registration activate agent agent ->> server.controller.trisolaris: gRPC.Sync(host-ip, host-mac) server.controller.trisolaris ->> mysql: lookup host-mac in vinterface table server.controller.trisolaris ->> mysql: lookup host-ip in vinterfaceip/ipresource table server.controller.trisolaris ->> server.controller.trisolaris: match vtap and host (vm) server.controller.trisolaris ->> mysql: inesrt vtap table, return vtap_id server.controller.trisolaris ->> mysql: lookup agent_config by vtap-group-id server.controller.trisolaris ->> agent: vtapid, agentconfig deactivate agent Note right of agent: 4. subsequent agent requests activate agent agent ->> server.controller.trisolaris: gRPC.Sync(vtap_id) server.controller.trisolaris ->> agent: agent_config deactivate agent ``` TODO TODO