Repository: kmesh-net/kmesh (go)

Issue:
Question about destination_service and destination_service_name metric labels
<!-- Please use this template while asking a new question. Thanks!
Please make sure you have read the FAQ and searched the issue list.
- FAQ: To be added
- Issues: https://github.com/kmesh-net/kmesh/issues?q=is%3Aissue+
-->
**Please provide an in-depth description of the question you have**:
I have found out a strange behavior about the metric labels exported by Kmesh. I'm testing in dual-engine mode. If you explore the tcp traffics and compare the `destination_service` and `destination_service_name` labels, you'll see that these two labels are exactly the same: they are Service FQDN.
![image](https://github.com/user-attachments/assets/7e03d17f-f387-420d-a563-ae264c090fae)

**What do you think about this question?**:
Istio has explained these two metrics in the documentation about Istio Standard Metrics(https://istio.io/latest/docs/reference/config/metrics/).

> Destination Service: This identifies destination service host responsible for an incoming request. Ex: details.default.svc.cluster.local.

> Destination Service Name: This identifies the destination service name. Ex: details.

As for Kmesh, I'm not sure if this is a bug, or this is designed by purpose. Therefore I just submit a question here.

A known effection of this issue is that Kiali may draw Service nodes on its traffic graph depends on the `destination_service_name` label.

**Environment**:
- Kmesh version: latest
- Others:


Pull request description:
update hostname and record unknown services of service metrics
**What type of PR is this?**

<!--
Add one of the following kinds:

/kind bug
/kind cleanup
/kind enhancement
/kind security
/kind documentation
/kind feature

-->

/kind bug

**What this PR does / why we need it**:

**Which issue(s) this PR fixes**:
Fixes #1089. Update `destination_service_name` to service name instead of host.

Here are some issues tha remain to be disccused:
- When a service is not found when building service metrics, I record the `dstIp` in `destination_service`, I'm not sure if it is appropriate.
- When recording waypoint metrics, services can not found since the `dstIp` is 15019, but only 15021 and 15008 are defined in waypoint services, so the following logic fails every time. Ztunnel uses actual service as destination service instead of the waypoint service, but seems like kmesh can not achieve that. I'm not sure how to handle this situation.
```go
namespacedhost := ""
if dstWorkload != nil {
	for k, portList := range dstWorkload.Services {
		for _, port := range portList.Ports {
			if port.TargetPort == uint32(data.dstPort) {
				namespacedhost = k
				break
			}
		}
		if namespacedhost != "" {
			break
		}
	}
}
```
```shell
# kmesh waypoint metric
kmesh_tcp_connections_closed_total{connection_security_policy="mutual_tls",destination_app="reviews-svc-waypoint",destination_canonical_revision="latest",destination_canonical_service="reviews-svc-waypoint",destination_cluster="Kubernetes",destination_principal="-",destination_service="10.244.1.11",destination_service_name="-",destination_service_namespace="-",destination_version="latest",destination_workload="reviews-svc-waypoint",destination_workload_namespace="default",reporter="source",request_protocol="tcp",response_flags="-",source_app="productpage",source_canonical_revision="v1",source_canonical_service="productpage",source_cluster="Kubernetes",source_principal="-",source_version="v1",source_workload="productpage-v1",source_workload_namespace="default"} 14

# ztunnel waypoint metric
istio_tcp_received_bytes_total{app="ztunnel", app_kubernetes_io_instance="istio", app_kubernetes_io_managed_by="Helm", app_kubernetes_io_name="ztunnel", app_kubernetes_io_part_of="istio", app_kubernetes_io_version="1.24.1", connection_security_policy="mutual_tls", controller_revision_hash="5d8567b5cd", destination_app="reviews-svc-waypoint", destination_canonical_revision="latest", destination_canonical_service="reviews-svc-waypoint", destination_cluster="Kubernetes", destination_principal="spiffe://cluster.local/ns/default/sa/reviews-svc-waypoint", destination_service="reviews.default.svc.cluster.local", destination_service_name="reviews", destination_service_namespace="default", destination_version="latest", destination_workload="reviews-svc-waypoint", destination_workload_namespace="default", helm_sh_chart="ztunnel-1.24.1", instance="10.17.40.69:15020", istio_io_dataplane_mode="none", job="kubernetes-pods", namespace="istio-system", node="ap-northeast-1.10.17.40.52", pod="ztunnel-28bpw", pod_template_generation="1", reporter="source", request_protocol="tcp", response_flags="-", sidecar_istio_io_inject="false", source_app="productpage", source_canonical_revision="v1", source_canonical_service="productpage", source_cluster="Kubernetes", source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage", source_version="v1", source_workload="productpage-v1", source_workload_namespace="default"}
```
- And a small question that why the metric are separated into service metric and workload metric?

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required.
-->
```release-note

```


Patch:
diff --git a/pkg/controller/ads/cache.go b/pkg/controller/ads/cache.go
index a3f80bed..fcd61f64 100644
--- a/pkg/controller/ads/cache.go
+++ b/pkg/controller/ads/cache.go
@@ -315,7 +315,7 @@ func (load *AdsCache) CreateApiRouteByRds(status core_v2.ApiStatus, routeConfig
 	load.RouteCache.SetApiRouteConfig(apiRouteConfig.GetName(), apiRouteConfig)
 }
 
-func (load *AdsCache) UpateApiRouteStatus(key string, status core_v2.ApiStatus) {
+func (load *AdsCache) UpdateApiRouteStatus(key string, status core_v2.ApiStatus) {
 	load.RouteCache.UpdateApiRouteStatus(key, status)
 }
 
diff --git a/pkg/controller/telemetry/accesslog.go b/pkg/controller/telemetry/accesslog.go
index 1d2aa00c..4e8dc10e 100644
--- a/pkg/controller/telemetry/accesslog.go
+++ b/pkg/controller/telemetry/accesslog.go
@@ -20,6 +20,8 @@ import (
 	"fmt"
 	"syscall"
 	"time"
+
+	"kmesh.net/kmesh/api/v2/workloadapi"
 )
 
 type logInfo struct {
@@ -34,6 +36,46 @@ type logInfo struct {
 	destinationNamespace string
 }
 
+func NewLogInfo() *logInfo {
+	return &logInfo{
+		direction:            DEFAULT_UNKNOWN,
+		sourceAddress:        DEFAULT_UNKNOWN,
+		sourceWorkload:       DEFAULT_UNKNOWN,
+		sourceNamespace:      DEFAULT_UNKNOWN,
+		destinationAddress:   DEFAULT_UNKNOWN,
+		destinationService:   DEFAULT_UNKNOWN,
+		destinationWorkload:  DEFAULT_UNKNOWN,
+		destinationNamespace: DEFAULT_UNKNOWN,
+	}
+}
+
+func (l *logInfo) withSource(workload *workloadapi.Workload) *logInfo {
+	if workload.GetNamespace() != "" {
+		l.sourceNamespace = workload.GetNamespace()
+	}
+	if workload.GetName() != "" {
+		l.sourceWorkload = workload.GetName()
+	}
+	return l
+}
+
+func (l *logInfo) withDestination(workload *workloadapi.Workload) *logInfo {
+	if workload.GetName() != "" {
+		l.destinationWorkload = workload.GetName()
+	}
+	return l
+}
+
+func (l *logInfo) withDestinationService(service *workloadapi.Service) *logInfo {
+	if service.GetNamespace() != "" {
+		l.destinationNamespace = service.GetNamespace()
+	}
+	if service.GetHostname() != "" {
+		l.destinationService = service.GetHostname()
+	}
+	return l
+}
+
 func OutputAccesslog(data requestMetric, accesslog logInfo) {
 	logStr := buildAccesslog(data, accesslog)
 	fmt.Println("accesslog:", logStr)
diff --git a/pkg/controller/telemetry/metric.go b/pkg/controller/telemetry/metric.go
index cf84d1ef..fa922535 100644
--- a/pkg/controller/telemetry/metric.go
+++ b/pkg/controller/telemetry/metric.go
@@ -23,7 +23,6 @@ import (
 	"fmt"
 	"net/netip"
 	"reflect"
-	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -46,6 +45,8 @@ const (
 	MSG_LEN = 112
 
 	metricFlushInterval = 5 * time.Second
+
+	DEFAULT_UNKNOWN = "-"
 )
 
 var osStartTime time.Time
@@ -55,6 +56,7 @@ type MetricController struct {
 	EnableMonitoring     atomic.Bool
 	EnableWorkloadMetric atomic.Bool
 	workloadCache        cache.WorkloadCache
+	serviceCache         cache.ServiceCache
 	workloadMetricCache  map[workloadMetricLabels]*workloadMetricInfo
 	serviceMetricCache   map[serviceMetricLabels]*serviceMetricInfo
 	mutex                sync.RWMutex
@@ -177,9 +179,90 @@ type serviceMetricLabels struct {
 	connectionSecurityPolicy string
 }
 
-func NewMetric(workloadCache cache.WorkloadCache, enableMonitoring bool) *MetricController {
+func NewServiceMetricLabel() *serviceMetricLabels {
+	return &serviceMetricLabels{}
+}
+
+func NewWorkloadMetricLabel() *workloadMetricLabels {
+	return &workloadMetricLabels{}
+}
+
+func (w *workloadMetricLabels) withSource(workload *workloadapi.Workload) *workloadMetricLabels {
+	if workload == nil {
+		return w
+	}
+	w.sourceWorkload = workload.GetWorkloadName()
+	w.sourceCanonicalService = workload.GetCanonicalName()
+	w.sourceCanonicalRevision = workload.GetCanonicalRevision()
+	w.sourceWorkloadNamespace = workload.GetNamespace()
+	w.sourceApp = workload.GetCanonicalName()
+	w.sourceVersion = workload.GetCanonicalRevision()
+	w.sourceCluster = workload.GetClusterId()
+	w.sourcePrincipal = buildPrincipal(workload)
+	return w
+}
+
+func (w *workloadMetricLabels) withDestination(workload *workloadapi.Workload) *workloadMetricLabels {
+	if workload == nil {
+		return w
+	}
+	w.destinationPodNamespace = workload.GetNamespace()
+	w.destinationPodName = workload.GetName()
+	w.destinationWorkload = workload.GetWorkloadName()
+	w.destinationCanonicalService = workload.GetCanonicalName()
+	w.destinationCanonicalRevision = workload.GetCanonicalRevision()
+	w.destinationWorkloadNamespace = workload.GetNamespace()
+	w.destinationApp = workload.GetCanonicalName()
+	w.destinationVersion = workload.GetCanonicalRevision()
+	w.destinationCluster = workload.GetClusterId()
+	w.destinationPrincipal = buildPrincipal(workload)
+	return w
+}
+
+func (s *serviceMetricLabels) withSource(workload *workloadapi.Workload) *serviceMetricLabels {
+	if workload == nil {
+		return s
+	}
+	s.sourceWorkload = workload.GetWorkloadName()
+	s.sourceCanonicalService = workload.GetCanonicalName()
+	s.sourceCanonicalRevision = workload.GetCanonicalRevision()
+	s.sourceWorkloadNamespace = workload.GetNamespace()
+	s.sourceApp = workload.GetCanonicalName()
+	s.sourceVersion = workload.GetCanonicalRevision()
+	s.sourceCluster = workload.GetClusterId()
+	s.sourcePrincipal = buildPrincipal(workload)
+	return s
+}
+
+func (s *serviceMetricLabels) withDestinationService(service *workloadapi.Service) *serviceMetricLabels {
+	if service == nil {
+		return s
+	}
+	s.destinationService = service.GetHostname()
+	s.destinationServiceName = service.GetName()
+	s.destinationServiceNamespace = service.GetNamespace()
+	return s
+}
+
+func (s *serviceMetricLabels) withDestination(workload *workloadapi.Workload) *serviceMetricLabels {
+	if workload == nil {
+		return s
+	}
+	s.destinationWorkload = workload.GetWorkloadName()
+	s.destinationCanonicalService = workload.GetCanonicalName()
+	s.destinationCanonicalRevision = workload.GetCanonicalRevision()
+	s.destinationWorkloadNamespace = workload.GetNamespace()
+	s.destinationApp = workload.GetCanonicalName()
+	s.destinationVersion = workload.GetCanonicalRevision()
+	s.destinationCluster = workload.GetClusterId()
+	s.destinationPrincipal = buildPrincipal(workload)
+	return s
+}
+
+func NewMetric(workloadCache cache.WorkloadCache, serviceCache cache.ServiceCache, enableMonitoring bool) *MetricController {
 	m := &MetricController{
 		workloadCache:       workloadCache,
+		serviceCache:        serviceCache,
 		workloadMetricCache: map[workloadMetricLabels]*workloadMetricInfo{},
 		serviceMetricCache:  map[serviceMetricLabels]*serviceMetricInfo{},
 	}
@@ -336,22 +419,22 @@ func (m *MetricController) buildWorkloadMetric(data *requestMetric) workloadMetr
 		return workloadMetricLabels{}
 	}
 
-	trafficLabels := buildWorkloadMetric(dstWorkload, srcWorkload)
-	trafficLabels.reporter = "-"
-
-	if data.direction == constants.INBOUND {
-		trafficLabels.reporter = "destination"
-	}
-	if data.direction == constants.OUTBOUND {
-		trafficLabels.reporter = "source"
-	}
+	trafficLabels := NewWorkloadMetricLabel()
+	trafficLabels.withSource(srcWorkload).withDestination(dstWorkload)
 
 	trafficLabels.destinationPodAddress = dstIP
 	trafficLabels.requestProtocol = "tcp"
-	trafficLabels.responseFlags = "-"
 	trafficLabels.connectionSecurityPolicy = "mutual_tls"
+	trafficLabels.reporter = DEFAULT_UNKNOWN
+
+	switch data.direction {
+	case constants.INBOUND:
+		trafficLabels.reporter = "destination"
+	case constants.OUTBOUND:
+		trafficLabels.reporter = "source"
+	}
 
-	return trafficLabels
+	return *trafficLabels
 }
 
 func (m *MetricController) buildServiceMetric(data *requestMetric) (serviceMetricLabels, logInfo) {
@@ -363,87 +446,13 @@ func (m *MetricController) buildServiceMetric(data *requestMetric) (serviceMetri
 
 	dstWorkload, dstIp := m.getWorkloadByAddress(restoreIPv4(dstAddr))
 	srcWorkload, srcIp := m.getWorkloadByAddress(restoreIPv4(srcAddr))
-	trafficLabels, accesslog := buildServiceMetric(dstWorkload, srcWorkload, data.dstPort)
-	trafficLabels.reporter = "-"
-
-	if data.direction == constants.INBOUND {
-		trafficLabels.reporter = "destination"
-		accesslog.direction = "INBOUND"
-	}
-	if data.direction == constants.OUTBOUND {
-		trafficLabels.reporter = "source"
-		accesslog.direction = "OUTBOUND"
-	}
-
-	trafficLabels.requestProtocol = "tcp"
-	trafficLabels.responseFlags = "-"
-	trafficLabels.connectionSecurityPolicy = "mutual_tls"
-
-	accesslog.destinationAddress = dstIp + ":" + fmt.Sprintf("%d", data.dstPort)
-	accesslog.sourceAddress = srcIp + ":" + fmt.Sprintf("%d", data.srcPort)
-
-	return trafficLabels, accesslog
-}
-
-func (m *MetricController) getWorkloadByAddress(address []byte) (*workloadapi.Workload, string) {
-	networkAddr := cache.NetworkAddress{}
-	networkAddr.Address, _ = netip.AddrFromSlice(address)
-	workload := m.workloadCache.GetWorkloadByAddr(networkAddr)
-	if workload == nil {
-		return nil, networkAddr.Address.String()
-	}
-	return workload, networkAddr.Address.String()
-}
-
-func buildWorkloadMetric(dstWorkload, srcWorkload *workloadapi.Workload) workloadMetricLabels {
-	trafficLabels := workloadMetricLabels{}
-
-	if dstWorkload != nil {
-		trafficLabels.destinationPodNamespace = dstWorkload.Namespace
-		trafficLabels.destinationPodName = dstWorkload.Name
-		trafficLabels.destinationWorkload = dstWorkload.WorkloadName
-		trafficLabels.destinationCanonicalService = dstWorkload.CanonicalName
-		trafficLabels.destinationCanonicalRevision = dstWorkload.CanonicalRevision
-		trafficLabels.destinationWorkloadNamespace = dstWorkload.Namespace
-		trafficLabels.destinationApp = dstWorkload.CanonicalName
-		trafficLabels.destinationVersion = dstWorkload.CanonicalRevision
-		trafficLabels.destinationCluster = dstWorkload.ClusterId
-		trafficLabels.destinationPrincipal = buildPrincipal(dstWorkload)
-	}
-
-	if srcWorkload != nil {
-		trafficLabels.sourceWorkload = srcWorkload.WorkloadName
-		trafficLabels.sourceCanonicalService = srcWorkload.CanonicalName
-		trafficLabels.sourceCanonicalRevision = srcWorkload.CanonicalRevision
-		trafficLabels.sourceWorkloadNamespace = srcWorkload.Namespace
-		trafficLabels.sourceApp = srcWorkload.CanonicalName
-		trafficLabels.sourceVersion = srcWorkload.CanonicalRevision
-		trafficLabels.sourceCluster = srcWorkload.ClusterId
-		trafficLabels.sourcePrincipal = buildPrincipal(srcWorkload)
-	}
-
-	return trafficLabels
-}
-
-func buildServiceMetric(dstWorkload, srcWorkload *workloadapi.Workload, dstPort uint16) (serviceMetricLabels, logInfo) {
-	trafficLabels := serviceMetricLabels{}
-	accesslog := logInfo{
-		direction:            "-",
-		sourceAddress:        "-",
-		sourceWorkload:       "-",
-		sourceNamespace:      "-",
-		destinationAddress:   "-",
-		destinationService:   "-",
-		destinationWorkload:  "-",
-		destinationNamespace: "-",
-	}
 
+	namespacedhost := ""
 	if dstWorkload != nil {
-		namespacedhost := ""
-		for host, portList := range dstWorkload.Services {
+		for k, portList := range dstWorkload.Services {
 			for _, port := range portList.Ports {
-				if port.TargetPort == uint32(dstPort) {
-					namespacedhost = host
+				if port.TargetPort == uint32(data.dstPort) {
+					namespacedhost = k
 					break
 				}
 			}
@@ -460,59 +469,56 @@ func buildServiceMetric(dstWorkload, srcWorkload *workloadapi.Workload, dstPort
 				}
 			}
 		}
+	}
 
-		svcHost := ""
-		svcNamespace := ""
-		if len(strings.Split(namespacedhost, "/")) == 2 {
-			svcNamespace = strings.Split(namespacedhost, "/")[0]
-			svcHost = strings.Split(namespacedhost, "/")[1]
-		}
-
-		trafficLabels.destinationService = svcHost
-		trafficLabels.destinationServiceNamespace = svcNamespace
-		trafficLabels.destinationServiceName = svcHost
-
-		trafficLabels.destinationWorkload = dstWorkload.WorkloadName
-		trafficLabels.destinationCanonicalService = dstWorkload.CanonicalName
-		trafficLabels.destinationCanonicalRevision = dstWorkload.CanonicalRevision
-		trafficLabels.destinationWorkloadNamespace = dstWorkload.Namespace
-		trafficLabels.destinationApp = dstWorkload.CanonicalName
-		trafficLabels.destinationVersion = dstWorkload.CanonicalRevision
-		trafficLabels.destinationCluster = dstWorkload.ClusterId
-		trafficLabels.destinationPrincipal = buildPrincipal(dstWorkload)
-		trafficLabels.destinationPrincipal = buildPrincipal(dstWorkload)
-
-		accesslog.destinationWorkload = dstWorkload.Name
-		if svcNamespace != "" {
-			accesslog.destinationNamespace = svcNamespace
-		}
-		if svcHost != "" {
-			accesslog.destinationService = svcHost
+	dstService := m.serviceCache.GetService(namespacedhost)
+	// when service not found, we use the address as hostname for metrics
+	if dstService == nil {
+		dstService = &workloadapi.Service{
+			Hostname: dstIp,
 		}
 	}
 
-	if srcWorkload != nil {
-		trafficLabels.sourceWorkload = srcWorkload.WorkloadName
-		trafficLabels.sourceCanonicalService = srcWorkload.CanonicalName
-		trafficLabels.sourceCanonicalRevision = srcWorkload.CanonicalRevision
-		trafficLabels.sourceWorkloadNamespace = srcWorkload.Namespace
-		trafficLabels.sourceApp = srcWorkload.CanonicalName
-		trafficLabels.sourceVersion = srcWorkload.CanonicalRevision
-		trafficLabels.sourceCluster = srcWorkload.ClusterId
-		trafficLabels.sourcePrincipal = buildPrincipal(srcWorkload)
-
-		accesslog.sourceNamespace = srcWorkload.Namespace
-		accesslog.sourceWorkload = srcWorkload.Name
+	trafficLabels := NewServiceMetricLabel()
+	trafficLabels.withSource(srcWorkload).withDestination(dstWorkload).withDestinationService(dstService)
+
+	trafficLabels.requestProtocol = "tcp"
+	trafficLabels.connectionSecurityPolicy = "mutual_tls"
+	trafficLabels.reporter = DEFAULT_UNKNOWN
+
+	accesslog := NewLogInfo()
+	accesslog.withSource(srcWorkload).withDestination(dstWorkload).withDestinationService(dstService)
+
+	accesslog.destinationAddress = dstIp + ":" + fmt.Sprintf("%d", data.dstPort)
+	accesslog.sourceAddress = srcIp + ":" + fmt.Sprintf("%d", data.srcPort)
+
+	switch data.direction {
+	case constants.INBOUND:
+		trafficLabels.reporter = "destination"
+		accesslog.direction = "INBOUND"
+	case constants.OUTBOUND:
+		trafficLabels.reporter = "source"
+		accesslog.direction = "OUTBOUND"
 	}
 
-	return trafficLabels, accesslog
+	return *trafficLabels, *accesslog
+}
+
+func (m *MetricController) getWorkloadByAddress(address []byte) (*workloadapi.Workload, string) {
+	networkAddr := cache.NetworkAddress{}
+	networkAddr.Address, _ = netip.AddrFromSlice(address)
+	workload := m.workloadCache.GetWorkloadByAddr(networkAddr)
+	if workload == nil {
+		return nil, networkAddr.Address.String()
+	}
+	return workload, networkAddr.Address.String()
 }
 
 func buildPrincipal(workload *workloadapi.Workload) string {
 	if workload.TrustDomain != "" && workload.ServiceAccount != "" && workload.Namespace != "" {
 		return fmt.Sprintf("spiffe://%s/ns/%s/sa/%s", workload.TrustDomain, workload.Namespace, workload.ServiceAccount)
 	}
-	return "-"
+	return DEFAULT_UNKNOWN
 }
 
 func (m *MetricController) updateWorkloadMetricCache(data requestMetric, labels workloadMetricLabels) {
@@ -628,7 +634,7 @@ func struct2map(labels interface{}) map[string]string {
 		for i := 0; i < num; i++ {
 			fieldInfo := val.Type().Field(i)
 			if val.Field(i).String() == "" {
-				trafficLabelsMap[labelsMap[fieldInfo.Name]] = "-"
+				trafficLabelsMap[labelsMap[fieldInfo.Name]] = DEFAULT_UNKNOWN
 			} else {
 				trafficLabelsMap[labelsMap[fieldInfo.Name]] = val.Field(i).String()
 			}
diff --git a/pkg/controller/workload/cache/waypoint_cache.go b/pkg/controller/workload/cache/waypoint_cache.go
index 9186f7e6..0e528fd0 100644
--- a/pkg/controller/workload/cache/waypoint_cache.go
+++ b/pkg/controller/workload/cache/waypoint_cache.go
@@ -29,7 +29,7 @@ type WaypointCache interface {
 	DeleteService(resourceName string)
 	// AddOrUpdateWorkload add or update workload in this cache, return true if the
 	// workload's waypoint doesn't need to be resolved or resolved successfully.
-	AddOrUpateWorkload(workload *workloadapi.Workload) bool
+	AddOrUpdateWorkload(workload *workloadapi.Workload) bool
 	DeleteWorkload(uid string)
 
 	// Refresh is used to process waypoint service.
@@ -148,12 +148,12 @@ func (w *waypointCache) DeleteService(resourceName string) {
 	delete(w.waypointAssociatedObjects, resourceName)
 }
 
-func (w *waypointCache) AddOrUpateWorkload(workload *workloadapi.Workload) bool {
+func (w *waypointCache) AddOrUpdateWorkload(workload *workloadapi.Workload) bool {
 	w.mutex.Lock()
 	defer w.mutex.Unlock()
 
 	uid := workload.GetUid()
-	// If this is a workload withwaypoint or  with an IP address type waypoint, no processing is required and
+	// If this is a workload with waypoint or with an IP address type waypoint, no processing is required and
 	// return directly.
 	if workload.GetWaypoint() == nil || workload.GetWaypoint().GetAddress() != nil {
 		// Workload may become unassociated with waypoint.
diff --git a/pkg/controller/workload/workload_controller.go b/pkg/controller/workload/workload_controller.go
index 311c8b8d..684882af 100644
--- a/pkg/controller/workload/workload_controller.go
+++ b/pkg/controller/workload/workload_controller.go
@@ -57,7 +57,7 @@ func NewController(bpfWorkload *bpfwl.BpfWorkload, enableMonitoring, enablePerfM
 		c.Processor.bpf.RestoreEndpointKeys()
 	}
 	c.Rbac = auth.NewRbac(c.Processor.WorkloadCache)
-	c.MetricController = telemetry.NewMetric(c.Processor.WorkloadCache, enableMonitoring)
+	c.MetricController = telemetry.NewMetric(c.Processor.WorkloadCache, c.Processor.ServiceCache, enableMonitoring)
 	if enablePerfMonitor {
 		c.OperationMetricController = telemetry.NewBpfProgMetric()
 		c.MapMetricController = telemetry.NewMapMetric()
diff --git a/pkg/controller/workload/workload_processor.go b/pkg/controller/workload/workload_processor.go
index 8421f1ab..4a2cbfe2 100644
--- a/pkg/controller/workload/workload_processor.go
+++ b/pkg/controller/workload/workload_processor.go
@@ -437,7 +437,7 @@ func (p *Processor) updateWorkloadInFrontendMap(workload *workloadapi.Workload)
 func (p *Processor) handleWorkload(workload *workloadapi.Workload) error {
 	log.Debugf("handle workload: %s", workload.ResourceName())
 
-	if resolved := p.WaypointCache.AddOrUpateWorkload(workload); !resolved {
+	if resolved := p.WaypointCache.AddOrUpdateWorkload(workload); !resolved {
 		// If the hostname type waypoint of workload has not been resolved, it will not be processed
 		// for the time being. The corresponding waypoint service should be processed immediately, and then
 		// it will be handled after the batch resolution is completed in `WaypointCache.Refresh`.


Test patch:
diff --git a/pkg/controller/telemetry/metric_test.go b/pkg/controller/telemetry/metric_test.go
index dd597ccb..1d1eeeb0 100644
--- a/pkg/controller/telemetry/metric_test.go
+++ b/pkg/controller/telemetry/metric_test.go
@@ -310,77 +310,6 @@ func TestBuildServiceMetricsToPrometheus(t *testing.T) {
 	}
 }
 
-func TestBuildWorkloadMetric(t *testing.T) {
-	type args struct {
-		dstWorkload *workloadapi.Workload
-		srcWorkload *workloadapi.Workload
-	}
-	tests := []struct {
-		name string
-		args args
-		want workloadMetricLabels
-	}{
-		{
-			name: "normal capability test",
-			args: args{
-				dstWorkload: &workloadapi.Workload{
-					Namespace:         "kmesh-system",
-					Name:              "kmesh",
-					WorkloadName:      "kmesh-daemon",
-					CanonicalName:     "dstCanonical",
-					CanonicalRevision: "dstVersion",
-					ClusterId:         "Kubernetes",
-					TrustDomain:       "cluster.local",
-					ServiceAccount:    "default",
-				},
-				srcWorkload: &workloadapi.Workload{
-					Namespace:         "kmesh-system",
-					Name:              "kmesh",
-					WorkloadName:      "kmesh-daemon",
-					CanonicalName:     "srcCanonical",
-					CanonicalRevision: "srcVersion",
-					ClusterId:         "Kubernetes",
-					TrustDomain:       "cluster.local",
-					ServiceAccount:    "default",
-				},
-			},
-			want: workloadMetricLabels{
-				reporter:                     "-",
-				sourceWorkload:               "kmesh-daemon",
-				sourceCanonicalService:       "srcCanonical",
-				sourceCanonicalRevision:      "srcVersion",
-				sourceWorkloadNamespace:      "kmesh-system",
-				sourcePrincipal:              "spiffe://cluster.local/ns/kmesh-system/sa/default",
-				sourceApp:                    "srcCanonical",
-				sourceVersion:                "srcVersion",
-				sourceCluster:                "Kubernetes",
-				destinationPodAddress:        "-",
-				destinationPodNamespace:      "kmesh-system",
-				destinationPodName:           "kmesh",
-				destinationWorkload:          "kmesh-daemon",
-				destinationCanonicalService:  "dstCanonical",
-				destinationCanonicalRevision: "dstVersion",
-				destinationWorkloadNamespace: "kmesh-system",
-				destinationPrincipal:         "spiffe://cluster.local/ns/kmesh-system/sa/default",
-				destinationApp:               "dstCanonical",
-				destinationVersion:           "dstVersion",
-				destinationCluster:           "Kubernetes",
-				requestProtocol:              "-",
-				responseFlags:                "-",
-				connectionSecurityPolicy:     "-",
-			},
-		},
-	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			actualLabels := buildWorkloadMetric(tt.args.dstWorkload, tt.args.srcWorkload)
-			expectMap := struct2map(tt.want)
-			actualMap := struct2map(actualLabels)
-			assert.Equal(t, expectMap, actualMap)
-		})
-	}
-}
-
 func TestMetricGetWorkloadByAddress(t *testing.T) {
 	workload := &workloadapi.Workload{
 		Name: "ut-workload",
@@ -487,7 +416,7 @@ func TestBuildworkloadMetric(t *testing.T) {
 				destinationVersion:           "dstVersion",
 				destinationCluster:           "Kubernetes",
 				requestProtocol:              "tcp",
-				responseFlags:                "-",
+				responseFlags:                "",
 				connectionSecurityPolicy:     "mutual_tls",
 				reporter:                     "-",
 			},
@@ -543,7 +472,22 @@ func TestRestoreIPv4(t *testing.T) {
 }
 
 func TestBuildServiceMetric(t *testing.T) {
-	dstWorkload := &workloadapi.Workload{
+	type args struct {
+		data *requestMetric
+	}
+	serviceCache := cache.NewServiceCache()
+	serviceCache.AddOrUpdateService(&workloadapi.Service{
+		Hostname:  "kmesh.kmesh-system.svc.cluster.local",
+		Namespace: "kmesh-system",
+		Name:      "kmesh",
+		Addresses: []*workloadapi.NetworkAddress{
+			{
+				Address: []byte("192.168.1.22"),
+			},
+		},
+	})
+	workloadCache := cache.NewWorkloadCache()
+	workloadCache.AddOrUpdateWorkload(&workloadapi.Workload{
 		Namespace:         "kmesh-system",
 		Name:              "kmesh",
 		WorkloadName:      "kmesh-daemon",
@@ -552,22 +496,21 @@ func TestBuildServiceMetric(t *testing.T) {
 		ClusterId:         "Kubernetes",
 		TrustDomain:       "cluster.local",
 		ServiceAccount:    "default",
-		Uid:               "123456",
-		Addresses: [][]byte{
-			{192, 168, 224, 22},
-		},
 		Services: map[string]*workloadapi.PortList{
 			"kmesh-system/kmesh.kmesh-system.svc.cluster.local": {
 				Ports: []*workloadapi.Port{
 					{
-						TargetPort:  80,
-						ServicePort: 8000,
+						ServicePort: 80,
+						TargetPort:  8000,
 					},
 				},
 			},
 		},
-	}
-	srcWorkload := &workloadapi.Workload{
+		Addresses: [][]byte{
+			{192, 168, 224, 22},
+		},
+	})
+	workloadCache.AddOrUpdateWorkload(&workloadapi.Workload{
 		Namespace:         "kmesh-system",
 		Name:              "kmesh",
 		WorkloadName:      "kmesh-daemon",
@@ -576,19 +519,18 @@ func TestBuildServiceMetric(t *testing.T) {
 		ClusterId:         "Kubernetes",
 		TrustDomain:       "cluster.local",
 		ServiceAccount:    "default",
-		Uid:               "654321",
 		Addresses: [][]byte{
 			{10, 19, 25, 31},
 		},
-	}
-	type args struct {
-		data *requestMetric
+	})
+	m := MetricController{
+		workloadCache: workloadCache,
+		serviceCache:  serviceCache,
 	}
 	tests := []struct {
 		name        string
 		args        args
 		want        serviceMetricLabels
-		wantErr     bool
 		wantLogInfo logInfo
 	}{
 		{
@@ -597,7 +539,7 @@ func TestBuildServiceMetric(t *testing.T) {
 				data: &requestMetric{
 					src:           [4]uint32{521736970, 0, 0, 0},
 					dst:           [4]uint32{383822016, 0, 0, 0},
-					dstPort:       uint16(80),
+					dstPort:       uint16(8000),
 					srcPort:       uint16(8000),
 					direction:     uint32(2),
 					sentBytes:     uint32(156),
@@ -615,7 +557,7 @@ func TestBuildServiceMetric(t *testing.T) {
 				sourceCluster:                "Kubernetes",
 				destinationService:           "kmesh.kmesh-system.svc.cluster.local",
 				destinationServiceNamespace:  "kmesh-system",
-				destinationServiceName:       "kmesh.kmesh-system.svc.cluster.local",
+				destinationServiceName:       "kmesh",
 				destinationWorkload:          "kmesh-daemon",
 				destinationCanonicalService:  "dstCanonical",
 				destinationCanonicalRevision: "dstVersion",
@@ -625,127 +567,35 @@ func TestBuildServiceMetric(t *testing.T) {
 				destinationVersion:           "dstVersion",
 				destinationCluster:           "Kubernetes",
 				requestProtocol:              "tcp",
-				responseFlags:                "-",
+				responseFlags:                "",
 				connectionSecurityPolicy:     "mutual_tls",
 				reporter:                     "source",
 			},
-			wantErr: false,
 			wantLogInfo: logInfo{
 				direction:            "OUTBOUND",
 				sourceAddress:        "10.19.25.31:8000",
 				sourceWorkload:       "kmesh",
 				sourceNamespace:      "kmesh-system",
-				destinationAddress:   "192.168.224.22:80",
+				destinationAddress:   "192.168.224.22:8000",
 				destinationService:   "kmesh.kmesh-system.svc.cluster.local",
 				destinationWorkload:  "kmesh",
 				destinationNamespace: "kmesh-system",
 			},
 		},
-	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			m := MetricController{
-				workloadCache: cache.NewWorkloadCache(),
-			}
-			m.workloadCache.AddOrUpdateWorkload(dstWorkload)
-			m.workloadCache.AddOrUpdateWorkload(srcWorkload)
-			got, accesslog := m.buildServiceMetric(tt.args.data)
-			assert.Equal(t, tt.want, got)
-			assert.Equal(t, tt.wantLogInfo, accesslog)
-		})
-	}
-}
-
-func Test_buildServiceMetric(t *testing.T) {
-	type args struct {
-		dstWorkload *workloadapi.Workload
-		srcWorkload *workloadapi.Workload
-		dstPort     uint16
-	}
-	tests := []struct {
-		name string
-		args args
-		want serviceMetricLabels
-	}{
-		{
-			name: "normal capability test",
-			args: args{
-				dstWorkload: &workloadapi.Workload{
-					Namespace:         "kmesh-system",
-					Name:              "kmesh",
-					WorkloadName:      "kmesh-daemon",
-					CanonicalName:     "dstCanonical",
-					CanonicalRevision: "dstVersion",
-					ClusterId:         "Kubernetes",
-					TrustDomain:       "cluster.local",
-					ServiceAccount:    "default",
-					Services: map[string]*workloadapi.PortList{
-						"kmesh-system/kmesh": {
-							Ports: []*workloadapi.Port{
-								{
-									ServicePort: 80,
-									TargetPort:  8000,
-								},
-							},
-						},
-					},
-				},
-				srcWorkload: &workloadapi.Workload{
-					Namespace:         "kmesh-system",
-					Name:              "kmesh",
-					WorkloadName:      "kmesh-daemon",
-					CanonicalName:     "srcCanonical",
-					CanonicalRevision: "srcVersion",
-					ClusterId:         "Kubernetes",
-					TrustDomain:       "cluster.local",
-					ServiceAccount:    "default",
-				},
-				dstPort: uint16(8000),
-			},
-			want: serviceMetricLabels{
-				reporter:                     "",
-				sourceWorkload:               "kmesh-daemon",
-				sourceCanonicalService:       "srcCanonical",
-				sourceCanonicalRevision:      "srcVersion",
-				sourceWorkloadNamespace:      "kmesh-system",
-				sourcePrincipal:              "spiffe://cluster.local/ns/kmesh-system/sa/default",
-				sourceApp:                    "srcCanonical",
-				sourceVersion:                "srcVersion",
-				sourceCluster:                "Kubernetes",
-				destinationService:           "kmesh",
-				destinationServiceNamespace:  "kmesh-system",
-				destinationServiceName:       "kmesh",
-				destinationWorkload:          "kmesh-daemon",
-				destinationCanonicalService:  "dstCanonical",
-				destinationCanonicalRevision: "dstVersion",
-				destinationWorkloadNamespace: "kmesh-system",
-				destinationPrincipal:         "spiffe://cluster.local/ns/kmesh-system/sa/default",
-				destinationApp:               "dstCanonical",
-				destinationVersion:           "dstVersion",
-				destinationCluster:           "Kubernetes",
-				requestProtocol:              "",
-				responseFlags:                "",
-				connectionSecurityPolicy:     "",
-			},
-		},
 		{
 			name: "nil destination workload",
 			args: args{
-				dstWorkload: &workloadapi.Workload{},
-				srcWorkload: &workloadapi.Workload{
-					Namespace:         "kmesh-system",
-					Name:              "kmesh",
-					WorkloadName:      "kmesh-daemon",
-					CanonicalName:     "srcCanonical",
-					CanonicalRevision: "srcVersion",
-					ClusterId:         "Kubernetes",
-					TrustDomain:       "cluster.local",
-					ServiceAccount:    "default",
+				data: &requestMetric{
+					src:           [4]uint32{521736970, 0, 0, 0},
+					dst:           [4]uint32{383822015, 0, 0, 0},
+					dstPort:       uint16(80),
+					srcPort:       uint16(8000),
+					direction:     uint32(2),
+					sentBytes:     uint32(156),
+					receivedBytes: uint32(1024),
 				},
-				dstPort: uint16(8000),
 			},
 			want: serviceMetricLabels{
-				reporter:                     "",
 				sourceWorkload:               "kmesh-daemon",
 				sourceCanonicalService:       "srcCanonical",
 				sourceCanonicalRevision:      "srcVersion",
@@ -754,27 +604,39 @@ func Test_buildServiceMetric(t *testing.T) {
 				sourceApp:                    "srcCanonical",
 				sourceVersion:                "srcVersion",
 				sourceCluster:                "Kubernetes",
-				destinationService:           "",
+				destinationService:           "191.168.224.22",
 				destinationServiceNamespace:  "",
 				destinationServiceName:       "",
 				destinationWorkload:          "",
 				destinationCanonicalService:  "",
 				destinationCanonicalRevision: "",
 				destinationWorkloadNamespace: "",
-				destinationPrincipal:         "-",
+				destinationPrincipal:         "",
 				destinationApp:               "",
 				destinationVersion:           "",
 				destinationCluster:           "",
-				requestProtocol:              "",
+				requestProtocol:              "tcp",
 				responseFlags:                "",
-				connectionSecurityPolicy:     "",
+				connectionSecurityPolicy:     "mutual_tls",
+				reporter:                     "source",
+			},
+			wantLogInfo: logInfo{
+				direction:            "OUTBOUND",
+				sourceAddress:        "10.19.25.31:8000",
+				sourceWorkload:       "kmesh",
+				sourceNamespace:      "kmesh-system",
+				destinationAddress:   "191.168.224.22:80",
+				destinationService:   "191.168.224.22",
+				destinationWorkload:  "-",
+				destinationNamespace: "-",
 			},
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			got, _ := buildServiceMetric(tt.args.dstWorkload, tt.args.srcWorkload, tt.args.dstPort)
+			got, loginfo := m.buildServiceMetric(tt.args.data)
 			assert.Equal(t, tt.want, got)
+			assert.Equal(t, tt.wantLogInfo, loginfo)
 		})
 	}
 }
diff --git a/pkg/controller/workload/cache/waypoint_cache_test.go b/pkg/controller/workload/cache/waypoint_cache_test.go
index fb48b26d..d6ba96ec 100644
--- a/pkg/controller/workload/cache/waypoint_cache_test.go
+++ b/pkg/controller/workload/cache/waypoint_cache_test.go
@@ -50,7 +50,7 @@ func TestBasic(t *testing.T) {
 	}
 
 	for _, wl := range []*workloadapi.Workload{wl1, wl2, wl3} {
-		cache.AddOrUpateWorkload(wl)
+		cache.AddOrUpdateWorkload(wl)
 	}
 
 	// Waypoint service has not been processed.
@@ -91,7 +91,7 @@ func TestBasic(t *testing.T) {
 	svc4 := createFakeService("svc4", "10.240.10.4", waypointHostname)
 	wl4 := createFakeWorkload("1.2.3.8", waypointHostname)
 	cache.AddOrUpdateService(svc4)
-	cache.AddOrUpateWorkload(wl4)
+	cache.AddOrUpdateWorkload(wl4)
 
 	// svc4 and wl4 have been added to the waypoint cache and hostname of waypoint has been resolved.
 	assert.Equal(t, isHostnameTypeWaypoint(associated.services[svc4.ResourceName()].Waypoint), false)
diff --git a/test/e2e/baseline_test.go b/test/e2e/baseline_test.go
index 1b829824..05e005b1 100644
--- a/test/e2e/baseline_test.go
+++ b/test/e2e/baseline_test.go
@@ -833,7 +833,7 @@ func buildL4Query(src, dst echo.Instance) prometheus.Query {
 		"destination_canonical_service":  dst.ServiceName(),
 		"destination_canonical_revision": dst.Config().Version,
 		"destination_service":            fmt.Sprintf("%s.%s.svc.cluster.local", dst.Config().Service, destns),
-		"destination_service_name":       fmt.Sprintf("%s.%s.svc.cluster.local", dst.Config().Service, destns),
+		"destination_service_name":       dst.Config().Service,
 		"destination_service_namespace":  destns,
 		"destination_principal":          "-",
 		"destination_version":            dst.Config().Version,