Repository: apache/skywalking (language: python)

Issue:
[Bug] Metrics are not aggregated properly as OTEL collector targeting per node
### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.


### Apache SkyWalking Component

OAP server (apache/skywalking)

### What happened

This issue was found during Clickhouse PR review, https://github.com/apache/skywalking/pull/11966#pullrequestreview-1912778007

We noticed this is a common bug case in previous monitoring. We should check them and update the status here.

- [x] MongoDB @weixiang1862 https://github.com/apache/skywalking/pull/12126
- [x] MySQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] MariaDB @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] PostgreSQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Redis @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Elasticsearch @innerpeacez 
- [x] Nginx @weixiang1862 https://github.com/apache/skywalking/pull/12096
- [x] APISIX @pg-yang https://github.com/apache/skywalking/pull/12108
- [x] Kafka @wankai123 
- [x] Pulsar @CodePrometheus https://github.com/apache/skywalking/pull/12174
- [x] RabbitMQ @innerpeacez https://github.com/apache/skywalking/pull/12144

### What you expected to happen

Service metrics should be labeled values, and only use MQE(query stage aggregation) to merge the data to build metrics accurately. 

In this case, if there are two nodes

In the case A
ck node1 -> OTEL -> OAP
ck node2-> OTEL -> OAP

In the case B
T1, ck node1 -> OTEL -> OAP
T2, ck node1 -> OTEL -> OAP
(T1 T2) are in the same minute(same for an hour/day)

Then from the OAP process perspective, it can't tell A and B, so, if you work on downsampling(default as AVG), it could be AVG of node1 and node2(case A), or AVG of ck node1 from T1 and T2(case B). 
For a more complex case, it could be a combination of case A and B, in one minute, N scrapings from node1 and M scrapings from node2, then the default sampling is doing AVG for N + M scrapings from node1 and node2. This actually is not the value of the aggregated metric for the whole service(multiple nodes).

### How to reproduce

Metrics from nodeA and nodeB will overlap with each other. Due to nodeA and nodeB will report data separately.

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)



Discussion / hints:
@CodePrometheus If you could help this, please let me know.
For all the owners, this list doesn't mean all monitoring at service level is inaccurate due to the way od scraping from OTEL. Please recheck first and only fix if it fits the bug descriptions.
ES checked, the metrics data from one cluster-level API.
Kafka monitoring service's MAL aggregates the metrics with `cluster and broker`, so it's good.
To build service level metrics with instance/node level metrics through `AggregateLabels Operation`,  I think `AggregateLabels`  should support aggregation through specified labels. Like the scenario below:

#### instance/node HTTP status metrics

```
# The number means the occurred times
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```

The MQE express to aggregate service level metrics should be  

```
aggregate_labels(http_status,'status',sum) # aggregate by status label
```


The result will be

```
http_status{status=401} 8

```


I don't think we support multiple labels after you have done the persistent.

```
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```
This is not a proper example, because OAP doesn't persistent data like this. 
In your case, the data in the DB should be `http_status_401{192.168.0.1=5, 192.168.0.3=1, 192.168.0.2=2}`. Please verify.
@weixiang1862 As you asked about, we posted https://github.com/apache/skywalking/issues/11992 for v10. 
@wu-sheng I would be willing, of course! I'll do recheck on this weekend.
> @wu-sheng I would be willing, of course! I'll do recheck on this weekend.

Which one will you check? Let's avoid duplicate works with others.
@wu-sheng I'll try to check pulsar first.
Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?
> > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> 
> The question is how to aggregate the metrics of the cluster?

The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.
> > > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> > 
> > 
> > The question is how to aggregate the metrics of the cluster?
> 
> The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.

OK. Then it is fine for now. But you could start a new issue about polishing the Pulsar monitoring as we discussed.
@pg-yang How is the APISIX? The metrics on the dashboard seem not correctly because they have cross-instance aggregation.
Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters
> Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters

Got it. That will need another one or two weeks.
@weixiang1862 @pg-yang @innerpeacez We have the new multi-label support merged.  You should be good to fix this.

Fix patch:
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 96182ba96aec..b46d648c1e96 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -106,6 +106,7 @@
 * Fix/Change UI init setting for Windows Swap -> Virtual Memory
 * Fix `Memory Swap Usage`/`Virtual Memory Usage` display with UI init.(Linux/Windows)
 * Fix inaccurate APISIX metrics
+* Fix inaccurate MongoDB Metrics.
 
 #### UI
 
diff --git a/docs/en/setup/backend/backend-mongodb-monitoring.md b/docs/en/setup/backend/backend-mongodb-monitoring.md
index d1e1845f53b8..58ea09182d95 100644
--- a/docs/en/setup/backend/backend-mongodb-monitoring.md
+++ b/docs/en/setup/backend/backend-mongodb-monitoring.md
@@ -25,21 +25,21 @@ MongoDB monitoring provides multidimensional metrics monitoring of MongoDB clust
 
 ### MongoDB Cluster Supported Metrics
 
-| Monitoring Panel                  | Unit | Metric Name                               | Description                                                         | Data Source      |
-|-----------------------------------|------|-------------------------------------------|---------------------------------------------------------------------|------------------|
-| Uptime (day)                      | day  | meter_mongodb_cluster_uptime              | Maximum uptime of nodes in the cluster                              | mongodb-exporter |
-| Data Size (GB)                    | GB   | meter_mongodb_cluster_data_size           | Total data size of the cluster                                      | mongodb-exporter |
-| Collection Count                  |      | meter_mongodb_cluster_collection_count    | Number of collection of the cluster                                 | mongodb-exporter |
-| Object Count                      |      | meter_mongodb_cluster_object_count        | Number of object of the cluster                                     | mongodb-exporter |
-| Document Avg QPS                  |      | meter_mongodb_cluster_document_avg_qps    | Avg document operations rate of nodes                               | mongodb-exporter |
-| Operation Avg QPS                 |      | meter_mongodb_cluster_operation_avg_qps   | Avg operations rate of nodes                                        | mongodb-exporter |
-| Total Connections                 |      | meter_mongodb_cluster_connections         | Cluster total connections of nodes                                  | mongodb-exporter |
-| Cursor Avg                        |      | meter_mongodb_cluster_cursor_avg          | Avg Opened cursor of nodes                                          | mongodb-exporter |
-| Replication Lag (ms)              | ms   | meter_mongodb_cluster_repl_lag            | Repl set member avg replication lag, this metric works in repl mode | mongodb-exporter |
-| DB Avg Data Size Per Shard (GB)   | GB   | meter_mongodb_cluster_db_data_size        | Avg data size per shard (replSet) of every database                 | mongodb-exporter |
-| DB Avg Index Size Per Shard (GB)  | GB   | meter_mongodb_cluster_db_index_size       | Avg index size per shard (replSet) of every database                | mongodb-exporter |
-| DB Avg Collection Count Per Shard |      | meter_mongodb_cluster_db_collection_count | Avg collection count per shard (replSet) of every database          | mongodb-exporter |
-| DB Avg Index Count Per Shard      |      | meter_mongodb_cluster_db_index_count      | Avg index count per shard (replSet) of every database               | mongodb-exporter |
+| Monitoring Panel          | Unit | Metric Name                               | Description                                                         | Data Source      |
+|---------------------------|------|-------------------------------------------|---------------------------------------------------------------------|------------------|
+| Uptime (day)              | day  | meter_mongodb_cluster_uptime              | Maximum uptime of nodes in the cluster                              | mongodb-exporter |
+| Data Size (GB)            | GB   | meter_mongodb_cluster_data_size           | Total data size of the cluster                                      | mongodb-exporter |
+| Collection Count          |      | meter_mongodb_cluster_collection_count    | Number of collection of the cluster                                 | mongodb-exporter |
+| Object Count              |      | meter_mongodb_cluster_object_count        | Number of object of the cluster                                     | mongodb-exporter |
+| Document Total QPS        |      | meter_mongodb_cluster_document_avg_qps    | Total document operations rate of nodes                             | mongodb-exporter |
+| Operation Total QPS       |      | meter_mongodb_cluster_operation_avg_qps   | Total operations rate of nodes                                      | mongodb-exporter |
+| Total Connections         |      | meter_mongodb_cluster_connections         | Cluster total connections of nodes                                  | mongodb-exporter |
+| Cursor Total              |      | meter_mongodb_cluster_cursor_avg          | Total Opened cursor of nodes                                        | mongodb-exporter |
+| Replication Lag (ms)      | ms   | meter_mongodb_cluster_repl_lag            | Repl set member avg replication lag, this metric works in repl mode | mongodb-exporter |
+| DB Total Data Size (GB)   | GB   | meter_mongodb_cluster_db_data_size        | Total data size  of every database                                  | mongodb-exporter |
+| DB Total Index Size (GB)  | GB   | meter_mongodb_cluster_db_index_size       | Total index size per of every database                              | mongodb-exporter |
+| DB Total Collection Count |      | meter_mongodb_cluster_db_collection_count | Total collection count of every database                            | mongodb-exporter |
+| DB Total Index Count      |      | meter_mongodb_cluster_db_index_count      | Total index count of every database                                 | mongodb-exporter |
 
 ### MongoDB Node Supported Metrics
 
@@ -47,7 +47,7 @@ MongoDB monitoring provides multidimensional metrics monitoring of MongoDB clust
 |------------------------------|------|-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|------------------|
 | Uptime (day)                 | day  | meter_mongodb_node_uptime                                                                                             | Uptime of the node                                         | mongodb-exporter |
 | QPS                          |      | meter_mongodb_node_qps                                                                                                | Operations per second of the node                          | mongodb-exporter | 
-| Latency                      | µs   | meter_mongodb_node_latency                                                                                            | Latency of operations                                      | mongodb-exporter |
+| Latency                      | µs   | meter_mongodb_node_op_rate<br/>meter_mongodb_node_latency_rate                                                        | Latency of operations                                      | mongodb-exporter |
 | Memory Usage                 | %    | meter_mongodb_node_memory_usage                                                                                       | Memory usage percent of RAM                                | mongodb-exporter |
 | Version                      |      | meter_mongodb_node_version                                                                                            | MongoDB edition and version                                | mongodb-exporter |
 | ReplSet State                |      | meter_mongodb_node_rs_state                                                                                           | Repl set state of the node, this metric works in repl mode | mongodb-exporter |
@@ -61,7 +61,7 @@ MongoDB monitoring provides multidimensional metrics monitoring of MongoDB clust
 | Document QPS                 |      | meter_mongodb_node_document_qps                                                                                       | Document operations per second                             | mongodb-exporter |
 | Operation QPS                |      | meter_mongodb_node_operation_qps                                                                                      | Operations per second                                      | mongodb-exporter |
 | Repl Operation QPS           |      | meter_mongodb_node_repl_operation_qps                                                                                 | Repl operations per second                                 | mongodb-exporter |
-| Operation Latency (µs)       | µs   | meter_mongodb_node_operation_latency                                                                                  | Latencies for different operation type                     | mongodb-exporter |
+| Operation Latency (µs)       | µs   | meter_mongodb_node_op_rate<br/>meter_mongodb_node_latency_rate                                                        | Latencies for different operation type                     | mongodb-exporter |
 | Cursor                       |      | meter_mongodb_node_cursor                                                                                             | Opened cursor of the node                                  | mongodb-exporter |
 | Server Status Memory (MB)    | MB   | meter_mongodb_node_mem_virtual<br/>meter_mongodb_node_mem_resident                                                    | Virtual and resident memory of the node                    | mongodb-exporter |
 | Asserts                      |      | meter_mongodb_node_asserts                                                                                            | The rate of raised assertions                              | mongodb-exporter |
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-cluster.yaml b/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-cluster.yaml
index 365eba0261da..711f46469bdf 100644
--- a/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-cluster.yaml
+++ b/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-cluster.yaml
@@ -42,22 +42,22 @@ metricsRules:
     exp: mongodb_dbstats_objects.tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'rs_nm'])
 
   - name: document_avg_qps
-    exp: mongodb_ss_metrics_document.max(['cluster','doc_op_type']).rate('PT1M')
+    exp: mongodb_ss_metrics_document.max(['cluster','doc_op_type','service_instance_id']).rate('PT1M')
   - name: operation_avg_qps
-    exp: mongodb_ss_opcounters.max(['cluster','legacy_op_type']).rate('PT1M')
+    exp: mongodb_ss_opcounters.max(['cluster','legacy_op_type','service_instance_id']).rate('PT1M')
 
   - name: connections
     exp: mongodb_ss_connections.tagEqual('conn_type','current').max(['cluster','service_instance_id'])
   - name: cursor_avg
-    exp: mongodb_ss_metrics_cursor_open.max(['cluster',"csr_type"])
+    exp: mongodb_ss_metrics_cursor_open.max(['cluster','csr_type','service_instance_id'])
   - name: repl_lag
-    exp: mongodb_mongod_replset_member_replication_lag.tag({tags -> tags.rs_nm = tags.set}).avg(['cluster','rs_nm'])
+    exp: mongodb_mongod_replset_member_replication_lag.tag({tags -> tags.rs_nm = tags.set}).tagNotEqual('state','ARBITER').avg(['cluster','rs_nm'])
 
   - name: db_data_size
-    exp: mongodb_dbstats_dataSize.tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database'])
+    exp: mongodb_dbstats_dataSize.tagEqual('rs_state', '1').tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database','service_instance_id'])
   - name: db_index_size
-    exp: mongodb_dbstats_indexSize.tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database'])
+    exp: mongodb_dbstats_indexSize.tagEqual('rs_state', '1').tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database','service_instance_id'])
   - name: db_collection_count
-    exp: mongodb_dbstats_collections.tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database'])
+    exp: mongodb_dbstats_collections.tagEqual('rs_state', '1').tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster', 'database','service_instance_id'])
   - name: db_index_count
-    exp: mongodb_dbstats_indexes.tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster' , 'database'])
\ No newline at end of file
+    exp: mongodb_dbstats_indexes.tagEqual('rs_state', '1').tagNotEqual('cl_role','mongos').tagNotEqual('database','local').sum(['cluster' , 'database','service_instance_id'])
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-node.yaml b/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-node.yaml
index 781d54114c1c..b670d066ab1d 100644
--- a/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-node.yaml
+++ b/oap-server/server-starter/src/main/resources/otel-rules/mongodb/mongodb-node.yaml
@@ -36,8 +36,10 @@ metricsRules:
     exp: mongodb_ss_uptime
   - name: qps
     exp: mongodb_ss_opcounters.sum(['cluster','service_instance_id']).rate('PT1M')
-  - name: latency
-    exp: mongodb_ss_opLatencies_latency.sum(['cluster','service_instance_id']).rate('PT1M') / mongodb_ss_opLatencies_ops.sum(['cluster','service_instance_id']).rate('PT1M')
+  - name: op_rate
+    exp: mongodb_ss_opLatencies_ops.sum(['cluster','service_instance_id', 'op_type']).rate('PT1M')
+  - name: latency_rate
+    exp: mongodb_ss_opLatencies_latency.sum(['cluster','service_instance_id', 'op_type']).rate('PT1M')
   - name: memory_usage
     exp: (mongodb_sys_memory_MemTotal_kb - mongodb_sys_memory_MemAvailable_kb) / mongodb_sys_memory_MemTotal_kb * 100
   - name: version
@@ -80,8 +82,6 @@ metricsRules:
   - name: repl_operation_qps
     exp: mongodb_ss_opcountersRepl.sum(['cluster','service_instance_id','legacy_op_type']).rate('PT1M')
 
-  - name: operation_latency
-    exp: mongodb_ss_opLatencies_latency.max(['cluster','service_instance_id',"op_type"]).rate('PT1M') / mongodb_ss_opLatencies_ops.max(['cluster','service_instance_id',"op_type"]).rate('PT1M')
   - name: cursor
     exp: mongodb_ss_metrics_cursor_open.max(['cluster','service_instance_id',"csr_type"])
   - name: mem_virtual
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-cluster.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-cluster.json
index db748e591cdd..e6959b782061 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-cluster.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-cluster.json
@@ -112,7 +112,7 @@
                   "i": "4",
                   "type": "Widget",
                   "expressions": [
-                    "meter_mongodb_cluster_document_avg_qps"
+                    "aggregate_labels(meter_mongodb_cluster_document_avg_qps, sum(doc_op_type))"
                   ],
                   "graph": {
                     "type": "Line",
@@ -123,7 +123,7 @@
                     "showYAxis": true
                   },
                   "widget": {
-                    "title": "Document Avg QPS"
+                    "title": "Document Total QPS"
                   }
                 },
                 {
@@ -134,7 +134,7 @@
                   "i": "5",
                   "type": "Widget",
                   "expressions": [
-                    "meter_mongodb_cluster_operation_avg_qps"
+                    "aggregate_labels(meter_mongodb_cluster_operation_avg_qps, sum(legacy_op_type))"
                   ],
                   "graph": {
                     "type": "Line",
@@ -145,7 +145,7 @@
                     "showYAxis": true
                   },
                   "widget": {
-                    "title": "Operation Avg QPS"
+                    "title": "Operation Total QPS"
                   }
                 },
                 {
@@ -158,6 +158,11 @@
                   "expressions": [
                     "aggregate_labels(meter_mongodb_cluster_connections,sum)"
                   ],
+                  "metricConfig": [
+                    {
+                      "label": "connections"
+                    }
+                  ],
                   "graph": {
                     "type": "Line",
                     "step": false,
@@ -178,7 +183,7 @@
                   "i": "7",
                   "type": "Widget",
                   "expressions": [
-                    "meter_mongodb_cluster_cursor_avg"
+                    "aggregate_labels(meter_mongodb_cluster_cursor_avg, sum(csr_type))"
                   ],
                   "graph": {
                     "type": "Line",
@@ -189,7 +194,7 @@
                     "showYAxis": true
                   },
                   "widget": {
-                    "title": "Cursor Avg"
+                    "title": "Cursor Total"
                   }
                 },
                 {
@@ -222,7 +227,8 @@
                   "i": "9",
                   "type": "Widget",
                   "widget": {
-                    "title": "DB Avg Data Size Per Shard (GB)"
+                    "title": "DB Total Data Size (GB)",
+                    "tips": "Summation according to database primary node"
                   },
                   "graph": {
                     "type": "Table",
@@ -231,7 +237,7 @@
                     "tableHeaderCol2": ""
                   },
                   "expressions": [
-                    "meter_mongodb_cluster_db_data_size/1024/1024/1024"
+                    "latest(aggregate_labels(meter_mongodb_cluster_db_data_size, sum(database)))/1024/1024/1024"
                   ]
                 },
                 {
@@ -242,7 +248,8 @@
                   "i": "10",
                   "type": "Widget",
                   "widget": {
-                    "title": "DB Avg Index Size Per Shard (GB)"
+                    "title": "DB Total Index Size (GB)",
+                    "tips": "Summation according to database primary node"
                   },
                   "graph": {
                     "type": "Table",
@@ -251,7 +258,7 @@
                     "tableHeaderCol2": ""
                   },
                   "expressions": [
-                    "meter_mongodb_cluster_db_index_size/1024/1024/1024"
+                    "latest(aggregate_labels(meter_mongodb_cluster_db_index_size, sum(database)))/1024/1024/1024"
                   ]
                 },
                 {
@@ -262,7 +269,8 @@
                   "i": "11",
                   "type": "Widget",
                   "widget": {
-                    "title": "DB Avg Collection Count Per Shard"
+                    "title": "DB Total Collection Count",
+                    "tips": "Summation according to database primary node"
                   },
                   "graph": {
                     "type": "Table",
@@ -271,7 +279,7 @@
                     "tableHeaderCol2": ""
                   },
                   "expressions": [
-                    "meter_mongodb_cluster_db_collection_count"
+                    "latest(aggregate_labels(meter_mongodb_cluster_db_collection_count, sum(database)))"
                   ]
                 },
                 {
@@ -282,7 +290,8 @@
                   "i": "12",
                   "type": "Widget",
                   "widget": {
-                    "title": "DB Avg Index Count Per Shard"
+                    "title": "DB Total Index Count",
+                    "tips": "Summation according to database primary node"
                   },
                   "graph": {
                     "type": "Table",
@@ -291,7 +300,7 @@
                     "tableHeaderCol2": ""
                   },
                   "expressions": [
-                    "meter_mongodb_cluster_db_index_count"
+                    "latest(aggregate_labels(meter_mongodb_cluster_db_index_count, sum(database)))"
                   ]
                 }
               ]
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-node.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-node.json
index f6f67c930bcc..3595461e03e6 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-node.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/mongodb/mongodb-node.json
@@ -65,7 +65,7 @@
             "title": "Latency"
           },
           "expressions": [
-            "avg(meter_mongodb_node_latency)"
+            "avg(aggregate_labels(meter_mongodb_node_latency_rate,sum)/aggregate_labels(meter_mongodb_node_op_rate,sum))"
           ],
           "metricConfig": [
             {
@@ -148,6 +148,11 @@
           "expressions": [
             "meter_mongodb_node_cpu_total_percentage"
           ],
+          "metricConfig": [
+            {
+              "label": "CPU usage"
+            }
+          ],
           "graph": {
             "type": "Line",
             "step": false,
@@ -271,6 +276,11 @@
           },
           "expressions": [
             "meter_mongodb_node_connections"
+          ],
+          "metricConfig": [
+            {
+              "label": "connections"
+            }
           ]
         },
         {
@@ -431,7 +441,7 @@
             "title": "Operation Latency (µs)"
           },
           "expressions": [
-            "meter_mongodb_node_operation_latency"
+            "aggregate_labels(meter_mongodb_node_latency_rate,sum(op_type))/aggregate_labels(meter_mongodb_node_op_rate,sum(op_type))"
           ]
         },
         {
@@ -523,6 +533,11 @@
           "expressions": [
             "meter_mongodb_node_repl_buffer_count"
           ],
+          "metricConfig": [
+            {
+              "label": "repl buffer count"
+            }
+          ],
           "graph": {
             "type": "Line",
             "step": false,
@@ -633,6 +648,11 @@
           "expressions": [
             "meter_mongodb_node_write_wait_time"
           ],
+          "metricConfig": [
+            {
+              "label": "write wait time"
+            }
+          ],
           "widget": {
             "title": "getLastError Write Time (ms)"
           }
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json
index 198c84f1eb74..8c033c6701cc 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json
@@ -23,6 +23,11 @@
           },
           "expressions": [
             "meter_nginx_endpoint_http_requests"
+          ],
+          "metricConfig": [
+            {
+              "label": "HTTP request trend"
+            }
           ]
         },
         {
@@ -101,6 +106,11 @@
           "expressions": [
             "(meter_nginx_endpoint_http_4xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100"
           ],
+          "metricConfig": [
+            {
+              "label": "HTTP 4xx percent"
+            }
+          ],
           "graph": {
             "type": "Area",
             "opacity": 0.4,
@@ -121,6 +131,11 @@
           "expressions": [
             "(meter_nginx_endpoint_http_5xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100"
           ],
+          "metricConfig": [
+            {
+              "label": "HTTP 5xx percent"
+            }
+          ],
           "graph": {
             "type": "Area",
             "opacity": 0.4,
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json
index 97b55af7bd4d..915c88ed5d01 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json
@@ -23,6 +23,11 @@
           },
           "expressions": [
             "meter_nginx_instance_http_requests"
+          ],
+          "metricConfig": [
+            {
+              "label": "HTTP request trend"
+            }
           ]
         },
         {
@@ -123,6 +128,11 @@
           "expressions": [
             "(meter_nginx_instance_http_4xx_requests_increment/meter_nginx_instance_http_requests_increment)*100"
           ],
+          "metricConfig": [
+            {
+              "label": "HTTP 4xx percent"
+            }
+          ],
           "graph": {
             "type": "Area",
             "opacity": 0.4,
@@ -143,6 +153,11 @@
           "expressions": [
             "(meter_nginx_instance_http_5xx_requests_increment/meter_nginx_instance_http_requests_increment)*100"
           ],
+          "metricConfig": [
+            {
+              "label": "HTTP 5xx percent"
+            }
+          ],
           "graph": {
             "type": "Area",
             "opacity": 0.4,
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
index f009aed5c64a..9585468d9e9d 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
@@ -34,6 +34,11 @@
                   },
                   "expressions": [
                     "aggregate_labels(meter_nginx_service_http_requests, sum)"
+                  ],
+                  "metricConfig": [
+                    {
+                      "label": "HTTP request trend"
+                    }
                   ]
                 },
                 {
@@ -134,6 +139,11 @@
                   "expressions": [
                     "aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100"
                   ],
+                  "metricConfig": [
+                    {
+                      "label": "HTTP 4xx percent"
+                    }
+                  ],
                   "graph": {
                     "type": "Area",
                     "opacity": 0.4,
@@ -154,6 +164,11 @@
                   "expressions": [
                     "aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100"
                   ],
+                  "metricConfig": [
+                    {
+                      "label": "HTTP 5xx percent"
+                    }
+                  ],
                   "graph": {
                     "type": "Area",
                     "opacity": 0.4,


Test patch:
diff --git a/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-csr-service_instance_id.yml b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-csr-service_instance_id.yml
new file mode 100644
index 000000000000..f5696a3bb1f5
--- /dev/null
+++ b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-csr-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: csr_type
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database.yml b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database-service_instance_id.yml
similarity index 94%
rename from test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database.yml
rename to test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database-service_instance_id.yml
index ef06d00a0233..008f18458092 100644
--- a/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database.yml
+++ b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-database-service_instance_id.yml
@@ -21,6 +21,8 @@ results:
         {{- contains .metric.labels }}
         - key: database
           value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
         {{- end}}
     values:
       {{- contains .values }}
diff --git a/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-doc-service_instance_id.yml b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-doc-service_instance_id.yml
new file mode 100644
index 000000000000..d35b7b6311b1
--- /dev/null
+++ b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-doc-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: doc_op_type
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-legacy-service_instance_id.yml b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-legacy-service_instance_id.yml
new file mode 100644
index 000000000000..7dee57e235e1
--- /dev/null
+++ b/test/e2e-v2/cases/mongodb/expected/metrics-has-value-label-legacy-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: legacy_op_type
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/mongodb/mongodb-cases.yaml b/test/e2e-v2/cases/mongodb/mongodb-cases.yaml
index 208aeccdf391..04e5bad406b8 100644
--- a/test/e2e-v2/cases/mongodb/mongodb-cases.yaml
+++ b/test/e2e-v2/cases/mongodb/mongodb-cases.yaml
@@ -28,23 +28,23 @@ cases:
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_object_count --service-name=mongodb::replset
     expected: expected/metrics-has-value-label.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_document_avg_qps --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-doc.yml
+    expected: expected/metrics-has-value-label-doc-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_operation_avg_qps --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-legacy.yml
+    expected: expected/metrics-has-value-label-legacy-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_connections --service-name=mongodb::replset
     expected: expected/metrics-has-value-label-ins.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_cursor_avg --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-csr.yml
+    expected: expected/metrics-has-value-label-csr-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_repl_lag --service-name=mongodb::replset
     expected: expected/metrics-has-value-label.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_db_data_size --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-database.yml
+    expected: expected/metrics-has-value-label-database-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_db_index_size --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-database.yml
+    expected: expected/metrics-has-value-label-database-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_db_collection_count --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-database.yml
+    expected: expected/metrics-has-value-label-database-service_instance_id.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_cluster_db_index_count --service-name=mongodb::replset
-    expected: expected/metrics-has-value-label-database.yml
+    expected: expected/metrics-has-value-label-database-service_instance_id.yml
 
   # instance cases
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql instance ls --service-name=mongodb::replset
@@ -53,8 +53,10 @@ cases:
     expected: expected/metrics-has-value.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_qps --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
     expected: expected/metrics-has-value.yml
-  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_latency --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
-    expected: expected/metrics-has-value.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_op_rate --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
+    expected: expected/metrics-has-value-label-op.yml
+  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_latency_rate --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
+    expected: expected/metrics-has-value-label-op.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_memory_usage --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
     expected: expected/metrics-has-value.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_version --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
@@ -93,8 +95,6 @@ cases:
     expected: expected/metrics-has-value-label-legacy.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_repl_operation_qps --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
     expected: expected/metrics-has-value-label-legacy.yml
-  - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_operation_latency --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
-    expected: expected/metrics-has-value-label-op.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_cursor --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216
     expected: expected/metrics-has-value-label-csr.yml
   - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_mongodb_node_mem_virtual --service-name=mongodb::replset --instance-name=mongodb-exporter-1:9216