Repository: apache/skywalking (language: python)

Issue:
[Bug] Metrics are not aggregated properly as OTEL collector targeting per node
### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.


### Apache SkyWalking Component

OAP server (apache/skywalking)

### What happened

This issue was found during Clickhouse PR review, https://github.com/apache/skywalking/pull/11966#pullrequestreview-1912778007

We noticed this is a common bug case in previous monitoring. We should check them and update the status here.

- [x] MongoDB @weixiang1862 https://github.com/apache/skywalking/pull/12126
- [x] MySQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] MariaDB @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] PostgreSQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Redis @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Elasticsearch @innerpeacez 
- [x] Nginx @weixiang1862 https://github.com/apache/skywalking/pull/12096
- [x] APISIX @pg-yang https://github.com/apache/skywalking/pull/12108
- [x] Kafka @wankai123 
- [x] Pulsar @CodePrometheus https://github.com/apache/skywalking/pull/12174
- [x] RabbitMQ @innerpeacez https://github.com/apache/skywalking/pull/12144

### What you expected to happen

Service metrics should be labeled values, and only use MQE(query stage aggregation) to merge the data to build metrics accurately. 

In this case, if there are two nodes

In the case A
ck node1 -> OTEL -> OAP
ck node2-> OTEL -> OAP

In the case B
T1, ck node1 -> OTEL -> OAP
T2, ck node1 -> OTEL -> OAP
(T1 T2) are in the same minute(same for an hour/day)

Then from the OAP process perspective, it can't tell A and B, so, if you work on downsampling(default as AVG), it could be AVG of node1 and node2(case A), or AVG of ck node1 from T1 and T2(case B). 
For a more complex case, it could be a combination of case A and B, in one minute, N scrapings from node1 and M scrapings from node2, then the default sampling is doing AVG for N + M scrapings from node1 and node2. This actually is not the value of the aggregated metric for the whole service(multiple nodes).

### How to reproduce

Metrics from nodeA and nodeB will overlap with each other. Due to nodeA and nodeB will report data separately.

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)



Discussion / hints:
@CodePrometheus If you could help this, please let me know.
For all the owners, this list doesn't mean all monitoring at service level is inaccurate due to the way od scraping from OTEL. Please recheck first and only fix if it fits the bug descriptions.
ES checked, the metrics data from one cluster-level API.
Kafka monitoring service's MAL aggregates the metrics with `cluster and broker`, so it's good.
To build service level metrics with instance/node level metrics through `AggregateLabels Operation`,  I think `AggregateLabels`  should support aggregation through specified labels. Like the scenario below:

#### instance/node HTTP status metrics

```
# The number means the occurred times
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```

The MQE express to aggregate service level metrics should be  

```
aggregate_labels(http_status,'status',sum) # aggregate by status label
```


The result will be

```
http_status{status=401} 8

```


I don't think we support multiple labels after you have done the persistent.

```
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```
This is not a proper example, because OAP doesn't persistent data like this. 
In your case, the data in the DB should be `http_status_401{192.168.0.1=5, 192.168.0.3=1, 192.168.0.2=2}`. Please verify.
@weixiang1862 As you asked about, we posted https://github.com/apache/skywalking/issues/11992 for v10. 
@wu-sheng I would be willing, of course! I'll do recheck on this weekend.
> @wu-sheng I would be willing, of course! I'll do recheck on this weekend.

Which one will you check? Let's avoid duplicate works with others.
@wu-sheng I'll try to check pulsar first.
Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?
> > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> 
> The question is how to aggregate the metrics of the cluster?

The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.
> > > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> > 
> > 
> > The question is how to aggregate the metrics of the cluster?
> 
> The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.

OK. Then it is fine for now. But you could start a new issue about polishing the Pulsar monitoring as we discussed.
@pg-yang How is the APISIX? The metrics on the dashboard seem not correctly because they have cross-instance aggregation.
Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters
> Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters

Got it. That will need another one or two weeks.
@weixiang1862 @pg-yang @innerpeacez We have the new multi-label support merged.  You should be good to fix this.

Fix patch:
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index e6cb16b541da..8de4efebbcd7 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -98,6 +98,7 @@
 * Remove unnecessary `componentIds` as series ID in the `ServiceRelationClientSideMetrics` and `ServiceRelationServerSideMetrics` entities.
 * Fix not throw error when part of expression not matched any expression node in the `MQE` and `PromQL.
 * Remove `kafka-fetcher/default/createTopicIfNotExist` as the creation is automatically since [#7326](https://github.com/apache/skywalking/issues/7326) (v8.7.0).
+* Fix inaccuracy nginx service metrics.
 
 #### UI
 
diff --git a/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml b/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml
index 380b43bd46b6..92f847fd7a4a 100644
--- a/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml
+++ b/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml
@@ -31,6 +31,6 @@
 metricPrefix: meter_nginx
 metricsRules:
   - name: service_error_log_count
-    exp: nginx_error_log_count.sum(['level','service']).downsampling(SUM).service(['service'], Layer.NGINX)
+    exp: nginx_error_log_count.sum(['level','service','service_instance_id']).downsampling(SUM).service(['service'], Layer.NGINX)
   - name: instance_error_log_count
     exp: nginx_error_log_count.sum(['level','service','service_instance_id']).downsampling(SUM).instance(['service'],['service_instance_id'], Layer.NGINX)
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml
index cb51df9a4021..e46cf98e07ed 100644
--- a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml
@@ -36,7 +36,7 @@ metricsRules:
   - name: http_requests
     exp: nginx_http_requests_total.sum(['service','route']).rate('PT1M')
   - name: http_latency
-    exp: nginx_http_latency.sum(['type','le','service','route']).histogram().histogram_percentile([50,70,90,99])
+    exp: nginx_http_latency.sum(['le','service','route']).histogram().histogram_percentile([50,70,90,99])
   - name: http_bandwidth
     exp: nginx_http_size_bytes.sum(['type','service','route']).rate('PT1M')
   - name: http_status
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml
index b72a4697068f..765c3fc33d23 100644
--- a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml
@@ -36,7 +36,7 @@ metricsRules:
   - name: http_requests
     exp: nginx_http_requests_total.sum(['service','service_instance_id']).rate('PT1M')
   - name: http_latency
-    exp: nginx_http_latency.sum(['type','le','service','service_instance_id']).histogram().histogram_percentile([50,70,90,99])
+    exp: nginx_http_latency.sum(['le','service','service_instance_id']).histogram().histogram_percentile([50,70,90,99])
   - name: http_bandwidth
     exp: nginx_http_size_bytes.sum(['type','service','service_instance_id']).rate('PT1M')
   - name: http_connections
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml
index 29dac9072ae4..78d82f9d3774 100644
--- a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml
@@ -34,18 +34,18 @@ expSuffix: service(['service'] , Layer.NGINX)
 metricPrefix: meter_nginx_service
 metricsRules:
   - name: http_requests
-    exp: nginx_http_requests_total.sum(['service']).rate('PT1M')
+    exp: nginx_http_requests_total.sum(['service','service_instance_id']).rate('PT1M')
   - name: http_latency
-    exp: nginx_http_latency.sum(['type','le','service']).histogram().histogram_percentile([50,70,90,99])
+    exp: nginx_http_latency.sum(['le','service']).histogram().histogram_percentile([50,70,90,99])
   - name: http_bandwidth
-    exp: nginx_http_size_bytes.sum(['type','service']).rate('PT1M')
+    exp: nginx_http_size_bytes.sum(['type','service','service_instance_id']).rate('PT1M')
   - name: http_connections
-    exp: nginx_http_connections.sum(['state','service'])
+    exp: nginx_http_connections.sum(['state','service','service_instance_id'])
   - name: http_status
-    exp: nginx_http_requests_total.sum(['status','service']).rate('PT1M')
+    exp: nginx_http_requests_total.sum(['status','service','service_instance_id']).rate('PT1M')
   - name: http_requests_increment
-    exp: nginx_http_requests_total.sum(['service']).increase('PT1M')
+    exp: nginx_http_requests_total.sum(['service','service_instance_id']).increase('PT1M')
   - name: http_4xx_requests_increment
-    exp: nginx_http_requests_total.tagMatch("status", "400|401|403|404|405").sum(['service']).increase('PT1M')
+    exp: nginx_http_requests_total.tagMatch("status", "400|401|403|404|405").sum(['service','service_instance_id']).increase('PT1M')
   - name: http_5xx_requests_increment
-    exp: nginx_http_requests_total.tagMatch("status", "500|502|503|504").sum(['service']).increase('PT1M')
\ No newline at end of file
+    exp: nginx_http_requests_total.tagMatch("status", "500|502|503|504").sum(['service','service_instance_id']).increase('PT1M')
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
index 317f7fa1998e..ced705bad1fc 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
@@ -33,7 +33,7 @@
                     "showYAxis": true
                   },
                   "expressions": [
-                    "meter_nginx_service_http_requests"
+                    "aggregate_labels(meter_nginx_service_http_requests, sum)"
                   ],
                   "metricMode": "Expression",
                   "typesOfMQE": [
@@ -85,7 +85,7 @@
                     "showYAxis": true
                   },
                   "expressions": [
-                    "meter_nginx_service_http_bandwidth/1024"
+                    "aggregate_labels(meter_nginx_service_http_bandwidth, sum(type))/1024"
                   ],
                   "metricMode": "Expression",
                   "typesOfMQE": [
@@ -100,7 +100,7 @@
                   "i": "4",
                   "type": "Widget",
                   "expressions": [
-                    "meter_nginx_service_http_connections"
+                    "aggregate_labels(meter_nginx_service_http_connections, sum(state))"
                   ],
                   "metricMode": "Expression",
                   "typesOfMQE": [
@@ -137,7 +137,7 @@
                     "showYAxis": true
                   },
                   "expressions": [
-                    "meter_nginx_service_http_status"
+                    "aggregate_labels(meter_nginx_service_http_status, sum(status))"
                   ],
                   "metricMode": "Expression",
                   "typesOfMQE": [
@@ -153,7 +153,7 @@
                   "type": "Widget",
                   "metricMode": "Expression",
                   "expressions": [
-                    "(meter_nginx_service_http_4xx_requests_increment/meter_nginx_service_http_requests_increment)*100"
+                    "aggregate_labels(meter_nginx_service_http_4xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100"
                   ],
                   "typesOfMQE": [
                     "TIME_SERIES_VALUES"
@@ -177,7 +177,7 @@
                   "type": "Widget",
                   "metricMode": "Expression",
                   "expressions": [
-                    "(meter_nginx_service_http_5xx_requests_increment/meter_nginx_service_http_requests_increment)*100"
+                    "aggregate_labels(meter_nginx_service_http_5xx_requests_increment, sum)/aggregate_labels(meter_nginx_service_http_requests_increment, sum)*100"
                   ],
                   "typesOfMQE": [
                     "TIME_SERIES_VALUES"
@@ -201,7 +201,7 @@
                   "type": "Widget",
                   "metricMode": "Expression",
                   "expressions": [
-                    "meter_nginx_service_error_log_count"
+                    "aggregate_labels(meter_nginx_service_error_log_count, sum(level))"
                   ],
                   "graph": {
                     "type": "Bar",
@@ -290,9 +290,9 @@
       "isRoot": false,
       "isDefault": true,
       "expressions": [
-        "avg(meter_nginx_service_http_status{status='200'})",
-        "avg(meter_nginx_service_http_status{status='404'})",
-        "avg(meter_nginx_service_http_status{status='500'})"
+        "avg(aggregate_labels(meter_nginx_service_http_status{status='200'}, sum))",
+        "avg(aggregate_labels(meter_nginx_service_http_status{status='404'}, sum))",
+        "avg(aggregate_labels(meter_nginx_service_http_status{status='500'}, sum))"
       ],
       "expressionsConfig": [
         {


Test patch:
diff --git a/test/e2e-v2/cases/nginx/e2e.yaml b/test/e2e-v2/cases/nginx/e2e.yaml
index 8fcfa49831eb..882354f229ee 100644
--- a/test/e2e-v2/cases/nginx/e2e.yaml
+++ b/test/e2e-v2/cases/nginx/e2e.yaml
@@ -47,19 +47,19 @@ verify:
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls
       expected: expected/service.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_requests --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value.yml
+      expected: expected/metrics-has-value-label-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_latency --service-name=nginx::e2e-test
       expected: expected/metrics-has-value-label-percentile.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_bandwidth --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value-label-type.yml
+      expected: expected/metrics-has-value-label-type-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_connections --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value-label-state.yml
+      expected: expected/metrics-has-value-label-state-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_status --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value-label-status.yml
+      expected: expected/metrics-has-value-label-status-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_requests_increment --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value.yml
+      expected: expected/metrics-has-value-label-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_error_log_count --service-name=nginx::e2e-test
-      expected: expected/metrics-has-value-label-level.yml
+      expected: expected/metrics-has-value-label-level-service_instance_id.yml
     - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs list --service-name=nginx::e2e-test
       expected: expected/logs.yml
 
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-level-service_instance_id.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-level-service_instance_id.yml
new file mode 100644
index 000000000000..d88b230a02d8
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-level-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: level
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-service_instance_id.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-service_instance_id.yml
new file mode 100644
index 000000000000..3d71dad534ca
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-service_instance_id.yml
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-state-service_instance_id.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-state-service_instance_id.yml
new file mode 100644
index 000000000000..73aed12ac03f
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-state-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: state
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-status-service_instance_id.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-status-service_instance_id.yml
new file mode 100644
index 000000000000..c041fc9390ec
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-status-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: status
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-type-service_instance_id.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-type-service_instance_id.yml
new file mode 100644
index 000000000000..0ae0af9cc746
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label-type-service_instance_id.yml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: type
+          value: {{ .value }}
+        - key: service_instance_id
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null