Repository: apache/skywalking (language: python)

Issue:
[Bug] Metrics are not aggregated properly as OTEL collector targeting per node
### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.


### Apache SkyWalking Component

OAP server (apache/skywalking)

### What happened

This issue was found during Clickhouse PR review, https://github.com/apache/skywalking/pull/11966#pullrequestreview-1912778007

We noticed this is a common bug case in previous monitoring. We should check them and update the status here.

- [x] MongoDB @weixiang1862 https://github.com/apache/skywalking/pull/12126
- [x] MySQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] MariaDB @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] PostgreSQL @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Redis @yswdqz https://github.com/apache/skywalking/pull/12130
- [x] Elasticsearch @innerpeacez 
- [x] Nginx @weixiang1862 https://github.com/apache/skywalking/pull/12096
- [x] APISIX @pg-yang https://github.com/apache/skywalking/pull/12108
- [x] Kafka @wankai123 
- [x] Pulsar @CodePrometheus https://github.com/apache/skywalking/pull/12174
- [x] RabbitMQ @innerpeacez https://github.com/apache/skywalking/pull/12144

### What you expected to happen

Service metrics should be labeled values, and only use MQE(query stage aggregation) to merge the data to build metrics accurately. 

In this case, if there are two nodes

In the case A
ck node1 -> OTEL -> OAP
ck node2-> OTEL -> OAP

In the case B
T1, ck node1 -> OTEL -> OAP
T2, ck node1 -> OTEL -> OAP
(T1 T2) are in the same minute(same for an hour/day)

Then from the OAP process perspective, it can't tell A and B, so, if you work on downsampling(default as AVG), it could be AVG of node1 and node2(case A), or AVG of ck node1 from T1 and T2(case B). 
For a more complex case, it could be a combination of case A and B, in one minute, N scrapings from node1 and M scrapings from node2, then the default sampling is doing AVG for N + M scrapings from node1 and node2. This actually is not the value of the aggregated metric for the whole service(multiple nodes).

### How to reproduce

Metrics from nodeA and nodeB will overlap with each other. Due to nodeA and nodeB will report data separately.

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)



Discussion / hints:
@CodePrometheus If you could help this, please let me know.
For all the owners, this list doesn't mean all monitoring at service level is inaccurate due to the way od scraping from OTEL. Please recheck first and only fix if it fits the bug descriptions.
ES checked, the metrics data from one cluster-level API.
Kafka monitoring service's MAL aggregates the metrics with `cluster and broker`, so it's good.
To build service level metrics with instance/node level metrics through `AggregateLabels Operation`,  I think `AggregateLabels`  should support aggregation through specified labels. Like the scenario below:

#### instance/node HTTP status metrics

```
# The number means the occurred times
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```

The MQE express to aggregate service level metrics should be  

```
aggregate_labels(http_status,'status',sum) # aggregate by status label
```


The result will be

```
http_status{status=401} 8

```


I don't think we support multiple labels after you have done the persistent.

```
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2
```
This is not a proper example, because OAP doesn't persistent data like this. 
In your case, the data in the DB should be `http_status_401{192.168.0.1=5, 192.168.0.3=1, 192.168.0.2=2}`. Please verify.
@weixiang1862 As you asked about, we posted https://github.com/apache/skywalking/issues/11992 for v10. 
@wu-sheng I would be willing, of course! I'll do recheck on this weekend.
> @wu-sheng I would be willing, of course! I'll do recheck on this weekend.

Which one will you check? Let's avoid duplicate works with others.
@wu-sheng I'll try to check pulsar first.
Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?
> > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> 
> The question is how to aggregate the metrics of the cluster?

The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.
> > > Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.
> > 
> > 
> > The question is how to aggregate the metrics of the cluster?
> 
> The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.

OK. Then it is fine for now. But you could start a new issue about polishing the Pulsar monitoring as we discussed.
@pg-yang How is the APISIX? The metrics on the dashboard seem not correctly because they have cross-instance aggregation.
Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters
> Yes, it's incorrect. I'm waiting for `aggregate_labels ` with label parameters

Got it. That will need another one or two weeks.
@weixiang1862 @pg-yang @innerpeacez We have the new multi-label support merged.  You should be good to fix this.
Rabbitmq server metrics are accurate（instance level used node tag）, thx for your help @wankai123 again.

Fix patch:
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 03aa9ab5bc56..7f91d6a8cdad 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -113,6 +113,7 @@
 * Calculate Nginx service HTTP Latency by MQE.
 * MQE query: make metadata not return `null`.
 * MQE labeled metrics Binary Operation: return empty value if the labels not match rather than report error.
+* Fix inaccurate Hierarchy of RabbitMQ Server monitoring metrics.
 
 #### UI
 
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rabbitmq/rabbitmq-cluster.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rabbitmq/rabbitmq-cluster.json
index af09eff594c9..8706a32e1fb6 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rabbitmq/rabbitmq-cluster.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rabbitmq/rabbitmq-cluster.json
@@ -530,11 +530,11 @@
       "isRoot": false,
       "isDefault": true,
       "expressions": [
-        "avg(meter_rabbitmq_messages_published)",
-        "avg(meter_rabbitmq_messages_unconfirmed)",
-        "avg(meter_rabbitmq_queues)",
-        "avg(meter_rabbitmq_channels)",
-        "avg(meter_rabbitmq_connections)"
+        "aggregate_labels(meter_rabbitmq_messages_published,sum)",
+        "aggregate_labels(meter_rabbitmq_messages_unconfirmed,sum)",
+        "aggregate_labels(meter_rabbitmq_queues,sum)",
+        "aggregate_labels(meter_rabbitmq_channels,sum)",
+        "aggregate_labels(meter_rabbitmq_connections,sum)"
       ],
       "expressionsConfig": [
         {


Test patch:
diff --git a/test/e2e-v2/cases/rabbitmq/otel-collector-config.yaml b/test/e2e-v2/cases/rabbitmq/otel-collector-config.yaml
index 2b01c217b194..6802f3cc6940 100644
--- a/test/e2e-v2/cases/rabbitmq/otel-collector-config.yaml
+++ b/test/e2e-v2/cases/rabbitmq/otel-collector-config.yaml
@@ -68,6 +68,8 @@ exporters:
     endpoint: oap:11800
     tls:
       insecure: true
+  logging:
+    loglevel: debug
 
 service:
   pipelines:
@@ -77,4 +79,5 @@ service:
       processors:
         - batch
       exporters:
-        - otlp
\ No newline at end of file
+        - otlp
+        - logging
\ No newline at end of file