Repository: open-telemetry/opentelemetry-collector-contrib (Go)

Issue:
[receiver/azuremonitorreceiver] ``metrics`` aggregation filter unused and slightly with current ``use_batch_api:true`` implementation
### Component(s)

receiver/azuremonitor

### What happened?

## Description
From the following thread https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/38895#discussion_r2087719777

There are two issues that should be fixed at once:
- **First** The metrics aggregation filter feature has been well backported to the scraper_batch, to add [aggregation into composite key](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/azuremonitorreceiver/scraper_batch.go#L375-L379) but the composite key is not used in [the query to get metrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/azuremonitorreceiver/scraper_batch.go#L441-L447) 😲
- **Second**, there is arbitrarily chosen ``if metricValue.Average != nil {`` before processing the metrics, which is obviously unwanted now that we can filter out the metrics. The reason of that choice and possible solutions are explained in the thread.


## Steps to Reproduce
Special thanks to @andrewegel with this perfect reproduction #40078 
I'm copy pasting it there
```yaml
azuremonitor:
    auth: default_credentials
    subscription_ids: [...one-subscription...]
    use_batch_api: true
    append_tags_as_attributes: true
    services:
    - "Microsoft.Network/loadBalancers"
    - "Microsoft.Network/natgateways"
    metrics:
      "Microsoft.Network/loadBalancers":
        "VipAvailability": [average]
        "DipAvailability": [average]
        "ByteCount": [total]
        "SYNCount": [count, total]
        "SnatConnectionCount": [total]
        "AllocatedSnatPorts": [average]
        "UsedSnatPorts": [average]
      "Microsoft.Network/natgateways":
        "SNATConnectionCount": [total]
        "ByteCount": [total]
        "DatapathAvailability": [average]
    collection_interval: 300s
    initial_delay: 1s
```

## Expected Result

We expect the metrics for which you don't have average in the filter (e,g ByteCount, SYNCount, ...) to appear in the result.

## Actual Result

Only metrics that does have "average" in their filter appear in the result.

### Collector version

v0.126.0

### Environment information

## Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")


### OpenTelemetry Collector configuration

```yaml

```

### Log output

```shell

```

### Additional context

_No response_


Discussion:
Pinging code owners:
- receiver/azuremonitor: @nslaughter @celian-garcia

 See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself.

/labels -needs-triage help-wanted good-first-issue
/labels never-stale
Well considered, I will check it myself, as I have a good understanding of the problem and it's easy to unblock.
/labels -help-wanted




Patch:
diff --git a/.chloggen/celian-garcia_fix-40079.yaml b/.chloggen/celian-garcia_fix-40079.yaml
new file mode 100644
index 0000000000000..97d5e7afb8f5d
--- /dev/null
+++ b/.chloggen/celian-garcia_fix-40079.yaml
@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: bug_fix
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: azuremonitorreceiver
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: "use ``metrics`` aggregation filter when ``use_batch_api: true``"
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [40079]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []
diff --git a/receiver/azuremonitorreceiver/scraper_batch.go b/receiver/azuremonitorreceiver/scraper_batch.go
index a220ff70e1b4e..15dedaadd5012 100644
--- a/receiver/azuremonitorreceiver/scraper_batch.go
+++ b/receiver/azuremonitorreceiver/scraper_batch.go
@@ -441,6 +441,7 @@ func (s *azureBatchScraper) getBatchMetricsValues(ctx context.Context, subscript
 					opts := newQueryResourcesOptions(
 						compositeKey.dimensions,
 						compositeKey.timeGrain,
+						compositeKey.aggregations,
 						startTime,
 						now,
 						s.cfg.MaximumNumberOfRecordsPerResource,
@@ -493,7 +494,7 @@ func (s *azureBatchScraper) getBatchMetricsValues(ctx context.Context, subscript
 								attributes["timegrain"] = &compositeKey.timeGrain
 								for i := len(timeseriesElement.Data) - 1; i >= 0; i-- { // reverse for loop because newest timestamp is at the end of the slice
 									metricValue := timeseriesElement.Data[i]
-									if metricValue.Average != nil {
+									if metricValueIsNotEmpty(metricValue) {
 										s.processQueryTimeseriesData(resID, metric, metricValue, attributes)
 										break
 									}
@@ -513,29 +514,28 @@ func (s *azureBatchScraper) getBatchMetricsValues(ctx context.Context, subscript
 func newQueryResourcesOptions(
 	dimensionsStr string,
 	timeGrain string,
+	aggregationsStr string,
 	start time.Time,
 	end time.Time,
 	top int32,
 ) azmetrics.QueryResourcesOptions {
 	return azmetrics.QueryResourcesOptions{
-		Aggregation: to.Ptr(strings.Join(
-			[]string{
-				string(armmonitor.AggregationTypeAverage),
-				string(armmonitor.AggregationTypeMaximum),
-				string(armmonitor.AggregationTypeMinimum),
-				string(armmonitor.AggregationTypeTotal),
-				string(armmonitor.AggregationTypeCount),
-			},
-			",",
-		)),
-		StartTime: to.Ptr(start.Format(time.RFC3339)),
-		EndTime:   to.Ptr(end.Format(time.RFC3339)),
-		Interval:  to.Ptr(timeGrain),
-		Top:       to.Ptr(top), // Defaults to 10 (may be limiting results)
-		Filter:    buildDimensionsFilter(dimensionsStr),
+		Aggregation: to.Ptr(aggregationsStr),
+		StartTime:   to.Ptr(start.Format(time.RFC3339)),
+		EndTime:     to.Ptr(end.Format(time.RFC3339)),
+		Interval:    to.Ptr(timeGrain),
+		Top:         to.Ptr(top), // Defaults to 10 (may be limiting results)
+		Filter:      buildDimensionsFilter(dimensionsStr),
 	}
 }
 
+// metricValueIsNotEmpty checks if the metric value is empty.
+// This is necessary to compensate for the fact that Azure Monitor sometimes returns empty values.
+func metricValueIsNotEmpty(metricValue azmetrics.MetricValue) bool {
+	// Using an "or" chain is a bet on performance improvement. Assuming that it's not checking others if one is not nil. Not strictly verified though.
+	return metricValue.Average != nil || metricValue.Count != nil || metricValue.Maximum != nil || metricValue.Minimum != nil || metricValue.Total != nil
+}
+
 func (s *azureBatchScraper) processQueryTimeseriesData(
 	resourceID string,
 	metric azmetrics.Metric,


Test patch:
diff --git a/receiver/azuremonitorreceiver/scraper_batch_test.go b/receiver/azuremonitorreceiver/scraper_batch_test.go
index 907a18301eac3..215bc58eab2e0 100644
--- a/receiver/azuremonitorreceiver/scraper_batch_test.go
+++ b/receiver/azuremonitorreceiver/scraper_batch_test.go
@@ -86,12 +86,15 @@ func getMetricsQueryResponseMockData() []QueryResourcesResponseMock {
 									{
 										Data: []azmetrics.MetricValue{
 											{
+												// Send only timestamp with all other values nil is a case that can
+												// happen in the Azure responses.
 												TimeStamp: to.Ptr(time.Now()),
-												Average:   to.Ptr(1.),
-												Count:     to.Ptr(1.),
-												Maximum:   to.Ptr(1.),
-												Minimum:   to.Ptr(1.),
-												Total:     to.Ptr(1.),
+											},
+											{
+												TimeStamp: to.Ptr(time.Now()),
+												// Keep only Total to make sure that all values are considered.
+												// Not only Average
+												Total: to.Ptr(1.),
 											},
 										},
 									},
diff --git a/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_golden.yaml b/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_golden.yaml
index e252596309eb4..838c0bbc6a967 100644
--- a/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_golden.yaml
+++ b/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_golden.yaml
@@ -412,110 +412,6 @@ resourceMetrics:
             stringValue: fake-tenant-id
     scopeMetrics:
       - metrics:
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_maximum
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_minimum
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_average
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_count
-            unit: BitsPerSecond
           - gauge:
               dataPoints:
                 - asDouble: 1
diff --git a/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_tags_golden.yaml b/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_tags_golden.yaml
index 1df71d4ec1d2a..2a6e60c63ab9e 100644
--- a/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_tags_golden.yaml
+++ b/receiver/azuremonitorreceiver/testdata/expected_metrics_batch/metrics_tags_golden.yaml
@@ -468,110 +468,6 @@ resourceMetrics:
                   timeUnixNano: "1000000"
             name: azure_metric7_total
             unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_maximum
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_minimum
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_average
-            unit: BitsPerSecond
-          - gauge:
-              dataPoints:
-                - asDouble: 1
-                  attributes:
-                    - key: azuremonitor.resource_id
-                      value:
-                        stringValue: /subscriptions/subscriptionId3/resourceGroups/group1/resourceId1
-                    - key: location
-                      value:
-                        stringValue: location1
-                    - key: name
-                      value:
-                        stringValue: name1
-                    - key: resource_group
-                      value:
-                        stringValue: group1
-                    - key: timegrain
-                      value:
-                        stringValue: PT1M
-                    - key: type
-                      value:
-                        stringValue: type1
-                  startTimeUnixNano: "2000000"
-                  timeUnixNano: "1000000"
-            name: azure_metric7_count
-            unit: BitsPerSecond
         scope:
           name: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/azuremonitorreceiver
           version: latest