Repository: open-telemetry/opentelemetry-python

Issue:
OTLPMetricExporter fails to send more than 4MB of data
**Describe your environment**
Python 3.6.8
Opentelemetry Python 1.12.0
Opentelemetry Collector 0.38.0

**Steps to reproduce**
Metrics SDK with quite a few observables generating more than 4MB of data (datapoints)

**What is the expected behavior?**
The datapoints are sent to the collector without any problem

**What is the actual behavior?**
The export to the collector fails with StatusCode.RESOURCE_EXHAUSTED
The exporter keeps on sending the same batch over and over until the data gets dropped

**Additional context**
One solutions would by to have a configurable "max batch size" in the OTLPMetricExporter, like there is today in the BatchLogProcessor for logs.
Another solution would be for the OTLP gRPC exporter to automatically retry with a smaller batch if it receives a StatusCode.RESOURCE_EXHAUSTED ?



Discussion:
Thanks for trying out the metrics SDK 🙂 

The difference between metrics and trace/logs here is that all the metrics come in at once and there is no batching in the SDK. It is simply evaluating all of observable instruments and it's causing the issues. Do folks think we should add this batching mechanism to the `PeriodicExportingMetricReader` so it can buffer spans into the exporter?

> Another solution would be for the OTLP gRPC exporter to automatically retry with a smaller batch if it receives a StatusCode.RESOURCE_EXHAUSTED ?

+1, there is this proposal https://github.com/open-telemetry/opentelemetry-proto/pull/390 which I believe would make this work?
> Do folks think we should add this batching mechanism to the PeriodicExportingMetricReader so it can buffer spans into the exporter?

How does this help? The volume of data that reaches exporter would still be of same size for each export cycle, right?

As a place to configure the batch size, and then the `PeriodicExportingMetricReader` will call the exporter once per batch. @srikanthccv would you prefer to just have the individual exporters handle batching on their own?
Yes, I think we already do this in some exporters which take care of some protocol/coding specific limits. I would like to hear more the batching in metrics. I am trying to understand if each `collect` only limits the number of points collected and then call the exporter or the collection get all data once and then calls exporter multiple times? 
We briefly discussed this today. We should look at the spec for what is the correct status code for errors related to payload size and see if the response from collector includes what's the acceptable size to devide the batch to chunks before exporting.
It would be great to have this kind of behavior but I think we should also be able to configure a "max batch size" in the OTLPMetricExporter.
If at each interval I generate 5000 data-points for a size of 6MB I don't want the first export request to fail every time.
The automatic downsizing of the batch when receiving StatusCode.RESOURCE_EXHAUSTED would be great for sporadic errors.
> It would be great to have this kind of behavior but I think we should also be able to configure a "max batch size" in the OTLPMetricExporter.

@overmeulen any chance you'd be willing to send a PR for this?
Sure. So the idea would be to do the fix directly in the gRPC exporter ?
https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py
Thanks, I'll assign you the issue. We haven't implemented the HTTP exporter yet, so that seems reasonable to me.
Is there some env or something similar spec'd to configure this max batch size?
I was thinking of doing something similar to BatchLogProcessor
https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/_logs/export/__init__.py#L146

Resolving patch:
diff --git a/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py b/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py
--- a/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py
+++ b/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py
@@ -11,9 +11,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import dataclasses
 from logging import getLogger
 from os import environ
-from typing import Dict, Optional, Sequence
+from typing import Dict, Iterable, List, Optional, Sequence
 from grpc import ChannelCredentials, Compression
 from opentelemetry.sdk.metrics._internal.aggregation import Aggregation
 from opentelemetry.exporter.otlp.proto.grpc.exporter import (
@@ -42,12 +43,15 @@
 )
 from opentelemetry.sdk.metrics.export import (
     AggregationTemporality,
+    DataPointT,
     Gauge,
     Histogram as HistogramType,
     Metric,
     MetricExporter,
     MetricExportResult,
     MetricsData,
+    ResourceMetrics,
+    ScopeMetrics,
     Sum,
 )
 
@@ -58,6 +62,14 @@ class OTLPMetricExporter(
     MetricExporter,
     OTLPExporterMixin[Metric, ExportMetricsServiceRequest, MetricExportResult],
 ):
+    """OTLP metric exporter
+
+    Args:
+        max_export_batch_size: Maximum number of data points to export in a single request. This is to deal with
+            gRPC's 4MB message size limit. If not set there is no limit to the number of data points in a request.
+            If it is set and the number of data points exceeds the max, the request will be split.
+    """
+
     _result = MetricExportResult
     _stub = MetricsServiceStub
 
@@ -71,6 +83,7 @@ def __init__(
         compression: Optional[Compression] = None,
         preferred_temporality: Dict[type, AggregationTemporality] = None,
         preferred_aggregation: Dict[type, Aggregation] = None,
+        max_export_batch_size: Optional[int] = None,
     ):
 
         if insecure is None:
@@ -122,6 +135,8 @@ def __init__(
             compression=compression,
         )
 
+        self._max_export_batch_size: Optional[int] = max_export_batch_size
+
     def _translate_data(
         self, data: MetricsData
     ) -> ExportMetricsServiceRequest:
@@ -223,8 +238,9 @@ def _translate_data(
                             )
                             pb2_metric.sum.data_points.append(pt)
                     else:
-                        _logger.warn(
-                            "unsupported datapoint type %s", metric.point
+                        _logger.warning(
+                            "unsupported data type %s",
+                            metric.data.__class__.__name__,
                         )
                         continue
 
@@ -245,7 +261,101 @@ def export(
         **kwargs,
     ) -> MetricExportResult:
         # TODO(#2663): OTLPExporterMixin should pass timeout to gRPC
-        return self._export(metrics_data)
+        if self._max_export_batch_size is None:
+            return self._export(data=metrics_data)
+
+        export_result = MetricExportResult.SUCCESS
+
+        for split_metrics_data in self._split_metrics_data(metrics_data):
+            split_export_result = self._export(data=split_metrics_data)
+
+            if split_export_result is MetricExportResult.FAILURE:
+                export_result = MetricExportResult.FAILURE
+
+        return export_result
+
+    def _split_metrics_data(
+        self,
+        metrics_data: MetricsData,
+    ) -> Iterable[MetricsData]:
+        batch_size: int = 0
+        split_resource_metrics: List[ResourceMetrics] = []
+
+        for resource_metrics in metrics_data.resource_metrics:
+            split_scope_metrics: List[ScopeMetrics] = []
+            split_resource_metrics.append(
+                dataclasses.replace(
+                    resource_metrics,
+                    scope_metrics=split_scope_metrics,
+                )
+            )
+            for scope_metrics in resource_metrics.scope_metrics:
+                split_metrics: List[Metric] = []
+                split_scope_metrics.append(
+                    dataclasses.replace(
+                        scope_metrics,
+                        metrics=split_metrics,
+                    )
+                )
+                for metric in scope_metrics.metrics:
+                    split_data_points: List[DataPointT] = []
+                    split_metrics.append(
+                        dataclasses.replace(
+                            metric,
+                            data=dataclasses.replace(
+                                metric.data,
+                                data_points=split_data_points,
+                            ),
+                        )
+                    )
+
+                    for data_point in metric.data.data_points:
+                        split_data_points.append(data_point)
+                        batch_size += 1
+
+                        if batch_size >= self._max_export_batch_size:
+                            yield MetricsData(
+                                resource_metrics=split_resource_metrics
+                            )
+                            # Reset all the variables
+                            batch_size = 0
+                            split_data_points = []
+                            split_metrics = [
+                                dataclasses.replace(
+                                    metric,
+                                    data=dataclasses.replace(
+                                        metric.data,
+                                        data_points=split_data_points,
+                                    ),
+                                )
+                            ]
+                            split_scope_metrics = [
+                                dataclasses.replace(
+                                    scope_metrics,
+                                    metrics=split_metrics,
+                                )
+                            ]
+                            split_resource_metrics = [
+                                dataclasses.replace(
+                                    resource_metrics,
+                                    scope_metrics=split_scope_metrics,
+                                )
+                            ]
+
+                    if not split_data_points:
+                        # If data_points is empty remove the whole metric
+                        split_metrics.pop()
+
+                if not split_metrics:
+                    # If metrics is empty remove the whole scope_metrics
+                    split_scope_metrics.pop()
+
+            if not split_scope_metrics:
+                # If scope_metrics is empty remove the whole resource_metrics
+                split_resource_metrics.pop()
+
+        if batch_size > 0:
+            yield MetricsData(resource_metrics=split_resource_metrics)
 
     def shutdown(self, timeout_millis: float = 30_000, **kwargs) -> None:
         pass


Test patch:
diff --git a/exporter/opentelemetry-exporter-otlp-proto-grpc/tests/metrics/test_otlp_metrics_exporter.py b/exporter/opentelemetry-exporter-otlp-proto-grpc/tests/metrics/test_otlp_metrics_exporter.py
--- a/exporter/opentelemetry-exporter-otlp-proto-grpc/tests/metrics/test_otlp_metrics_exporter.py
+++ b/exporter/opentelemetry-exporter-otlp-proto-grpc/tests/metrics/test_otlp_metrics_exporter.py
@@ -12,7 +12,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# pylint: disable=too-many-lines
 from concurrent.futures import ThreadPoolExecutor
+from typing import List
 from unittest import TestCase
 from unittest.mock import patch
 
@@ -52,13 +54,14 @@
     ObservableUpDownCounter,
     UpDownCounter,
 )
-from opentelemetry.sdk.metrics.export import AggregationTemporality
+from opentelemetry.sdk.metrics.export import AggregationTemporality, Gauge
 from opentelemetry.sdk.metrics.export import Histogram as HistogramType
 from opentelemetry.sdk.metrics.export import (
     HistogramDataPoint,
     Metric,
     MetricExportResult,
     MetricsData,
+    NumberDataPoint,
     ResourceMetrics,
     ScopeMetrics,
 )
@@ -969,3 +972,312 @@ def test_translate_multiple_scope_histogram(self):
         # pylint: disable=protected-access
         actual = self.exporter._translate_data(self.multiple_scope_histogram)
         self.assertEqual(expected, actual)
+
+    def test_split_metrics_data_many_data_points(self):
+        # GIVEN
+        metrics_data = MetricsData(
+            resource_metrics=[
+                _resource_metrics(
+                    index=1,
+                    scope_metrics=[
+                        _scope_metrics(
+                            index=1,
+                            metrics=[
+                                _gauge(
+                                    index=1,
+                                    data_points=[
+                                        _number_data_point(11),
+                                        _number_data_point(12),
+                                        _number_data_point(13),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ],
+                ),
+            ]
+        )
+        # WHEN
+        split_metrics_data: List[MetricsData] = list(
+            # pylint: disable=protected-access
+            OTLPMetricExporter(max_export_batch_size=2)._split_metrics_data(
+                metrics_data=metrics_data,
+            )
+        )
+        # THEN
+        self.assertEqual(
+            [
+                MetricsData(
+                    resource_metrics=[
+                        _resource_metrics(
+                            index=1,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=1,
+                                    metrics=[
+                                        _gauge(
+                                            index=1,
+                                            data_points=[
+                                                _number_data_point(11),
+                                                _number_data_point(12),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ]
+                ),
+                MetricsData(
+                    resource_metrics=[
+                        _resource_metrics(
+                            index=1,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=1,
+                                    metrics=[
+                                        _gauge(
+                                            index=1,
+                                            data_points=[
+                                                _number_data_point(13),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ]
+                ),
+            ],
+            split_metrics_data,
+        )
+
+    def test_split_metrics_data_nb_data_points_equal_batch_size(self):
+        # GIVEN
+        metrics_data = MetricsData(
+            resource_metrics=[
+                _resource_metrics(
+                    index=1,
+                    scope_metrics=[
+                        _scope_metrics(
+                            index=1,
+                            metrics=[
+                                _gauge(
+                                    index=1,
+                                    data_points=[
+                                        _number_data_point(11),
+                                        _number_data_point(12),
+                                        _number_data_point(13),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ],
+                ),
+            ]
+        )
+        # WHEN
+        split_metrics_data: List[MetricsData] = list(
+            # pylint: disable=protected-access
+            OTLPMetricExporter(max_export_batch_size=3)._split_metrics_data(
+                metrics_data=metrics_data,
+            )
+        )
+        # THEN
+        self.assertEqual(
+            [
+                MetricsData(
+                    resource_metrics=[
+                        _resource_metrics(
+                            index=1,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=1,
+                                    metrics=[
+                                        _gauge(
+                                            index=1,
+                                            data_points=[
+                                                _number_data_point(11),
+                                                _number_data_point(12),
+                                                _number_data_point(13),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ]
+                ),
+            ],
+            split_metrics_data,
+        )
+
+    def test_split_metrics_data_many_resources_scopes_metrics(self):
+        # GIVEN
+        metrics_data = MetricsData(
+            resource_metrics=[
+                _resource_metrics(
+                    index=1,
+                    scope_metrics=[
+                        _scope_metrics(
+                            index=1,
+                            metrics=[
+                                _gauge(
+                                    index=1,
+                                    data_points=[
+                                        _number_data_point(11),
+                                    ],
+                                ),
+                                _gauge(
+                                    index=2,
+                                    data_points=[
+                                        _number_data_point(12),
+                                    ],
+                                ),
+                            ],
+                        ),
+                        _scope_metrics(
+                            index=2,
+                            metrics=[
+                                _gauge(
+                                    index=3,
+                                    data_points=[
+                                        _number_data_point(13),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ],
+                ),
+                _resource_metrics(
+                    index=2,
+                    scope_metrics=[
+                        _scope_metrics(
+                            index=3,
+                            metrics=[
+                                _gauge(
+                                    index=4,
+                                    data_points=[
+                                        _number_data_point(14),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ],
+                ),
+            ]
+        )
+        # WHEN
+        split_metrics_data: List[MetricsData] = list(
+            # pylint: disable=protected-access
+            OTLPMetricExporter(max_export_batch_size=2)._split_metrics_data(
+                metrics_data=metrics_data,
+            )
+        )
+        # THEN
+        self.assertEqual(
+            [
+                MetricsData(
+                    resource_metrics=[
+                        _resource_metrics(
+                            index=1,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=1,
+                                    metrics=[
+                                        _gauge(
+                                            index=1,
+                                            data_points=[
+                                                _number_data_point(11),
+                                            ],
+                                        ),
+                                        _gauge(
+                                            index=2,
+                                            data_points=[
+                                                _number_data_point(12),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ]
+                ),
+                MetricsData(
+                    resource_metrics=[
+                        _resource_metrics(
+                            index=1,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=2,
+                                    metrics=[
+                                        _gauge(
+                                            index=3,
+                                            data_points=[
+                                                _number_data_point(13),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                        _resource_metrics(
+                            index=2,
+                            scope_metrics=[
+                                _scope_metrics(
+                                    index=3,
+                                    metrics=[
+                                        _gauge(
+                                            index=4,
+                                            data_points=[
+                                                _number_data_point(14),
+                                            ],
+                                        ),
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ]
+                ),
+            ],
+            split_metrics_data,
+        )
+
+
+def _resource_metrics(
+    index: int, scope_metrics: List[ScopeMetrics]
+) -> ResourceMetrics:
+    return ResourceMetrics(
+        resource=Resource(
+            attributes={"a": index},
+            schema_url=f"resource_url_{index}",
+        ),
+        schema_url=f"resource_url_{index}",
+        scope_metrics=scope_metrics,
+    )
+
+
+def _scope_metrics(index: int, metrics: List[Metric]) -> ScopeMetrics:
+    return ScopeMetrics(
+        scope=InstrumentationScope(name=f"scope_{index}"),
+        schema_url=f"scope_url_{index}",
+        metrics=metrics,
+    )
+
+
+def _gauge(index: int, data_points: List[NumberDataPoint]) -> Metric:
+    return Metric(
+        name=f"gauge_{index}",
+        description="description",
+        unit="unit",
+        data=Gauge(data_points=data_points),
+    )
+
+
+def _number_data_point(value: int) -> NumberDataPoint:
+    return NumberDataPoint(
+        attributes={"a": 1, "b": True},
+        start_time_unix_nano=1641946015139533244,
+        time_unix_nano=1641946016139533244,
+        value=value,
+    )