Repository: elastic/apm-agent-go
Pull request #1407 (2023-03-31 07:40:53)

OpenTelemetry metrics exporter
This PR is setting up an OpenTelemetry metrics exporter, based on the specification in https://github.com/elastic/apm/pull/742.

apmmachine: 
## :green_heart: Build Succeeded


<!-- BUILD BADGES-->
> _the below badges are clickable and redirect to their specific view in the CI or DOCS_
[![Pipeline View](https://img.shields.io/badge/pipeline-pipeline%20-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-go%2Fapm-agent-go-mbp%2FPR-1407/detail/PR-1407/1//pipeline) [![Test View](https://img.shields.io/badge/test-test-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-go%2Fapm-agent-go-mbp%2FPR-1407/detail/PR-1407/1//tests) [![Changes](https://img.shields.io/badge/changes-changes-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-go%2Fapm-agent-go-mbp%2FPR-1407/detail/PR-1407/1//changes) [![Artifacts](https://img.shields.io/badge/artifacts-artifacts-yellow)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-go%2Fapm-agent-go-mbp%2FPR-1407/detail/PR-1407/1//artifacts) [![preview](https://img.shields.io/badge/docs-preview-yellowgreen)](http://apm-agent-go_1407.docs-preview.app.elstc.co/diff)  [![preview](https://img.shields.io/badge/elastic-observability-blue)](https://ci-stats.elastic.co/app/apm/services/apm-ci/transactions/view?rangeFrom=2023-03-31T07:31:09.476Z&rangeTo=2023-03-31T07:51:09.476Z&transactionName=BUILD+apm-agent-go%2Fapm-agent-go-mbp%2FPR-%7Bnumber%7D&transactionType=job&latencyAggregationType=avg&traceId=ba7ad1516ed3c5b6fbaef6c373a7fe12&transactionId=27842b405ec2f759)[![Snapshots](https://img.shields.io/badge/snapshots-artifacts-blue)](https://console.cloud.google.com/storage/browser/oblt-artifacts/apm-agent-go/PR-1407;tab=objects?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%257B_22k_22_3A_22Name%2520contains_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_235ebaa2_5C_22_22_2C_22i_22_3A_22displayName_22%257D%255D%22)))

<!-- BUILD SUMMARY-->
<details><summary>Expand to view the summary</summary>
<p>

#### Build stats




* Start Time: 2023-03-31T07:41:09.476+0000


* Duration: 3 min 10 sec





</p>
</details>

<!-- TEST RESULTS IF ANY-->


<!-- STEPS ERRORS IF ANY -->














## :robot: GitHub comments

<details><summary>Expand to view the GitHub comments</summary>
<p>

To re-run your PR in the CI, just comment with:



- `/test` : Re-trigger the build.


- `run benchmark tests` : Run the benchmark test.


- `run` `elasticsearch-ci/docs` : Re-trigger the docs validation. (use unformatted text in the comment!)




</p>
</details>

<!--COMMENT_GENERATED_WITH_ID_comment.id-->
dmathieu: Supporting Go versions down to 1.15 with OpenTelemetry is going to be very tricky.

Otel follows the [Golang release policy](https://go.dev/doc/devel/release#policy):
> Each major Go release is supported until there are two newer major releases.

Which means it only support 1.19 and 1.20, and has therefore started using generics, making it impossible to use this version with any Go < 1.18.
Going to a lower version of the SDK will not be possible either, as the metrics API wasn't finalized back then.

Patch:
diff --git a/docs/opentelemetry.asciidoc b/docs/opentelemetry.asciidoc
new file mode 100644
index 00000000..a57f4aeb
--- /dev/null
+++ b/docs/opentelemetry.asciidoc
@@ -0,0 +1,66 @@
+[[opentelemetry]]
+== OpenTelemetry API
+
+The Elastic APM Go Agent provides wrappers to interact with the https://opentelemetry.io/[OpenTelemetry API].
+
+Metrics created through the OpenTelemetry API will be translated to Elastic APM metrics.
+
+[float]
+[[opentelemetry-init]]
+=== Initializing the metrics exporter
+
+The OpenTelemetry Metrics bridge is implemented as a manual reader exporter.
+The Elastic APM Agent will regularly ask OpenTelemetry for its latest metrics,
+and emit them as its own.
+
+To initialize this exporter, you first need to import the `apmotel` package.
+
+
+[source,go]
+----
+import (
+	"go.elastic.co/apm/module/apmotel"
+)
+----
+
+The apmotel package exposes a `NewGatherer` method, which returns an implementation of both an https://pkg.go.dev/github.com/elastic/apm-agent-go#MetricsGatherer[Elastic MetricsGatherer], and an https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric#Reader[OpenTelemetry metric.Reader].
+
+[source,go]
+----
+exporter := apmotel.NewGatherer()
+----
+
+The method allows passing some options, such as `WithAggregationSelector`, to
+specify a custom OpenTelemetry aggregation selector.
+
+Once you have obtained this exporter, you can configure the OpenTelemetry SDK
+so it reports emitted metrics, as well as the Elastic Agent to read those
+metrics.
+
+
+[source,go]
+----
+import (
+	"go.elastic.co/apm"
+	"go.elastic.co/apm/module/apmotel"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/sdk/metric"
+)
+
+func main() {
+	exporter := apmotel.NewGatherer()
+
+	// Configure OpenTelemetry
+	provider := metric.NewMeterProvider(metric.WithReader(exporter))
+	otel.SetMeterProvider(provider)
+
+	// Configure the Elastic APM
+	apm.DefaultTracer().RegisterMetricsGatherer(exporter)
+
+	// Record a metric with OpenTelemetry which will be exported to the Elastic Agent
+	meter := provider.Meter("my_application")
+	counter, _ := meter.Float64Counter("metric_called")
+	counter.Add(context.TODO(), 1)
+}
+----
diff --git a/module/apmotel/config.go b/module/apmotel/config.go
new file mode 100644
index 00000000..98339f9d
--- /dev/null
+++ b/module/apmotel/config.go
@@ -0,0 +1,80 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build go1.18
+// +build go1.18
+
+package apmotel // import "go.elastic.co/apm/module/apmotel/v2"
+
+import (
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/metric/aggregation"
+)
+
+var customHistogramBoundaries = []float64{
+	0.00390625, 0.00552427, 0.0078125, 0.0110485, 0.015625, 0.0220971, 0.03125,
+	0.0441942, 0.0625, 0.0883883, 0.125, 0.176777, 0.25, 0.353553, 0.5, 0.707107,
+	1, 1.41421, 2, 2.82843, 4, 5.65685, 8, 11.3137, 16, 22.6274, 32, 45.2548, 64,
+	90.5097, 128, 181.019, 256, 362.039, 512, 724.077, 1024, 1448.15, 2048,
+	2896.31, 4096, 5792.62, 8192, 11585.2, 16384, 23170.5, 32768, 46341.0, 65536,
+	92681.9, 131072,
+}
+
+type config struct {
+	aggregation metric.AggregationSelector
+}
+
+type Option func(config) config
+
+func newConfig(opts ...Option) config {
+	cfg := config{
+		aggregation: customAggregationSelector,
+	}
+	for _, opt := range opts {
+		opt(cfg)
+	}
+
+	return cfg
+}
+
+func (cfg config) manualReaderOptions() []metric.ManualReaderOption {
+	opts := []metric.ManualReaderOption{}
+	opts = append(opts, metric.WithAggregationSelector(cfg.aggregation))
+	return opts
+}
+
+// WithAggregationSelector configure the Aggregation Selector the exporter will
+// use. If no AggregationSelector is provided the DefaultAggregationSelector is
+// used.
+func WithAggregationSelector(agg metric.AggregationSelector) Option {
+	return func(cfg config) config {
+		cfg.aggregation = agg
+		return cfg
+	}
+}
+
+func customAggregationSelector(ik metric.InstrumentKind) aggregation.Aggregation {
+	switch ik {
+	case metric.InstrumentKindHistogram:
+		return aggregation.ExplicitBucketHistogram{
+			Boundaries: customHistogramBoundaries,
+			NoMinMax:   false,
+		}
+	default:
+		return metric.DefaultAggregationSelector(ik)
+	}
+}
diff --git a/module/apmotel/doc.go b/module/apmotel/doc.go
new file mode 100644
index 00000000..607bf10c
--- /dev/null
+++ b/module/apmotel/doc.go
@@ -0,0 +1,19 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package apmotel provides helpers for exporting OpenTelemetry data to the Elastic APM Agent.
+package apmotel // import "go.elastic.co/apm/module/apmotel/v2"
diff --git a/module/apmotel/gatherer.go b/module/apmotel/gatherer.go
new file mode 100644
index 00000000..2d04777a
--- /dev/null
+++ b/module/apmotel/gatherer.go
@@ -0,0 +1,152 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build go1.18
+// +build go1.18
+
+package apmotel // import "go.elastic.co/apm/module/apmotel/v2"
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/metric/metricdata"
+
+	"go.elastic.co/apm/v2"
+)
+
+// NewGatherer creates a new gatherer/exporter to bridge between agent metrics and OpenTelemetry
+func NewGatherer(opts ...Option) (Gatherer, error) {
+	cfg := newConfig(opts...)
+	reader := metric.NewManualReader(cfg.manualReaderOptions()...)
+
+	return Gatherer{reader}, nil
+}
+
+// Gatherer is a gatherer/exporter which can act as an OpenTelemetry manual
+// reader, to retrieve metrics and expose them to the agent.
+type Gatherer struct {
+	metric.Reader
+}
+
+var _ metric.Reader = Gatherer{}
+var _ apm.MetricsGatherer = Gatherer{}
+
+// GatherMetrics gathers metrics into out.
+func (e Gatherer) GatherMetrics(ctx context.Context, out *apm.Metrics) error {
+	metrics := metricdata.ResourceMetrics{}
+	err := e.Reader.Collect(ctx, &metrics)
+	if err != nil {
+		return err
+	}
+
+	for _, scopeMetrics := range metrics.ScopeMetrics {
+
+		for _, sm := range scopeMetrics.Metrics {
+			switch m := sm.Data.(type) {
+			case metricdata.Histogram[int64]:
+				addHistogramMetric(out, sm, m)
+			case metricdata.Histogram[float64]:
+				addHistogramMetric(out, sm, m)
+			case metricdata.Sum[int64]:
+				for _, dp := range m.DataPoints {
+					out.Add(sm.Name, makeLabels(dp.Attributes), float64(dp.Value))
+				}
+			case metricdata.Sum[float64]:
+				for _, dp := range m.DataPoints {
+					out.Add(sm.Name, makeLabels(dp.Attributes), dp.Value)
+				}
+			case metricdata.Gauge[int64]:
+				for _, dp := range m.DataPoints {
+					out.Add(sm.Name, makeLabels(dp.Attributes), float64(dp.Value))
+				}
+			case metricdata.Gauge[float64]:
+				for _, dp := range m.DataPoints {
+					out.Add(sm.Name, makeLabels(dp.Attributes), dp.Value)
+				}
+			default:
+				return fmt.Errorf("unknown metric type %q", m)
+			}
+		}
+	}
+
+	return nil
+}
+
+func addHistogramMetric[N int64 | float64](out *apm.Metrics, sm metricdata.Metrics, m metricdata.Histogram[N]) {
+	for _, dp := range m.DataPoints {
+		if len(dp.BucketCounts) != len(dp.Bounds)+1 || len(dp.Bounds) == 0 {
+			continue
+		}
+
+		bounds := make([]float64, 0, len(dp.Bounds))
+		counts := make([]uint64, 0, len(dp.BucketCounts))
+
+		for i, _ := range dp.BucketCounts {
+			bound, count := computeCountAndBounds(i, dp.Bounds, dp.BucketCounts)
+			if count == 0 {
+				continue
+			}
+
+			counts = append(counts, count)
+			bounds = append(bounds, bound)
+		}
+
+		out.AddHistogram(sm.Name, makeLabels(dp.Attributes), bounds, counts)
+	}
+}
+
+func computeCountAndBounds(i int, bounds []float64, counts []uint64) (float64, uint64) {
+	count := counts[i]
+	if count == 0 {
+		return 0, 0
+	}
+
+	var bound float64
+	switch i {
+	// (-infinity, explicit_bounds[i]]
+	case 0:
+		bound = bounds[i]
+		if bound > 0 {
+			bound /= 2
+		}
+
+	// (explicit_bounds[i], +infinity)
+	case len(counts) - 1:
+		bound = bounds[i-1]
+
+	// [explicit_bounds[i-1], explicit_bounds[i])
+	default:
+		// Use the midpoint between the boundaries.
+		bound = bounds[i-1] + (bounds[i]-bounds[i-1])/2.0
+	}
+
+	return bound, count
+}
+
+func makeLabels(attrs attribute.Set) []apm.MetricLabel {
+	labels := make([]apm.MetricLabel, attrs.Len())
+	iter := attrs.Iter()
+	for iter.Next() {
+		i, kv := iter.IndexedAttribute()
+		labels[i] = apm.MetricLabel{Name: string(kv.Key), Value: kv.Value.Emit()}
+	}
+
+	return labels
+}
diff --git a/module/apmotel/go.mod b/module/apmotel/go.mod
new file mode 100644
index 00000000..e0945cc2
--- /dev/null
+++ b/module/apmotel/go.mod
@@ -0,0 +1,38 @@
+module go.elastic.co/apm/module/apmotel/v2
+
+require (
+	github.com/stretchr/testify v1.8.2
+	go.elastic.co/apm/v2 v2.0.0-00010101000000-000000000000
+	go.opentelemetry.io/otel v1.15.0-rc.2
+	go.opentelemetry.io/otel/metric v1.15.0-rc.2
+	go.opentelemetry.io/otel/sdk/metric v0.38.0-rc.2
+)
+
+require (
+	github.com/armon/go-radix v1.0.0 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/elastic/go-licenser v0.4.0 // indirect
+	github.com/elastic/go-sysinfo v1.7.1 // indirect
+	github.com/elastic/go-windows v1.0.1 // indirect
+	github.com/go-logr/logr v1.2.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/google/go-cmp v0.5.9 // indirect
+	github.com/jcchavezs/porto v0.1.0 // indirect
+	github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/prometheus/procfs v0.7.3 // indirect
+	go.elastic.co/fastjson v1.1.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.15.0-rc.2 // indirect
+	go.opentelemetry.io/otel/trace v1.15.0-rc.2 // indirect
+	golang.org/x/mod v0.5.1 // indirect
+	golang.org/x/sys v0.6.0 // indirect
+	golang.org/x/tools v0.1.9 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+	howett.net/plist v1.0.0 // indirect
+)
+
+replace go.elastic.co/apm/v2 => ../..
+
+go 1.20
diff --git a/module/apmotel/go.sum b/module/apmotel/go.sum
new file mode 100644
index 00000000..eacbaf3a
--- /dev/null
+++ b/module/apmotel/go.sum
@@ -0,0 +1,122 @@
+github.com/armon/go-radix v1.0.0 h1:F4z6KzEeeQIMeLFa97iZU6vupzoecKdU5TX24SNppXI=
+github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/elastic/go-licenser v0.4.0 h1:jLq6A5SilDS/Iz1ABRkO6BHy91B9jBora8FwGRsDqUI=
+github.com/elastic/go-licenser v0.4.0/go.mod h1:V56wHMpmdURfibNBggaSBfqgPxyT1Tldns1i87iTEvU=
+github.com/elastic/go-sysinfo v1.7.1 h1:Wx4DSARcKLllpKT2TnFVdSUJOsybqMYCNQZq1/wO+s0=
+github.com/elastic/go-sysinfo v1.7.1/go.mod h1:i1ZYdU10oLNfRzq4vq62BEwD2fH8KaWh6eh0ikPT9F0=
+github.com/elastic/go-windows v1.0.0/go.mod h1:TsU0Nrp7/y3+VwE82FoZF8gC/XFg/Elz6CcloAxnPgU=
+github.com/elastic/go-windows v1.0.1 h1:AlYZOldA+UJ0/2nBuqWdo90GFCgG9xuyw9SYzGUtJm0=
+github.com/elastic/go-windows v1.0.1/go.mod h1:FoVvqWSun28vaDQPbj2Elfc0JahhPB7WQEGa3c814Ss=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
+github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/jcchavezs/porto v0.1.0 h1:Xmxxn25zQMmgE7/yHYmh19KcItG81hIwfbEEFnd6w/Q=
+github.com/jcchavezs/porto v0.1.0/go.mod h1:fESH0gzDHiutHRdX2hv27ojnOVFco37hg1W6E9EZF4A=
+github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
+github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 h1:rp+c0RAYOWj8l6qbCUTSiRLG/iKnW3K3/QfPPuSsBt4=
+github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901/go.mod h1:Z86h9688Y0wesXCyonoVr47MasHilkuLMqGhRZ4Hpak=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/prometheus/procfs v0.0.0-20190425082905-87a4384529e0/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
+github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU=
+github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
+github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
+github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
+github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
+github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+go.elastic.co/fastjson v1.1.0 h1:3MrGBWWVIxe/xvsbpghtkFoPciPhOCmjsR/HfwEeQR4=
+go.elastic.co/fastjson v1.1.0/go.mod h1:boNGISWMjQsUPy/t6yqt2/1Wx4YNPSe+mZjlyw9vKKI=
+go.opentelemetry.io/otel v1.15.0-rc.2 h1:ujPMbp/CAdjMTF9E93fr6X2W7Ub2MulkQAmXcTaNkJI=
+go.opentelemetry.io/otel v1.15.0-rc.2/go.mod h1:AMBt1VNvSP4KmoaUOQcTWUVj6Te4ZRmD/v3tu0vfytI=
+go.opentelemetry.io/otel/metric v1.15.0-rc.2 h1:MA7asjRHRgVdbvqYLQozHkxViPFZJ21qzi9IOKZVTnQ=
+go.opentelemetry.io/otel/metric v1.15.0-rc.2/go.mod h1:/e8iKMaajFDmOXPoQsFrtr+B3XADzJS+HHePsQh2gGs=
+go.opentelemetry.io/otel/sdk v1.15.0-rc.2 h1:4t4aauPUWLswNEUXAi5s3X94/kg7+NQY+6CeR9F6nEk=
+go.opentelemetry.io/otel/sdk v1.15.0-rc.2/go.mod h1:/dv7lLx8SqyUgUxQeFeUUGvHbl45K4dssbaYjj/ObTQ=
+go.opentelemetry.io/otel/sdk/metric v0.38.0-rc.2 h1:0hZK6hT73+T5BNuFW6AsBwfARsgeThwAQ7JagJ1hbYE=
+go.opentelemetry.io/otel/sdk/metric v0.38.0-rc.2/go.mod h1:GUyzM862531kqKgJkhS/LIeekJ652QyjN94dzN1zmgQ=
+go.opentelemetry.io/otel/trace v1.15.0-rc.2 h1:LizBnU5zjqWkHUlZu0tC2k7qICXqPCJqoDlWdc07uw8=
+go.opentelemetry.io/otel/trace v1.15.0-rc.2/go.mod h1:JBu+wEp/Ra+YOzZlvPjE6raGRlrUJE/R8aW7Wucsn/o=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
+golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
+golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
+golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191025021431-6c3a3bfe00ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211102192858-4dd72447c267/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
+golang.org/x/tools v0.1.9 h1:j9KsMiaP1c3B0OTQGth0/k+miLGTgLsAFUCrF2vLcF8=
+golang.org/x/tools v0.1.9/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v1 v1.0.0-20140924161607-9f9df34309c0/go.mod h1:WDnlLJ4WF5VGsH/HVa3CI79GS0ol3YnhVnKP89i0kNg=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+howett.net/plist v0.0.0-20181124034731-591f970eefbb/go.mod h1:vMygbs4qMhSZSc4lCUl2OEE+rDiIIJAIdR4m7MiMcm0=
+howett.net/plist v1.0.0 h1:7CrbWYbPPO/PyNy38b2EB/+gYbjCe2DXBxgtOOZbSQM=
+howett.net/plist v1.0.0/go.mod h1:lqaXoTrLY4hg8tnEzNru53gicrbv7rrk+2xJA/7hw9g=


Test patch:
diff --git a/module/apmotel/config_test.go b/module/apmotel/config_test.go
new file mode 100644
index 00000000..f443a19d
--- /dev/null
+++ b/module/apmotel/config_test.go
@@ -0,0 +1,89 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build go1.18
+// +build go1.18
+
+package apmotel // import "go.elastic.co/apm/module/apmotel/v2"
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/metric/aggregation"
+)
+
+func TestNewConfig(t *testing.T) {
+	aggregationSelector := func(metric.InstrumentKind) aggregation.Aggregation { return nil }
+
+	testCases := []struct {
+		name       string
+		options    []Option
+		wantConfig config
+	}{
+		{
+			name:       "Default",
+			options:    nil,
+			wantConfig: config{},
+		},
+		{
+			name: "WithAggregationSelector",
+			options: []Option{
+				WithAggregationSelector(aggregationSelector),
+			},
+			wantConfig: config{},
+		},
+	}
+	for _, tt := range testCases {
+		t.Run(tt.name, func(t *testing.T) {
+			cfg := newConfig(tt.options...)
+			// tested by TestConfigManualReaderOptions
+			cfg.aggregation = nil
+
+			assert.Equal(t, tt.wantConfig, cfg)
+		})
+	}
+}
+
+func TestConfigManualReaderOptions(t *testing.T) {
+	aggregationSelector := func(metric.InstrumentKind) aggregation.Aggregation { return nil }
+
+	testCases := []struct {
+		name            string
+		config          config
+		wantOptionCount int
+	}{
+		{
+			name:            "Default",
+			config:          config{},
+			wantOptionCount: 1,
+		},
+
+		{
+			name:            "WithAggregationSelector",
+			config:          config{aggregation: aggregationSelector},
+			wantOptionCount: 1,
+		},
+	}
+	for _, tt := range testCases {
+		t.Run(tt.name, func(t *testing.T) {
+			opts := tt.config.manualReaderOptions()
+			assert.Len(t, opts, tt.wantOptionCount)
+		})
+	}
+}
diff --git a/module/apmotel/gatherer_test.go b/module/apmotel/gatherer_test.go
new file mode 100644
index 00000000..0fad1b67
--- /dev/null
+++ b/module/apmotel/gatherer_test.go
@@ -0,0 +1,362 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build go1.18
+// +build go1.18
+
+package apmotel
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/metric"
+	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/metric/aggregation"
+
+	"go.elastic.co/apm/v2"
+	"go.elastic.co/apm/v2/apmtest"
+	"go.elastic.co/apm/v2/model"
+)
+
+func TestGatherer(t *testing.T) {
+	for _, tt := range []struct {
+		name string
+
+		recordMetrics   func(ctx context.Context, meter metric.Meter)
+		expectedMetrics []model.Metrics
+	}{
+		{
+			name: "with a float64 counter",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Float64Counter("foo")
+				assert.NoError(t, err)
+				counter.Add(ctx, 5, attribute.Key("A").String("B"))
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"foo": {
+							Value: 5,
+						},
+					},
+					Labels: model.StringMap{
+						model.StringMapItem{Key: "A", Value: "B"},
+					},
+				},
+			},
+		},
+		{
+			name: "with an int64 counter",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Int64Counter("foo")
+				assert.NoError(t, err)
+				counter.Add(ctx, 5, attribute.Key("A").String("B"))
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"foo": {
+							Value: 5,
+						},
+					},
+					Labels: model.StringMap{
+						model.StringMapItem{Key: "A", Value: "B"},
+					},
+				},
+			},
+		},
+		{
+			name: "with a float64 gauge",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Float64UpDownCounter("foo")
+				assert.NoError(t, err)
+				counter.Add(ctx, 5, attribute.Key("A").String("B"))
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"foo": {
+							Value: 5,
+						},
+					},
+					Labels: model.StringMap{
+						model.StringMapItem{Key: "A", Value: "B"},
+					},
+				},
+			},
+		},
+		{
+			name: "with an int64 gauge",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Float64UpDownCounter("foo")
+				assert.NoError(t, err)
+				counter.Add(ctx, 5, attribute.Key("A").String("B"))
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"foo": {
+							Value: 5,
+						},
+					},
+					Labels: model.StringMap{
+						model.StringMapItem{Key: "A", Value: "B"},
+					},
+				},
+			},
+		},
+		{
+			name: "with a float64 histogram",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Float64Histogram("histogram_foo")
+				assert.NoError(t, err)
+				counter.Record(ctx, 3.4,
+					attribute.Key("code").String("200"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 3.4,
+					attribute.Key("code").String("200"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 3.4,
+					attribute.Key("code").String("200"),
+					attribute.Key("method").String("GET"),
+				)
+
+				counter.Record(ctx, 5.5,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 5.5,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 5.5,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+
+				counter.Record(ctx, 11.2,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 11.2,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+				counter.Record(ctx, 11.2,
+					attribute.Key("code").String("302"),
+					attribute.Key("method").String("GET"),
+				)
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"histogram_foo": {
+							Type:   "histogram",
+							Values: []float64{4.828425, 9.65685},
+							Counts: []uint64{3, 3},
+						},
+					},
+					Labels: model.StringMap{
+						{Key: "code", Value: "302"},
+						{Key: "method", Value: "GET"},
+					},
+				},
+				{
+					Samples: map[string]model.Metric{
+						"histogram_foo": {
+							Type:   "histogram",
+							Values: []float64{3.414215},
+							Counts: []uint64{3},
+						},
+					},
+					Labels: model.StringMap{
+						{Key: "code", Value: "200"},
+						{Key: "method", Value: "GET"},
+					},
+				},
+			},
+		},
+		{
+			name: "with an int64 histogram",
+			recordMetrics: func(ctx context.Context, meter metric.Meter) {
+				counter, err := meter.Int64Histogram("foo")
+				assert.NoError(t, err)
+				counter.Record(ctx, 5, attribute.Key("A").String("B"))
+			},
+			expectedMetrics: []model.Metrics{
+				{
+					Samples: map[string]model.Metric{
+						"foo": {
+							Type:   "histogram",
+							Values: []float64{4.828425},
+							Counts: []uint64{1},
+						},
+					},
+					Labels: model.StringMap{
+						model.StringMapItem{Key: "A", Value: "B"},
+					},
+				},
+			},
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			ctx := context.Background()
+
+			gatherer, err := NewGatherer()
+			assert.NoError(t, err)
+			provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(gatherer))
+			meter := provider.Meter("apmotel_test")
+
+			tt.recordMetrics(ctx, meter)
+
+			metrics := gatherMetrics(gatherer)
+
+			assert.ElementsMatch(t, tt.expectedMetrics, metrics)
+		})
+	}
+}
+
+func TestGathererWithCustomView(t *testing.T) {
+	ctx := context.Background()
+
+	gatherer, err := NewGatherer()
+	assert.NoError(t, err)
+	provider := sdkmetric.NewMeterProvider(
+		sdkmetric.WithReader(gatherer),
+		sdkmetric.WithView(sdkmetric.NewView(
+			sdkmetric.Instrument{Name: "*"},
+			sdkmetric.Stream{Aggregation: aggregation.ExplicitBucketHistogram{
+				Boundaries: []float64{0, 5, 10, 25, 50, 75, 100, 250, 500, 1000},
+			}},
+		)),
+	)
+	meter := provider.Meter("apmotel_test")
+
+	counter, err := meter.Float64Histogram("histogram_foo")
+	assert.NoError(t, err)
+	counter.Record(ctx, 3.4,
+		attribute.Key("code").String("200"),
+		attribute.Key("method").String("GET"),
+	)
+
+	metrics := gatherMetrics(gatherer)
+
+	assert.ElementsMatch(t, []model.Metrics{
+		{
+			Samples: map[string]model.Metric{
+				"histogram_foo": {
+					Type:   "histogram",
+					Values: []float64{2.5},
+					Counts: []uint64{1},
+				},
+			},
+			Labels: model.StringMap{
+				{Key: "code", Value: "200"},
+				{Key: "method", Value: "GET"},
+			},
+		},
+	}, metrics)
+}
+
+func TestComputeCountAndBounds(t *testing.T) {
+	for _, tt := range []struct {
+		name string
+
+		index  int
+		bounds []float64
+		counts []uint64
+
+		expectedBound float64
+		expectedCount uint64
+	}{
+		{
+			name:   "with a zero count",
+			index:  0,
+			bounds: []float64{5},
+			counts: []uint64{0, 0},
+
+			expectedBound: 0,
+			expectedCount: 0,
+		},
+		{
+			name:   "with the -infinity bucket (zero index)",
+			index:  0,
+			bounds: []float64{6},
+			counts: []uint64{1, 0},
+
+			expectedBound: 3,
+			expectedCount: 1,
+		},
+		{
+			name:   "with the +infinity bucket (last index)",
+			index:  2,
+			bounds: []float64{6, 8},
+			counts: []uint64{1, 2, 1},
+
+			expectedBound: 8,
+			expectedCount: 1,
+		},
+		{
+			name:   "with midpoint boundaries",
+			index:  1,
+			bounds: []float64{6, 8},
+			counts: []uint64{1, 2, 1},
+
+			expectedBound: 7,
+			expectedCount: 2,
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			bound, count := computeCountAndBounds(tt.index, tt.bounds, tt.counts)
+			assert.Equal(t, tt.expectedBound, bound)
+			assert.Equal(t, tt.expectedCount, count)
+		})
+	}
+}
+
+func gatherMetrics(g apm.MetricsGatherer) []model.Metrics {
+	tracer := apmtest.NewRecordingTracer()
+	defer tracer.Close()
+	tracer.RegisterMetricsGatherer(g)
+	tracer.SendMetrics(nil)
+	metrics := tracer.Payloads().Metrics
+	for i := range metrics {
+		metrics[i].Timestamp = model.Time{}
+	}
+
+	// Remove internal metrics
+	for i, m := range metrics {
+		for k := range m.Samples {
+			if strings.HasPrefix(k, "golang.") || strings.HasPrefix(k, "system.") {
+				delete(m.Samples, k)
+			}
+		}
+
+		if len(m.Samples) == 0 {
+			metrics[i] = metrics[len(metrics)-1]
+			metrics = metrics[:len(metrics)-1]
+		}
+	}
+	return metrics
+}
diff --git a/scripts/Dockerfile-testing b/scripts/Dockerfile-testing
index b8c3b1d9..4c18236b 100644
--- a/scripts/Dockerfile-testing
+++ b/scripts/Dockerfile-testing
@@ -36,6 +36,7 @@ COPY module/apmlogrus/go.mod module/apmlogrus/go.sum /go/src/go.elastic.co/apm/m
 COPY module/apmmongo/go.mod module/apmmongo/go.sum /go/src/go.elastic.co/apm/module/apmmongo/
 COPY module/apmnegroni/go.mod module/apmnegroni/go.sum /go/src/go.elastic.co/apm/module/apmnegroni/
 COPY module/apmot/go.mod module/apmot/go.sum /go/src/go.elastic.co/apm/module/apmot/
+COPY module/apmotel/go.mod module/apmotel/go.sum /go/src/go.elastic.co/apm/module/apmotel/
 COPY module/apmpgx/go.mod module/apmpgx/go.sum /go/src/go.elastic.co/apm/module/apmpgx/
 COPY module/apmpgxv5/go.mod module/apmpgxv5/go.sum /go/src/go.elastic.co/apm/module/apmpgxv5/
 COPY module/apmprometheus/go.mod module/apmprometheus/go.sum /go/src/go.elastic.co/apm/module/apmprometheus/
@@ -81,6 +82,7 @@ RUN cd /go/src/go.elastic.co/apm/module/apmlogrus && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmmongo && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmnegroni && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmot && go mod download
+RUN cd /go/src/go.elastic.co/apm/module/apmotel && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmpgx && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmpgxv5 && go mod download
 RUN cd /go/src/go.elastic.co/apm/module/apmprometheus && go mod download