Repository: wundergraph/cosmo
Pull request #1862: feat: add new metric store with trace connection metrics
Author: login: SkArchon
Labels: name: router
Created: 2025-05-14T20:11:13Z | Merged: 2025-05-22T17:37:15Z

<!--
Important: Before developing new features, please open an issue to discuss your ideas with the maintainers. This ensures project alignment and helps avoid unnecessary work for you.

Thank you for your contribution! Please provide a detailed description below and ensure you've met all the requirements.

Contributors Guide: https://github.com/wundergraph/cosmo/blob/main/CONTRIBUTING.md

Squashed commit messages must follow the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) standard to facilitate changelog generation.

Please ensure your PR title follows the Conventional Commits specification, using the appropriate type (e.g., feat, fix, docs) and scope.

Examples of good PR titles:

- 💥feat!: change implementation in an non-backward compatible way
- ✨feat(auth): add support for OAuth2 login
- 🐞fix(router): add support for custom metrics
- 📚docs(README): update installation instructions
- 🧹chore(deps): bump dependencies to latest versions
-->

## Motivation and Context

This PR adds a set of new metric store called "ConnectionMetricStore", which is used to collect measurements related to information when making a subgraph request. We introduce the following metric attributes as part of the `router connection metric` store.

All metrics are slicecable (dimensions) by `subgraph`, `server.address` and `server.port` (the last two are semconv standard attributes, we went with these names because other attributes like net.peer.name have been deprecated in favour of it)

**router.http.client.max_connections**: Static gauge which keeps values for any maxConnsPerHost configuration
**router.http.client.active_connections**: A gauge which keeps track of active connections
**router.http.client.acquire_connection_duration**: A histogram of the total time taken to acquire a connection

In order to do this we introduce a traceclient, as well as a tracedialer. The first has hooks for operations such as get_conn, etc. The last is a wrapper on the `http.transport` `WrapDialer` method. `WrapDialer` is ONLY called when a new connection is created, thus we can use the function call as a fact that a connection was created.

## Checklist

- [ ] I have discussed my proposed changes in an issue and have received approval to proceed.
- [ ] I have followed the coding standards of the project.
- [ ] Tests or benchmarks have been added or updated.
- [ ] Documentation has been updated on [https://github.com/wundergraph/cosmo-docs](https://github.com/wundergraph/cosmo-docs).
- [ ] I have read the [Contributors Guide](https://github.com/wundergraph/cosmo/blob/main/CONTRIBUTING.md).

<!--
Please add any additional information or context regarding your changes here.
-->


Diff:
diff --git a/router-tests/prometheus_improved_test.go b/router-tests/prometheus_improved_test.go
index f9e7003b3b..ad778de323 100644
--- a/router-tests/prometheus_improved_test.go
+++ b/router-tests/prometheus_improved_test.go
@@ -1,8 +1,8 @@
 package integration
 
 import (
+	rmetric "github.com/wundergraph/cosmo/router/pkg/metric"
 	"regexp"
-	"strings"
 	"testing"
 
 	"github.com/prometheus/client_golang/prometheus"
@@ -332,7 +332,7 @@ func assertLabelValue(t *testing.T, labels []*io_prometheus_client.LabelPair, la
 }
 
 func findLabel(labels []*io_prometheus_client.LabelPair, labelKey attribute.Key) *io_prometheus_client.LabelPair {
-	key := strings.ReplaceAll(string(labelKey), ".", "_")
+	key := rmetric.SanitizeName(string(labelKey))
 
 	for _, label := range labels {
 		if label.Name != nil && *label.Name == key {
diff --git a/router-tests/prometheus_test.go b/router-tests/prometheus_test.go
index e043675466..018fe3ff07 100644
--- a/router-tests/prometheus_test.go
+++ b/router-tests/prometheus_test.go
@@ -1,8 +1,12 @@
 package integration
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
+	rmetric "github.com/wundergraph/cosmo/router/pkg/metric"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+	"go.opentelemetry.io/otel/sdk/metric/metricdata"
 	"net/http"
 	"regexp"
 	"strings"
@@ -4227,6 +4231,22 @@ func TestPrometheus(t *testing.T) {
 
 }
 
+func getPort(connectionTotal *io_prometheus_client.Metric) string {
+	serverPortKey := rmetric.SanitizeName(string(otel.ServerPort.String("").Key))
+
+	for _, label := range connectionTotal.Label {
+		if label.Name == nil || label.Value == nil {
+			continue
+		}
+
+		if *label.Name == serverPortKey {
+			return *label.Value
+		}
+	}
+
+	return ""
+}
+
 func TestPrometheusWithModule(t *testing.T) {
 	t.Parallel()
 
@@ -4421,6 +4441,335 @@ func TestPrometheusWithModule(t *testing.T) {
 
 }
 
+func TestFlakyPrometheusRouterConnectionMetrics(t *testing.T) {
+	t.Parallel()
+
+	t.Run("validate router connection metrics are not present by default", func(t *testing.T) {
+		t.Parallel()
+
+		promRegistry := prometheus.NewRegistry()
+		metricReader := metric.NewManualReader()
+
+		testenv.Run(t, &testenv.Config{
+			MetricReader:       metricReader,
+			PrometheusRegistry: promRegistry,
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			mf, err := promRegistry.Gather()
+			require.NoError(t, err)
+
+			routerConnectionTotal := findMetricFamilyByName(mf, "router_connection_total")
+			require.Nil(t, routerConnectionTotal)
+		})
+	})
+
+	t.Run("validate router connection metrics are present when enabled", func(t *testing.T) {
+		t.Parallel()
+
+		promRegistry := prometheus.NewRegistry()
+		metricReader := metric.NewManualReader()
+
+		testenv.Run(t, &testenv.Config{
+			MetricReader:       metricReader,
+			PrometheusRegistry: promRegistry,
+			MetricOptions: testenv.MetricOptions{
+				EnablePrometheusConnectionMetrics: true,
+			},
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id isAvailable } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			mf, err := promRegistry.Gather()
+			require.NoError(t, err)
+
+			t.Run("verify max connections", func(t *testing.T) {
+				metricFamily := findMetricFamilyByName(mf, "router_http_client_max_connections")
+
+				metrics := metricFamily.GetMetric()
+				require.Len(t, metrics, 1)
+
+				connectionTotal := metrics[0]
+
+				require.Equal(t, 100.0, *connectionTotal.Gauge.Value)
+
+				expected := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+				}
+				require.Equal(t, expected, connectionTotal.Label)
+
+			})
+
+			t.Run("verify connections active", func(t *testing.T) {
+				metricFamily := findMetricFamilyByName(mf, "router_http_client_active_connections")
+				metrics := metricFamily.GetMetric()
+				require.Len(t, metrics, 2)
+
+				metricDataPoint1 := metrics[0]
+				require.Greater(t, *metricDataPoint1.Gauge.Value, 0.0)
+				expected1 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint1)),
+					},
+				}
+				require.Equal(t, expected1, metricDataPoint1.Label)
+
+				metricDataPoint2 := metrics[1]
+				require.Greater(t, *metricDataPoint1.Gauge.Value, 0.0)
+				expected2 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint2)),
+					},
+				}
+				require.Equal(t, expected2, metricDataPoint2.Label)
+			})
+
+			t.Run("verify connection total duration", func(t *testing.T) {
+				metricFamily := findMetricFamilyByName(mf, "router_http_client_connection_acquire_duration")
+				metrics := metricFamily.GetMetric()
+				require.Len(t, metrics, 2)
+
+				metricDataPoint1 := metrics[0]
+				require.Greater(t, *metricDataPoint1.Histogram.SampleSum, 0.0)
+				expected1 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint1)),
+					},
+					{
+						Name:  PointerOf("wg_http_client_reused_connection"),
+						Value: PointerOf("false"),
+					},
+					{
+						Name:  PointerOf("wg_subgraph_name"),
+						Value: PointerOf("employees"),
+					},
+				}
+				require.Equal(t, expected1, metricDataPoint1.Label)
+
+				metricDataPoint2 := metrics[1]
+				require.Greater(t, *metricDataPoint2.Histogram.SampleSum, 0.0)
+				expected2 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint2)),
+					},
+					{
+						Name:  PointerOf("wg_http_client_reused_connection"),
+						Value: PointerOf("false"),
+					},
+					{
+						Name:  PointerOf("wg_subgraph_name"),
+						Value: PointerOf("availability"),
+					},
+				}
+				require.Equal(t, expected2, metricDataPoint2.Label)
+			})
+
+		})
+	})
+
+	t.Run("verify with custom subgraph transport configs", func(t *testing.T) {
+		t.Parallel()
+
+		promRegistry := prometheus.NewRegistry()
+		metricReader := metric.NewManualReader()
+
+		trafficConfig := config.TrafficShapingRules{
+			All: config.GlobalSubgraphRequestRule{
+				RequestTimeout: PointerOf(200 * time.Millisecond),
+			},
+			Subgraphs: map[string]*config.GlobalSubgraphRequestRule{
+				"availability": {
+					RequestTimeout: PointerOf(300 * time.Millisecond),
+				},
+			},
+		}
+
+		testenv.Run(t, &testenv.Config{
+			MetricReader:       metricReader,
+			PrometheusRegistry: promRegistry,
+			MetricOptions: testenv.MetricOptions{
+				EnablePrometheusConnectionMetrics: true,
+			},
+			RouterOptions: []core.Option{
+				core.WithSubgraphTransportOptions(
+					core.NewSubgraphTransportOptions(trafficConfig)),
+			},
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id isAvailable } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			mf, err := promRegistry.Gather()
+			require.NoError(t, err)
+
+			t.Run("verify max connections", func(t *testing.T) {
+				metricFamily := findMetricFamilyByName(mf, "router_http_client_max_connections")
+
+				metrics := metricFamily.GetMetric()
+				require.Len(t, metrics, 2)
+
+				metricDataPoint1 := metrics[0]
+				require.Equal(t, 100.0, *metricDataPoint1.Gauge.Value)
+				expected1 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+				}
+				require.Equal(t, expected1, metricDataPoint1.Label)
+
+				metricDataPoint2 := metrics[1]
+				require.Equal(t, 100.0, *metricDataPoint2.Gauge.Value)
+				expected2 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("wg_subgraph_name"),
+						Value: PointerOf("availability"),
+					},
+				}
+				require.Equal(t, expected2, metricDataPoint2.Label)
+
+			})
+
+			t.Run("verify connections active", func(t *testing.T) {
+				metricFamily := findMetricFamilyByName(mf, "router_http_client_active_connections")
+				metrics := metricFamily.GetMetric()
+				require.Len(t, metrics, 2)
+
+				metricDataPoint1 := metrics[0]
+				require.Greater(t, *metricDataPoint1.Gauge.Value, 0.0)
+				expected1 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint1)),
+					},
+				}
+				require.Equal(t, expected1, metricDataPoint1.Label)
+
+				metricDataPoint2 := metrics[1]
+				require.Greater(t, *metricDataPoint1.Gauge.Value, 0.0)
+				expected2 := []*io_prometheus_client.LabelPair{
+					{
+						Name:  PointerOf("otel_scope_name"),
+						Value: PointerOf("cosmo.router.connections.prometheus"),
+					},
+					{
+						Name:  PointerOf("otel_scope_version"),
+						Value: PointerOf("0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_address"),
+						Value: PointerOf("127.0.0.1"),
+					},
+					{
+						Name:  PointerOf("server_port"),
+						Value: PointerOf(getPort(metricDataPoint2)),
+					},
+					{
+						Name:  PointerOf("wg_subgraph_name"),
+						Value: PointerOf("availability"),
+					},
+				}
+				require.Equal(t, expected2, metricDataPoint2.Label)
+			})
+		})
+	})
+}
+
 func TestExcludeAttributesWithCustomExporterPrometheus(t *testing.T) {
 	const (
 		UseCloudExporter                           = "use_cloud_exporter"
diff --git a/router-tests/telemetry/connection_metrics_test.go b/router-tests/telemetry/connection_metrics_test.go
new file mode 100644
index 0000000000..530f236a31
--- /dev/null
+++ b/router-tests/telemetry/connection_metrics_test.go
@@ -0,0 +1,274 @@
+package telemetry
+
+import (
+	"context"
+	"github.com/stretchr/testify/require"
+	integration "github.com/wundergraph/cosmo/router-tests"
+	"github.com/wundergraph/cosmo/router-tests/testenv"
+	"github.com/wundergraph/cosmo/router/core"
+	"github.com/wundergraph/cosmo/router/pkg/config"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/metric/metricdata"
+	"go.opentelemetry.io/otel/sdk/metric/metricdata/metricdatatest"
+	"testing"
+	"time"
+)
+
+func TestFlakyConnectionMetrics(t *testing.T) {
+	t.Parallel()
+
+	t.Run("validate router connection metrics are not present by default", func(t *testing.T) {
+		t.Parallel()
+
+		metricReader := metric.NewManualReader()
+		testenv.Run(t, &testenv.Config{
+			MetricReader: metricReader,
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			scopeMetric := integration.GetMetricScopeByName(rm.ScopeMetrics, "cosmo.router.connections")
+			require.Nil(t, scopeMetric)
+		})
+	})
+
+	t.Run("validate router connection metrics are present when enabled", func(t *testing.T) {
+		t.Parallel()
+
+		metricReader := metric.NewManualReader()
+		testenv.Run(t, &testenv.Config{
+			MetricReader: metricReader,
+			MetricOptions: testenv.MetricOptions{
+				EnableOTLPConnectionMetrics: true,
+			},
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id } }`,
+			})
+
+			time.Sleep(1 * time.Second)
+
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id isAvailable } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			scopeMetric := *integration.GetMetricScopeByName(rm.ScopeMetrics, "cosmo.router.connections")
+
+			require.Len(t, scopeMetric.Metrics, 3)
+
+			t.Run("verify max connections", func(t *testing.T) {
+				expected := metricdata.Metrics{
+					Name:        "router.http.client.max_connections",
+					Description: "Total number of max connections per subgraph",
+					Unit:        "",
+					Data: metricdata.Gauge[int64]{
+						DataPoints: []metricdata.DataPoint[int64]{
+							{},
+						},
+					},
+				}
+				metricdatatest.AssertEqual(t, expected, scopeMetric.Metrics[0], metricdatatest.IgnoreTimestamp(), metricdatatest.IgnoreValue())
+			})
+
+			t.Run("verify connections active", func(t *testing.T) {
+				metrics := scopeMetric.Metrics[2]
+
+				expected := metricdata.Metrics{
+					Name:        "router.http.client.active_connections",
+					Description: "Connections active",
+					Unit:        "",
+					Data: metricdata.Gauge[int64]{
+						DataPoints: []metricdata.DataPoint[int64]{
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 0, "gauge")),
+								),
+								Value: 1,
+							},
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 1, "gauge")),
+								),
+								Value: 1,
+							},
+						},
+					},
+				}
+
+				metricdatatest.AssertEqual(t, expected, metrics, metricdatatest.IgnoreTimestamp(), metricdatatest.IgnoreValue())
+			})
+
+			t.Run("verify connection total duration", func(t *testing.T) {
+				metrics := scopeMetric.Metrics[1]
+
+				actualHistogram, ok := metrics.Data.(metricdata.Histogram[float64])
+				require.True(t, ok)
+				require.Greater(t, actualHistogram.DataPoints[0].Sum, 0.0)
+
+				expected := metricdata.Metrics{
+					Name:        "router.http.client.connection.acquire_duration",
+					Description: "Total connection acquire duration",
+					Unit:        "s",
+					Data: metricdata.Histogram[float64]{
+						Temporality: metricdata.CumulativeTemporality,
+						DataPoints: []metricdata.HistogramDataPoint[float64]{
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 0, "")),
+									otel.WgClientReusedConnection.Bool(false),
+									otel.WgSubgraphName.String("employees"),
+								),
+							},
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 1, "")),
+									otel.WgClientReusedConnection.Bool(true),
+									otel.WgSubgraphName.String("employees"),
+								),
+							},
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 2, "")),
+									otel.WgClientReusedConnection.Bool(false),
+									otel.WgSubgraphName.String("availability"),
+								),
+							},
+						},
+					},
+				}
+
+				metricdatatest.AssertEqual(t, expected, metrics, metricdatatest.IgnoreTimestamp(), metricdatatest.IgnoreValue())
+			})
+
+		})
+	})
+
+	t.Run("verify custom subgraph transport configs", func(t *testing.T) {
+		t.Parallel()
+
+		trafficConfig := config.TrafficShapingRules{
+			All: config.GlobalSubgraphRequestRule{
+				RequestTimeout: integration.ToPtr(200 * time.Millisecond),
+			},
+			Subgraphs: map[string]*config.GlobalSubgraphRequestRule{
+				"availability": {
+					RequestTimeout: integration.ToPtr(300 * time.Millisecond),
+				},
+			},
+		}
+
+		metricReader := metric.NewManualReader()
+		testenv.Run(t, &testenv.Config{
+			MetricReader: metricReader,
+			MetricOptions: testenv.MetricOptions{
+				EnableOTLPConnectionMetrics: true,
+			},
+			RouterOptions: []core.Option{
+				core.WithSubgraphTransportOptions(
+					core.NewSubgraphTransportOptions(trafficConfig)),
+			},
+		}, func(t *testing.T, xEnv *testenv.Environment) {
+			xEnv.MakeGraphQLRequestOK(testenv.GraphQLRequest{
+				Query: `query { employees { id isAvailable } }`,
+			})
+			rm := metricdata.ResourceMetrics{}
+			err := metricReader.Collect(context.Background(), &rm)
+			require.NoError(t, err)
+
+			scopeMetric := *integration.GetMetricScopeByName(rm.ScopeMetrics, "cosmo.router.connections")
+
+			require.Len(t, scopeMetric.Metrics, 3)
+
+			t.Run("verify max connections", func(t *testing.T) {
+				expected := metricdata.Metrics{
+					Name:        "router.http.client.max_connections",
+					Description: "Total number of max connections per subgraph",
+					Unit:        "",
+					Data: metricdata.Gauge[int64]{
+						DataPoints: []metricdata.DataPoint[int64]{
+							{},
+							{
+								Attributes: attribute.NewSet(
+									otel.WgSubgraphName.String("availability"),
+								),
+							},
+						},
+					},
+				}
+				metricdatatest.AssertEqual(t, expected, scopeMetric.Metrics[0], metricdatatest.IgnoreTimestamp(), metricdatatest.IgnoreValue())
+			})
+
+			t.Run("verify connections active", func(t *testing.T) {
+				metrics := scopeMetric.Metrics[2]
+
+				expected := metricdata.Metrics{
+					Name:        "router.http.client.active_connections",
+					Description: "Connections active",
+					Unit:        "",
+					Data: metricdata.Gauge[int64]{
+						DataPoints: []metricdata.DataPoint[int64]{
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 0, "gauge")),
+								),
+								Value: 1,
+							},
+							{
+								Attributes: attribute.NewSet(
+									otel.ServerAddress.String("127.0.0.1"),
+									otel.ServerPort.String(getPort(t, metrics, 1, "gauge")),
+									otel.WgSubgraphName.String("availability"),
+								),
+								Value: 1,
+							},
+						},
+					},
+				}
+
+				metricdatatest.AssertEqual(t, expected, metrics, metricdatatest.IgnoreTimestamp())
+			})
+		})
+	})
+
+}
+
+// Since we cannot really predict the host and port we use this to get the host
+func getPort(t *testing.T, metric metricdata.Metrics, dataPointIndex int, typeName string) string {
+	t.Helper()
+
+	if typeName == "" {
+		typeName = "histogram"
+	}
+
+	var hostAttribute attribute.KeyValue
+	var retrieved bool
+
+	switch typeName {
+	case "histogram":
+		histogramDp := metric.Data.(metricdata.Histogram[float64]).DataPoints[dataPointIndex]
+		hostAttribute, retrieved = histogramDp.Attributes.Get(1)
+	case "gauge":
+		actualDp := metric.Data.(metricdata.Gauge[int64]).DataPoints[dataPointIndex]
+		hostAttribute, retrieved = actualDp.Attributes.Get(1)
+	}
+
+	require.True(t, retrieved)
+	require.True(t, hostAttribute.Valid())
+
+	return hostAttribute.Value.AsString()
+}
diff --git a/router-tests/testenv/testenv.go b/router-tests/testenv/testenv.go
index 3d5e7eba41..1e9fc28d6b 100644
--- a/router-tests/testenv/testenv.go
+++ b/router-tests/testenv/testenv.go
@@ -251,13 +251,15 @@ type EngineStatOptions struct {
 }
 
 type MetricOptions struct {
-	MetricExclusions             MetricExclusions
-	EnableRuntimeMetrics         bool
-	EnableOTLPRouterCache        bool
-	EnablePrometheusRouterCache  bool
-	OTLPEngineStatsOptions       EngineStatOptions
-	PrometheusEngineStatsOptions EngineStatOptions
-	PrometheusSchemaFieldUsage   PrometheusSchemaFieldUsage
+	MetricExclusions                  MetricExclusions
+	EnableRuntimeMetrics              bool
+	EnableOTLPRouterCache             bool
+	EnablePrometheusRouterCache       bool
+	OTLPEngineStatsOptions            EngineStatOptions
+	PrometheusEngineStatsOptions      EngineStatOptions
+	PrometheusSchemaFieldUsage        PrometheusSchemaFieldUsage
+	EnableOTLPConnectionMetrics       bool
+	EnablePrometheusConnectionMetrics bool
 }
 
 type PrometheusSchemaFieldUsage struct {
@@ -1377,11 +1379,12 @@ func configureRouter(listenerAddr string, testConfig *Config, routerConfig *node
 
 	if testConfig.PrometheusRegistry != nil {
 		prometheusConfig = rmetric.PrometheusConfig{
-			Enabled:      true,
-			ListenAddr:   fmt.Sprintf("localhost:%d", testConfig.PrometheusPort),
-			Path:         "/metrics",
-			TestRegistry: testConfig.PrometheusRegistry,
-			GraphqlCache: testConfig.MetricOptions.EnablePrometheusRouterCache,
+			Enabled:         true,
+			ListenAddr:      fmt.Sprintf("localhost:%d", testConfig.PrometheusPort),
+			Path:            "/metrics",
+			TestRegistry:    testConfig.PrometheusRegistry,
+			GraphqlCache:    testConfig.MetricOptions.EnablePrometheusRouterCache,
+			ConnectionStats: testConfig.MetricOptions.EnablePrometheusConnectionMetrics,
 			EngineStats: rmetric.EngineStatsConfig{
 				Subscription: testConfig.MetricOptions.PrometheusEngineStatsOptions.EnableSubscription,
 			},
@@ -1406,9 +1409,10 @@ func configureRouter(listenerAddr string, testConfig *Config, routerConfig *node
 					Enabled: true,
 				},
 				OTLP: config.MetricsOTLP{
-					Enabled:       true,
-					RouterRuntime: testConfig.MetricOptions.EnableRuntimeMetrics,
-					GraphqlCache:  testConfig.MetricOptions.EnableOTLPRouterCache,
+					Enabled:         true,
+					RouterRuntime:   testConfig.MetricOptions.EnableRuntimeMetrics,
+					GraphqlCache:    testConfig.MetricOptions.EnableOTLPRouterCache,
+					ConnectionStats: testConfig.MetricOptions.EnableOTLPConnectionMetrics,
 					EngineStats: config.EngineStats{
 						Subscriptions: testConfig.MetricOptions.OTLPEngineStatsOptions.EnableSubscription,
 					},
diff --git a/router/core/engine_loader_hooks.go b/router/core/engine_loader_hooks.go
index 9cb95d9e40..e17a01cbb6 100644
--- a/router/core/engine_loader_hooks.go
+++ b/router/core/engine_loader_hooks.go
@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"github.com/wundergraph/cosmo/router/internal/httpclient"
 	"github.com/wundergraph/cosmo/router/internal/requestlogger"
 	"github.com/wundergraph/cosmo/router/internal/unique"
 	"github.com/wundergraph/cosmo/router/pkg/metric"
@@ -77,6 +78,8 @@ func (f *engineLoaderHooks) OnLoad(ctx context.Context, ds resolve.DataSourceInf
 		return ctx
 	}
 
+	ctx = context.WithValue(ctx, httpclient.CurrentSubgraphContextKey{}, ds.Name)
+
 	ctx, _ = f.tracer.Start(ctx, "Engine - Fetch",
 		trace.WithAttributes([]attribute.KeyValue{
 			rotel.WgSubgraphName.String(ds.Name),
diff --git a/router/core/executor.go b/router/core/executor.go
index 7ad8b76afd..49dc7ea8c6 100644
--- a/router/core/executor.go
+++ b/router/core/executor.go
@@ -61,6 +61,7 @@ type ExecutorBuildOptions struct {
 	ApolloCompatibilityFlags       config.ApolloCompatibilityFlags
 	ApolloRouterCompatibilityFlags config.ApolloRouterCompatibilityFlags
 	HeartbeatInterval              time.Duration
+	TraceClientRequired            bool
 	PluginsEnabled                 bool
 }
 
diff --git a/router/core/graph_server.go b/router/core/graph_server.go
index 24d166bf22..9645bbbbed 100644
--- a/router/core/graph_server.go
+++ b/router/core/graph_server.go
@@ -99,8 +99,10 @@ type (
 		runtimeMetrics          *rmetric.RuntimeMetrics
 		otlpEngineMetrics       *rmetric.EngineMetrics
 		prometheusEngineMetrics *rmetric.EngineMetrics
+		connectionMetrics       *rmetric.ConnectionMetrics
 		hostName                string
 		routerListenAddr        string
+		traceDialer             *TraceDialer
 		pluginHost              *routerplugin.Host
 	}
 )
@@ -117,13 +119,19 @@ func newGraphServer(ctx context.Context, r *Router, routerConfig *nodev1.RouterC
 		return nil, fmt.Errorf(`the compatibility version "%s" is not compatible with this router version`, routerConfig.CompatibilityVersion)
 	}
 
+	isConnStoreEnabled := r.Config.metricConfig.OpenTelemetry.ConnectionStats || r.Config.metricConfig.Prometheus.ConnectionStats
+	var traceDialer *TraceDialer
+	if isConnStoreEnabled {
+		traceDialer = NewTraceDialer()
+	}
+
 	// Base transport
-	baseTransport := newHTTPTransport(r.subgraphTransportOptions.TransportRequestOptions, proxy)
+	baseTransport := newHTTPTransport(r.subgraphTransportOptions.TransportRequestOptions, proxy, traceDialer, "")
 
 	// Subgraph transports
 	subgraphTransports := map[string]*http.Transport{}
 	for subgraph, subgraphOpts := range r.subgraphTransportOptions.SubgraphMap {
-		subgraphBaseTransport := newHTTPTransport(subgraphOpts, proxy)
+		subgraphBaseTransport := newHTTPTransport(subgraphOpts, proxy, traceDialer, subgraph)
 		subgraphTransports[subgraph] = subgraphBaseTransport
 	}
 
@@ -136,6 +144,7 @@ func newGraphServer(ctx context.Context, r *Router, routerConfig *nodev1.RouterC
 		baseTransport:           baseTransport,
 		subgraphTransports:      subgraphTransports,
 		playgroundHandler:       r.playgroundHandler,
+		traceDialer:             traceDialer,
 		baseRouterConfigVersion: routerConfig.GetVersion(),
 		inFlightRequests:        &atomic.Uint64{},
 		graphMuxList:            make([]*graphMux, 0, 1),
@@ -183,6 +192,21 @@ func newGraphServer(ctx context.Context, r *Router, routerConfig *nodev1.RouterC
 		}
 	}
 
+	if isConnStoreEnabled {
+		connStore, err := rmetric.NewConnectionMetricStore(
+			s.logger,
+			nil,
+			s.otlpMeterProvider,
+			s.promMeterProvider,
+			s.metricConfig,
+			s.traceDialer.connectionPoolStats,
+		)
+		if err != nil {
+			return nil, err
+		}
+		s.connectionMetrics = connStore
+	}
+
 	if err := s.setupEngineStatistics(mappedMetricAttributes); err != nil {
 		return nil, fmt.Errorf("failed to setup engine statistics: %w", err)
 	}
@@ -984,6 +1008,7 @@ func (s *graphServer) buildGraphMux(ctx context.Context,
 			PreHandlers:              s.preOriginHandlers,
 			PostHandlers:             s.postOriginHandlers,
 			MetricStore:              gm.metricStore,
+			ConnectionMetricStore:    s.connectionMetrics,
 			RetryOptions: retrytransport.RetryOptions{
 				Enabled:       s.retryOptions.Enabled,
 				MaxRetryCount: s.retryOptions.MaxRetryCount,
@@ -997,6 +1022,7 @@ func (s *graphServer) buildGraphMux(ctx context.Context,
 			TracePropagators:              s.compositePropagator,
 			LocalhostFallbackInsideDocker: s.localhostFallbackInsideDocker,
 			Logger:                        s.logger,
+			EnableTraceClient:             s.connectionMetrics != nil,
 		},
 	}
 
@@ -1468,6 +1494,12 @@ func (s *graphServer) Shutdown(ctx context.Context) error {
 		}
 	}
 
+	if s.connectionMetrics != nil {
+		if aErr := s.connectionMetrics.Shutdown(ctx); aErr != nil {
+			finalErr = errors.Join(finalErr, aErr)
+		}
+	}
+
 	if s.otlpEngineMetrics != nil {
 		if err := s.otlpEngineMetrics.Shutdown(); err != nil {
 			finalErr = errors.Join(finalErr, err)
diff --git a/router/core/router.go b/router/core/router.go
index b750f26f98..07e23a241e 100644
--- a/router/core/router.go
+++ b/router/core/router.go
@@ -2029,14 +2029,14 @@ func WithDemoMode(demoMode bool) Option {
 
 type ProxyFunc func(req *http.Request) (*url.URL, error)
 
-func newHTTPTransport(opts *TransportRequestOptions, proxy ProxyFunc) *http.Transport {
+func newHTTPTransport(opts *TransportRequestOptions, proxy ProxyFunc, traceDialer *TraceDialer, subgraph string) *http.Transport {
 	dialer := &net.Dialer{
 		Timeout:   opts.DialTimeout,
 		KeepAlive: opts.KeepAliveProbeInterval,
 	}
 	// Great source of inspiration: https://gitlab.com/gitlab-org/gitlab-pages
 	// A pages proxy in go that handles tls to upstreams, rate limiting, and more
-	return &http.Transport{
+	transport := &http.Transport{
 		DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
 			return dialer.DialContext(ctx, network, addr)
 		},
@@ -2059,6 +2059,13 @@ func newHTTPTransport(opts *TransportRequestOptions, proxy ProxyFunc) *http.Tran
 		// This will prevent the transport from handling the proxy when it is not needed.
 		Proxy: proxy,
 	}
+
+	if traceDialer != nil {
+		transport.DialContext = traceDialer.WrapDial(transport.DialContext, subgraph)
+		traceDialer.connectionPoolStats.AddSubgraphHostCount(subgraph, int64(opts.MaxConnsPerHost))
+	}
+
+	return transport
 }
 
 func TraceConfigFromTelemetry(cfg *config.Telemetry) *rtrace.Config {
@@ -2180,9 +2187,10 @@ func MetricConfigFromTelemetry(cfg *config.Telemetry) *rmetric.Config {
 		Attributes:         cfg.Metrics.Attributes,
 		ResourceAttributes: buildResourceAttributes(cfg.ResourceAttributes),
 		OpenTelemetry: rmetric.OpenTelemetry{
-			Enabled:       cfg.Metrics.OTLP.Enabled,
-			RouterRuntime: cfg.Metrics.OTLP.RouterRuntime,
-			GraphqlCache:  cfg.Metrics.OTLP.GraphqlCache,
+			Enabled:         cfg.Metrics.OTLP.Enabled,
+			RouterRuntime:   cfg.Metrics.OTLP.RouterRuntime,
+			GraphqlCache:    cfg.Metrics.OTLP.GraphqlCache,
+			ConnectionStats: cfg.Metrics.OTLP.ConnectionStats,
 			EngineStats: rmetric.EngineStatsConfig{
 				Subscription: cfg.Metrics.OTLP.EngineStats.Subscriptions,
 			},
@@ -2191,10 +2199,11 @@ func MetricConfigFromTelemetry(cfg *config.Telemetry) *rmetric.Config {
 			ExcludeMetricLabels: cfg.Metrics.OTLP.ExcludeMetricLabels,
 		},
 		Prometheus: rmetric.PrometheusConfig{
-			Enabled:      cfg.Metrics.Prometheus.Enabled,
-			ListenAddr:   cfg.Metrics.Prometheus.ListenAddr,
-			Path:         cfg.Metrics.Prometheus.Path,
-			GraphqlCache: cfg.Metrics.Prometheus.GraphqlCache,
+			Enabled:         cfg.Metrics.Prometheus.Enabled,
+			ListenAddr:      cfg.Metrics.Prometheus.ListenAddr,
+			Path:            cfg.Metrics.Prometheus.Path,
+			GraphqlCache:    cfg.Metrics.Prometheus.GraphqlCache,
+			ConnectionStats: cfg.Metrics.Prometheus.ConnectionStats,
 			EngineStats: rmetric.EngineStatsConfig{
 				Subscription: cfg.Metrics.Prometheus.EngineStats.Subscriptions,
 			},
diff --git a/router/core/trace_dialer.go b/router/core/trace_dialer.go
new file mode 100644
index 0000000000..8f3f0c9fce
--- /dev/null
+++ b/router/core/trace_dialer.go
@@ -0,0 +1,58 @@
+package core
+
+import (
+	"context"
+	"github.com/wundergraph/cosmo/router/pkg/metric"
+	"net"
+)
+
+type TraceDialer struct {
+	connectionPoolStats *metric.ConnectionPoolStats
+}
+
+func NewTraceDialer() *TraceDialer {
+	return &TraceDialer{
+		connectionPoolStats: metric.NewConnectionPoolStats(),
+	}
+}
+
+type dialerFunc func(ctx context.Context, network, address string) (net.Conn, error)
+
+func (t *TraceDialer) WrapDial(base dialerFunc, subgraph string) dialerFunc {
+	return func(ctx context.Context, network, address string) (net.Conn, error) {
+		key := metric.SubgraphHostKey{
+			Subgraph: subgraph,
+			Host:     address,
+		}
+
+		counter := t.connectionPoolStats.GetCounter(key)
+		counter.Add(1)
+
+		conn, err := base(ctx, network, address)
+		if err != nil {
+			counter.Add(-1)
+			return conn, err
+		}
+
+		// wrap conn to decrement on Close
+		return &trackedConn{
+			Conn: conn,
+			onClose: func() {
+				counter.Add(-1)
+			},
+		}, nil
+	}
+}
+
+type trackedConn struct {
+	net.Conn
+	onClose func()
+}
+
+func (c *trackedConn) Close() error {
+	err := c.Conn.Close()
+	if c.onClose != nil {
+		c.onClose()
+	}
+	return err
+}
diff --git a/router/core/transport.go b/router/core/transport.go
index 233548db53..1f932d2351 100644
--- a/router/core/transport.go
+++ b/router/core/transport.go
@@ -3,6 +3,7 @@ package core
 import (
 	"bytes"
 	"fmt"
+	traceclient "github.com/wundergraph/cosmo/router/internal/httpclient"
 	"io"
 	"net/http"
 	"net/url"
@@ -59,16 +60,21 @@ type sfCacheItem struct {
 }
 
 func NewCustomTransport(
-	logger *zap.Logger,
-	roundTripper http.RoundTripper,
+	baseRoundTripper http.RoundTripper,
 	retryOptions retrytransport.RetryOptions,
 	metricStore metric.Store,
+	connectionMetricStore metric.ConnectionMetricStore,
 	enableSingleFlight bool,
+	enableTraceClient bool,
 ) *CustomTransport {
 	ct := &CustomTransport{
 		metricStore: metricStore,
 	}
 
+	if enableTraceClient {
+		baseRoundTripper = traceclient.NewTraceInjectingRoundTripper(baseRoundTripper, connectionMetricStore)
+	}
+
 	if retryOptions.Enabled {
 		// The round trip method is almost always called via the http.Client RoundTripper interface
 		// as a result we cannot pass in the request context logger directly, since this will break the interface
@@ -79,9 +85,9 @@ func NewCustomTransport(
 			reqContext := getRequestContext(req.Context())
 			return reqContext.Logger()
 		}
-		ct.roundTripper = retrytransport.NewRetryHTTPTransport(roundTripper, retryOptions, getRequestContextLogger)
+		ct.roundTripper = retrytransport.NewRetryHTTPTransport(baseRoundTripper, retryOptions, getRequestContextLogger)
 	} else {
-		ct.roundTripper = roundTripper
+		ct.roundTripper = baseRoundTripper
 	}
 	if enableSingleFlight {
 		ct.sf = make(map[uint64]*sfCacheItem)
@@ -318,9 +324,11 @@ type TransportFactory struct {
 	retryOptions                  retrytransport.RetryOptions
 	localhostFallbackInsideDocker bool
 	metricStore                   metric.Store
+	connectionMetricStore         metric.ConnectionMetricStore
 	logger                        *zap.Logger
 	tracerProvider                *sdktrace.TracerProvider
 	tracePropagators              propagation.TextMapPropagator
+	enableTraceClient             bool
 }
 
 var _ ApiTransportFactory = TransportFactory{}
@@ -332,9 +340,11 @@ type TransportOptions struct {
 	RetryOptions                  retrytransport.RetryOptions
 	LocalhostFallbackInsideDocker bool
 	MetricStore                   metric.Store
+	ConnectionMetricStore         metric.ConnectionMetricStore
 	Logger                        *zap.Logger
 	TracerProvider                *sdktrace.TracerProvider
 	TracePropagators              propagation.TextMapPropagator
+	EnableTraceClient             bool
 }
 
 type SubscriptionClientOptions struct {
@@ -351,9 +361,11 @@ func NewTransport(opts *TransportOptions) *TransportFactory {
 		retryOptions:                  opts.RetryOptions,
 		localhostFallbackInsideDocker: opts.LocalhostFallbackInsideDocker,
 		metricStore:                   opts.MetricStore,
+		connectionMetricStore:         opts.ConnectionMetricStore,
 		logger:                        opts.Logger,
 		tracerProvider:                opts.TracerProvider,
 		tracePropagators:              opts.TracePropagators,
+		enableTraceClient:             opts.EnableTraceClient,
 	}
 }
 
@@ -393,11 +405,12 @@ func (t TransportFactory) RoundTripper(enableSingleFlight bool, baseTransport ht
 		}),
 	)
 	tp := NewCustomTransport(
-		t.logger,
 		traceTransport,
 		t.retryOptions,
 		t.metricStore,
+		t.connectionMetricStore,
 		enableSingleFlight,
+		t.enableTraceClient,
 	)
 
 	tp.preHandlers = t.preHandlers
diff --git a/router/internal/httpclient/traceclient.go b/router/internal/httpclient/traceclient.go
new file mode 100644
index 0000000000..857a6ade40
--- /dev/null
+++ b/router/internal/httpclient/traceclient.go
@@ -0,0 +1,124 @@
+package httpclient
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptrace"
+	"time"
+
+	"github.com/wundergraph/cosmo/router/pkg/metric"
+	rotel "github.com/wundergraph/cosmo/router/pkg/otel"
+)
+
+type AcquiredConnection struct {
+	Time     time.Time
+	IdleTime time.Duration
+	Reused   bool
+	WasIdle  bool
+}
+
+type GetConnection struct {
+	Time     time.Time
+	HostPort string
+}
+
+type ClientTrace struct {
+	ConnectionGet      *GetConnection
+	ConnectionAcquired *AcquiredConnection
+}
+
+type ClientTraceContextKey struct{}
+type CurrentSubgraphContextKey struct{}
+
+type TraceInjectingRoundTripper struct {
+	base                  http.RoundTripper
+	connectionMetricStore metric.ConnectionMetricStore
+}
+
+func NewTraceInjectingRoundTripper(base http.RoundTripper, connectionMetricStore metric.ConnectionMetricStore) *TraceInjectingRoundTripper {
+	return &TraceInjectingRoundTripper{
+		base:                  base,
+		connectionMetricStore: connectionMetricStore,
+	}
+}
+
+func GetClientTraceFromContext(ctx context.Context) *ClientTrace {
+	value := ctx.Value(ClientTraceContextKey{})
+	// Return no-op context if the subgraph context key was never set
+	if value == nil {
+		return &ClientTrace{}
+	}
+	return value.(*ClientTrace)
+}
+
+func InitTraceContext(ctx context.Context) context.Context {
+	trace := &ClientTrace{}
+	ctx = context.WithValue(ctx, ClientTraceContextKey{}, trace)
+	return ctx
+}
+
+func (t *TraceInjectingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	ctx := req.Context()
+	ctx = InitTraceContext(ctx)
+
+	trace := t.getClientTrace(ctx)
+
+	req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
+	trip, err := t.base.RoundTrip(req)
+
+	CalculateConnectionMetrics(ctx, t.connectionMetricStore)
+
+	return trip, err
+}
+
+func (t *TraceInjectingRoundTripper) getClientTrace(ctx context.Context) *httptrace.ClientTrace {
+	eC := GetClientTraceFromContext(ctx)
+
+	trace := &httptrace.ClientTrace{
+		GetConn: func(hostPort string) {
+			eC.ConnectionGet = &GetConnection{
+				Time:     time.Now(),
+				HostPort: hostPort,
+			}
+		},
+		GotConn: func(info httptrace.GotConnInfo) {
+			eC.ConnectionAcquired = &AcquiredConnection{
+				Time:     time.Now(),
+				Reused:   info.Reused,
+				WasIdle:  info.WasIdle,
+				IdleTime: info.IdleTime,
+			}
+		},
+	}
+	return trace
+}
+
+func CalculateConnectionMetrics(ctx context.Context, store metric.ConnectionMetricStore) {
+	if store == nil {
+		return
+	}
+
+	var subgraph string
+	subgraphCtxVal := ctx.Value(CurrentSubgraphContextKey{})
+	if subgraphCtxVal != nil {
+		subgraph = subgraphCtxVal.(string)
+	}
+
+	trace := GetClientTraceFromContext(ctx)
+
+	if trace.ConnectionGet != nil && trace.ConnectionAcquired != nil {
+		connAcquireTime := trace.ConnectionAcquired.Time.Sub(trace.ConnectionGet.Time).Seconds()
+
+		serverAttributes := rotel.GetServerAttributes(trace.ConnectionGet.HostPort)
+		serverAttributes = append(
+			serverAttributes,
+			rotel.WgClientReusedConnection.Bool(trace.ConnectionAcquired.Reused),
+			rotel.WgSubgraphName.String(subgraph),
+		)
+
+		store.MeasureConnectionAcquireDuration(ctx,
+			connAcquireTime,
+			serverAttributes...,
+		)
+	}
+}
diff --git a/router/pkg/config/config.go b/router/pkg/config/config.go
index 3e3f478b90..1e3619b7bb 100644
--- a/router/pkg/config/config.go
+++ b/router/pkg/config/config.go
@@ -94,6 +94,7 @@ type Prometheus struct {
 	Path                string      `yaml:"path" envDefault:"/metrics" env:"PROMETHEUS_HTTP_PATH"`
 	ListenAddr          string      `yaml:"listen_addr" envDefault:"127.0.0.1:8088" env:"PROMETHEUS_LISTEN_ADDR"`
 	GraphqlCache        bool        `yaml:"graphql_cache" envDefault:"false" env:"PROMETHEUS_GRAPHQL_CACHE"`
+	ConnectionStats     bool        `yaml:"connection_stats" envDefault:"false" env:"PROMETHEUS_CONNECTION_STATS"`
 	EngineStats         EngineStats `yaml:"engine_stats" envPrefix:"PROMETHEUS_"`
 	ExcludeMetrics      RegExArray  `yaml:"exclude_metrics,omitempty" env:"PROMETHEUS_EXCLUDE_METRICS"`
 	ExcludeMetricLabels RegExArray  `yaml:"exclude_metric_labels,omitempty" env:"PROMETHEUS_EXCLUDE_METRIC_LABELS"`
@@ -126,6 +127,7 @@ type MetricsOTLP struct {
 	Enabled             bool                  `yaml:"enabled" envDefault:"true" env:"METRICS_OTLP_ENABLED"`
 	RouterRuntime       bool                  `yaml:"router_runtime" envDefault:"true" env:"METRICS_OTLP_ROUTER_RUNTIME"`
 	GraphqlCache        bool                  `yaml:"graphql_cache" envDefault:"false" env:"METRICS_OTLP_GRAPHQL_CACHE"`
+	ConnectionStats     bool                  `yaml:"connection_stats" envDefault:"false" env:"METRICS_OTLP_CONNECTION_STATS"`
 	EngineStats         EngineStats           `yaml:"engine_stats" envPrefix:"METRICS_OTLP_"`
 	ExcludeMetrics      RegExArray            `yaml:"exclude_metrics,omitempty" env:"METRICS_OTLP_EXCLUDE_METRICS"`
 	ExcludeMetricLabels RegExArray            `yaml:"exclude_metric_labels,omitempty" env:"METRICS_OTLP_EXCLUDE_METRIC_LABELS"`
diff --git a/router/pkg/config/config.schema.json b/router/pkg/config/config.schema.json
index ba1499b74a..4db935f30e 100644
--- a/router/pkg/config/config.schema.json
+++ b/router/pkg/config/config.schema.json
@@ -1000,6 +1000,11 @@
                   "default": false,
                   "description": "Enable the collection of metrics for the GraphQL operation router caches. The default value is false."
                 },
+                "connection_stats": {
+                  "type": "boolean",
+                  "default": false,
+                  "description": "Enable the collection of connection stats. The default value is false."
+                },
                 "engine_stats": {
                   "type": "object",
                   "additionalProperties": false,
@@ -1095,6 +1100,11 @@
                   "default": false,
                   "description": "Enable the collection of metrics for the GraphQL operation router caches. The default value is false."
                 },
+                "connection_stats": {
+                  "type": "boolean",
+                  "default": false,
+                  "description": "Enable the collection of connection stats. The default value is false."
+                },
                 "engine_stats": {
                   "type": "object",
                   "additionalProperties": false,
diff --git a/router/pkg/config/fixtures/full.yaml b/router/pkg/config/fixtures/full.yaml
index 498919e478..92da596238 100644
--- a/router/pkg/config/fixtures/full.yaml
+++ b/router/pkg/config/fixtures/full.yaml
@@ -137,6 +137,7 @@ telemetry:
       enabled: true
       router_runtime: true
       graphql_cache: true
+      connection_stats: true
       engine_stats:
         subscriptions: true
       exclude_metric_labels: []
@@ -156,6 +157,7 @@ telemetry:
       path: '/metrics'
       listen_addr: '127.0.0.1:8088'
       graphql_cache: true
+      connection_stats: true
       engine_stats:
         subscriptions: true
       exclude_metrics: []
diff --git a/router/pkg/config/testdata/config_defaults.json b/router/pkg/config/testdata/config_defaults.json
index 910091117d..7cf527fb35 100644
--- a/router/pkg/config/testdata/config_defaults.json
+++ b/router/pkg/config/testdata/config_defaults.json
@@ -34,6 +34,7 @@
         "Enabled": true,
         "RouterRuntime": true,
         "GraphqlCache": false,
+        "ConnectionStats": false,
         "EngineStats": {
           "Subscriptions": false
         },
@@ -46,6 +47,7 @@
         "Path": "/metrics",
         "ListenAddr": "127.0.0.1:8088",
         "GraphqlCache": false,
+        "ConnectionStats": false,
         "EngineStats": {
           "Subscriptions": false
         },
diff --git a/router/pkg/config/testdata/config_full.json b/router/pkg/config/testdata/config_full.json
index 1c329e42c3..5ecf5861cf 100644
--- a/router/pkg/config/testdata/config_full.json
+++ b/router/pkg/config/testdata/config_full.json
@@ -44,6 +44,7 @@
         "Enabled": true,
         "RouterRuntime": true,
         "GraphqlCache": true,
+        "ConnectionStats": true,
         "EngineStats": {
           "Subscriptions": true
         },
@@ -65,6 +66,7 @@
         "Path": "/metrics",
         "ListenAddr": "127.0.0.1:8088",
         "GraphqlCache": true,
+        "ConnectionStats": true,
         "EngineStats": {
           "Subscriptions": true
         },
diff --git a/router/pkg/metric/config.go b/router/pkg/metric/config.go
index b2af2c4d2d..5c95a5ec9c 100644
--- a/router/pkg/metric/config.go
+++ b/router/pkg/metric/config.go
@@ -15,11 +15,12 @@ import (
 const DefaultServerName = "cosmo-router"
 
 type PrometheusConfig struct {
-	Enabled      bool
-	ListenAddr   string
-	Path         string
-	GraphqlCache bool
-	EngineStats  EngineStatsConfig
+	Enabled         bool
+	ConnectionStats bool
+	ListenAddr      string
+	Path            string
+	GraphqlCache    bool
+	EngineStats     EngineStatsConfig
 	// Metrics to exclude from Prometheus exporter
 	ExcludeMetrics []*regexp.Regexp
 	// Metric labels to exclude from Prometheus exporter
@@ -61,11 +62,12 @@ func (e *EngineStatsConfig) Enabled() bool {
 }
 
 type OpenTelemetry struct {
-	Enabled       bool
-	RouterRuntime bool
-	GraphqlCache  bool
-	EngineStats   EngineStatsConfig
-	Exporters     []*OpenTelemetryExporter
+	Enabled         bool
+	ConnectionStats bool
+	RouterRuntime   bool
+	GraphqlCache    bool
+	EngineStats     EngineStatsConfig
+	Exporters       []*OpenTelemetryExporter
 	// Metrics to exclude from the OTLP exporter.
 	ExcludeMetrics []*regexp.Regexp
 	// Metric labels to exclude from the OTLP exporter.
diff --git a/router/pkg/metric/connection_measurements.go b/router/pkg/metric/connection_measurements.go
new file mode 100644
index 0000000000..1407cc412a
--- /dev/null
+++ b/router/pkg/metric/connection_measurements.go
@@ -0,0 +1,66 @@
+package metric
+
+import (
+	"fmt"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// Connection metric constants
+const (
+	maxConnections            = "router.http.client.max_connections"
+	connectionsActive         = "router.http.client.active_connections"
+	connectionAcquireDuration = "router.http.client.connection.acquire_duration"
+)
+
+var (
+	maxConnectionOptions = []otelmetric.Int64GaugeOption{
+		otelmetric.WithDescription("Total number of max connections per subgraph"),
+	}
+
+	connectionAcquireDurationOptions = []otelmetric.Float64HistogramOption{
+		otelmetric.WithUnit("s"),
+		otelmetric.WithDescription("Total connection acquire duration"),
+	}
+
+	connectionsActiveOptions = []otelmetric.Int64ObservableGaugeOption{
+		otelmetric.WithDescription("Connections active"),
+	}
+)
+
+type connectionInstruments struct {
+	maxConnections            otelmetric.Int64Gauge
+	connectionAcquireDuration otelmetric.Float64Histogram
+	connectionsActive         otelmetric.Int64ObservableGauge
+}
+
+func newConnectionInstruments(meter otelmetric.Meter) (*connectionInstruments, error) {
+	maxConnectionsGauge, err := meter.Int64Gauge(
+		maxConnections,
+		maxConnectionOptions...,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create connection total counter: %w", err)
+	}
+
+	acquireDurationHistogram, err := meter.Float64Histogram(
+		connectionAcquireDuration,
+		connectionAcquireDurationOptions...,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create connection acquire duration histogram: %w", err)
+	}
+
+	connectionsActiveGauge, err := meter.Int64ObservableGauge(
+		connectionsActive,
+		connectionsActiveOptions...,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create connections available: %w", err)
+	}
+
+	return &connectionInstruments{
+		maxConnections:            maxConnectionsGauge,
+		connectionAcquireDuration: acquireDurationHistogram,
+		connectionsActive:         connectionsActiveGauge,
+	}, nil
+}
diff --git a/router/pkg/metric/connection_metric_store.go b/router/pkg/metric/connection_metric_store.go
new file mode 100644
index 0000000000..2ed2ba9b15
--- /dev/null
+++ b/router/pkg/metric/connection_metric_store.go
@@ -0,0 +1,127 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+
+	"go.opentelemetry.io/otel/attribute"
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+// ConnectionMetricProvider is the interface that wraps the basic connection metric methods.
+// We maintain two providers, one for OTEL and one for Prometheus.
+type ConnectionMetricProvider interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption)
+	MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption)
+	Flush(ctx context.Context) error
+	Shutdown() error
+}
+
+// ConnectionMetricStore is the interface for connection and pool metrics only.
+type ConnectionMetricStore interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, attrs ...attribute.KeyValue)
+	Flush(ctx context.Context) error
+	Shutdown(ctx context.Context) error
+}
+
+type ConnectionMetrics struct {
+	baseAttributes []attribute.KeyValue
+	logger         *zap.Logger
+
+	otlpConnectionMetrics ConnectionMetricProvider
+	promConnectionMetrics ConnectionMetricProvider
+}
+
+func NewConnectionMetricStore(
+	logger *zap.Logger,
+	baseAttributes []attribute.KeyValue,
+	otelProvider, promProvider *metric.MeterProvider,
+	metricsConfig *Config,
+	connectionPoolStats *ConnectionPoolStats,
+) (*ConnectionMetrics, error) {
+	connMetrics := &ConnectionMetrics{
+		baseAttributes:        baseAttributes,
+		logger:                logger,
+		otlpConnectionMetrics: &noopConnectionMetricProvider{},
+		promConnectionMetrics: &noopConnectionMetricProvider{},
+	}
+
+	if metricsConfig.OpenTelemetry.ConnectionStats {
+		otlpMetrics, err := newOtlpConnectionMetrics(logger, otelProvider, connectionPoolStats, baseAttributes)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create otlp connection metrics: %w", err)
+		}
+		connMetrics.otlpConnectionMetrics = otlpMetrics
+	}
+
+	if metricsConfig.Prometheus.ConnectionStats {
+		promMetrics, err := newPromConnectionMetrics(logger, promProvider, connectionPoolStats, baseAttributes)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create prometheus connection metrics: %w", err)
+		}
+		connMetrics.promConnectionMetrics = promMetrics
+	}
+
+	return connMetrics, nil
+}
+
+func (c *ConnectionMetrics) MeasureMaxConnections(ctx context.Context, reused bool, attrs ...attribute.KeyValue) {
+	// Add the reused attribute to the base attributes
+	reusedAttr := otel.WgClientReusedConnection.Bool(reused)
+	allAttrs := append([]attribute.KeyValue{}, c.baseAttributes...)
+	allAttrs = append(allAttrs, reusedAttr)
+	allAttrs = append(allAttrs, attrs...)
+
+	opts := otelmetric.WithAttributes(allAttrs...)
+
+	c.otlpConnectionMetrics.MeasureMaxConnections(ctx, 1, opts)
+	c.promConnectionMetrics.MeasureMaxConnections(ctx, 1, opts)
+}
+
+func (c *ConnectionMetrics) MeasureConnectionAcquireDuration(ctx context.Context, duration float64, attrs ...attribute.KeyValue) {
+	copied := append([]attribute.KeyValue{}, c.baseAttributes...)
+	opts := otelmetric.WithAttributes(append(copied, attrs...)...)
+
+	c.otlpConnectionMetrics.MeasureConnectionAcquireDuration(ctx, duration, opts)
+	c.promConnectionMetrics.MeasureConnectionAcquireDuration(ctx, duration, opts)
+}
+
+// Flush flushes the metrics to the backend synchronously.
+func (h *ConnectionMetrics) Flush(ctx context.Context) error {
+	var err error
+
+	errOtlp := h.otlpConnectionMetrics.Flush(ctx)
+	if errOtlp != nil {
+		err = errors.Join(err, fmt.Errorf("failed to flush otlp metrics: %w", errOtlp))
+	}
+
+	errProm := h.promConnectionMetrics.Flush(ctx)
+	if errProm != nil {
+		err = errors.Join(err, fmt.Errorf("failed to flush prometheus metrics: %w", errProm))
+	}
+
+	return err
+}
+
+// Shutdown flushes the metrics and stops the runtime metrics.
+func (h *ConnectionMetrics) Shutdown(ctx context.Context) error {
+	var err error
+
+	if errFlush := h.Flush(ctx); errFlush != nil {
+		err = errors.Join(err, fmt.Errorf("failed to flush metrics: %w", errFlush))
+	}
+
+	if errProm := h.promConnectionMetrics.Shutdown(); errProm != nil {
+		err = errors.Join(err, fmt.Errorf("failed to shutdown prom metrics: %w", errProm))
+	}
+
+	if errOtlp := h.otlpConnectionMetrics.Shutdown(); errOtlp != nil {
+		err = errors.Join(err, fmt.Errorf("failed to shutdown otlp metrics: %w", errOtlp))
+	}
+
+	return err
+}
diff --git a/router/pkg/metric/connection_pool_stats.go b/router/pkg/metric/connection_pool_stats.go
new file mode 100644
index 0000000000..998fc5cb51
--- /dev/null
+++ b/router/pkg/metric/connection_pool_stats.go
@@ -0,0 +1,60 @@
+package metric
+
+import (
+	"sync"
+	"sync/atomic"
+)
+
+type ConnectionPoolStats struct {
+	mu           sync.RWMutex
+	connsPerHost map[SubgraphHostKey]*atomic.Int64
+
+	MaxConnsPerSubgraph map[string]int64
+}
+
+type SubgraphHostKey struct {
+	Subgraph string
+	Host     string
+}
+
+func NewConnectionPoolStats() *ConnectionPoolStats {
+	return &ConnectionPoolStats{
+		connsPerHost:        make(map[SubgraphHostKey]*atomic.Int64),
+		MaxConnsPerSubgraph: make(map[string]int64),
+	}
+}
+
+func (t *ConnectionPoolStats) AddSubgraphHostCount(subgraph string, maxHostCount int64) {
+	t.MaxConnsPerSubgraph[subgraph] = maxHostCount
+}
+
+func (t *ConnectionPoolStats) GetCounter(key SubgraphHostKey) *atomic.Int64 {
+	// Try a read lock first
+	t.mu.RLock()
+	counter, ok := t.connsPerHost[key]
+	t.mu.RUnlock()
+	if ok {
+		return counter
+	}
+
+	// Create new counter, if this is the first host call
+	t.mu.Lock()
+	defer t.mu.Unlock()
+	// Double-check because another goroutine may have still created the key
+	if counter, ok = t.connsPerHost[key]; ok {
+		return counter
+	}
+	t.connsPerHost[key] = new(atomic.Int64)
+	return t.connsPerHost[key]
+}
+
+// GetStats Return a snapshot of the map of connection counts
+func (t *ConnectionPoolStats) GetStats() map[SubgraphHostKey]int64 {
+	t.mu.RLock()
+	defer t.mu.RUnlock()
+	snapshot := make(map[SubgraphHostKey]int64, len(t.connsPerHost))
+	for addr, counter := range t.connsPerHost {
+		snapshot[addr] = counter.Load()
+	}
+	return snapshot
+}
diff --git a/router/pkg/metric/meter.go b/router/pkg/metric/meter.go
index 903759c794..452e36c3a8 100644
--- a/router/pkg/metric/meter.go
+++ b/router/pkg/metric/meter.go
@@ -98,6 +98,7 @@ var (
 	defaultCloudTemporalitySelector = func(kind sdkmetric.InstrumentKind) metricdata.Temporality {
 		switch kind {
 		case sdkmetric.InstrumentKindCounter,
+			sdkmetric.InstrumentKindGauge,
 			sdkmetric.InstrumentKindUpDownCounter,
 			sdkmetric.InstrumentKindHistogram:
 			return metricdata.DeltaTemporality
@@ -354,7 +355,7 @@ func defaultPrometheusMetricOptions(ctx context.Context, serviceInstanceID strin
 		if isKeyInSlice(value.Key, defaultExcludedOtelKeys) {
 			return false
 		}
-		name := sanitizeName(string(value.Key))
+		name := SanitizeName(string(value.Key))
 		for _, re := range c.Prometheus.ExcludeMetricLabels {
 			if re.MatchString(name) {
 				return false
@@ -376,7 +377,7 @@ func defaultPrometheusMetricOptions(ctx context.Context, serviceInstanceID strin
 
 		// Filter out metrics that match the excludeMetrics regexes
 		for _, re := range c.Prometheus.ExcludeMetrics {
-			promName := sanitizeName(i.Name)
+			promName := SanitizeName(i.Name)
 			if re.MatchString(promName) {
 				// Drop the metric
 				s.Aggregation = sdkmetric.AggregationDrop{}
diff --git a/router/pkg/metric/noop_connection_metrics.go b/router/pkg/metric/noop_connection_metrics.go
new file mode 100644
index 0000000000..8d9566de28
--- /dev/null
+++ b/router/pkg/metric/noop_connection_metrics.go
@@ -0,0 +1,24 @@
+package metric
+
+import (
+	"context"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// A noop metric provider so we do not need to do nil checks for each provider call from the store
+type noopConnectionMetricProvider struct{}
+
+func (h *noopConnectionMetricProvider) MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption) {
+}
+
+func (h *noopConnectionMetricProvider) MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption) {
+
+}
+
+func (h *noopConnectionMetricProvider) Flush(ctx context.Context) error {
+	return nil
+}
+
+func (h *noopConnectionMetricProvider) Shutdown() error {
+	return nil
+}
diff --git a/router/pkg/metric/oltp_connection_metric_store.go b/router/pkg/metric/oltp_connection_metric_store.go
new file mode 100644
index 0000000000..0a4b59e27b
--- /dev/null
+++ b/router/pkg/metric/oltp_connection_metric_store.go
@@ -0,0 +1,105 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+	"go.opentelemetry.io/otel/attribute"
+
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+const (
+	cosmoRouterConnectionMeterName    = "cosmo.router.connections"
+	cosmoRouterConnectionMeterVersion = "0.0.1"
+)
+
+type otlpConnectionMetrics struct {
+	instruments             *connectionInstruments
+	meterProvider           *metric.MeterProvider
+	logger                  *zap.Logger
+	meter                   otelmetric.Meter
+	instrumentRegistrations []otelmetric.Registration
+}
+
+func newOtlpConnectionMetrics(logger *zap.Logger, meterProvider *metric.MeterProvider, stats *ConnectionPoolStats, baseAttributes []attribute.KeyValue) (*otlpConnectionMetrics, error) {
+	meter := meterProvider.Meter(
+		cosmoRouterConnectionMeterName,
+		otelmetric.WithInstrumentationVersion(cosmoRouterConnectionMeterVersion),
+	)
+
+	instruments, err := newConnectionInstruments(meter)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create otlp connection instruments: %w", err)
+	}
+
+	metrics := &otlpConnectionMetrics{
+		instruments:   instruments,
+		meterProvider: meterProvider,
+		logger:        logger,
+		meter:         meter,
+	}
+
+	metrics.startInitMetrics(stats, baseAttributes)
+	return metrics, nil
+}
+
+func (h *otlpConnectionMetrics) startInitMetrics(connStats *ConnectionPoolStats, attributes []attribute.KeyValue) error {
+	for subgraph, maxConns := range connStats.MaxConnsPerSubgraph {
+		attrs := make([]attribute.KeyValue, 0, 1)
+		if subgraph != "" {
+			attrs = append(attrs, otel.WgSubgraphName.String(subgraph))
+		}
+		h.MeasureMaxConnections(context.Background(), maxConns, otelmetric.WithAttributes(attrs...))
+	}
+
+	rc, err := h.meter.RegisterCallback(func(_ context.Context, o otelmetric.Observer) error {
+		stats := connStats.GetStats()
+		for key, activeConnections := range stats {
+			attrs := otel.GetServerAttributes(key.Host)
+			if key.Subgraph != "" {
+				attrs = append(attrs, otel.WgSubgraphName.String(key.Subgraph))
+			}
+
+			o.ObserveInt64(h.instruments.connectionsActive, activeConnections,
+				otelmetric.WithAttributes(attributes...),
+				otelmetric.WithAttributes(attrs...),
+			)
+		}
+
+		return nil
+	}, h.instruments.connectionsActive)
+	if err != nil {
+		return err
+	}
+
+	h.instrumentRegistrations = append(h.instrumentRegistrations, rc)
+	return nil
+}
+
+func (h *otlpConnectionMetrics) MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption) {
+	h.instruments.connectionAcquireDuration.Record(ctx, duration, opts...)
+}
+
+func (h *otlpConnectionMetrics) MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption) {
+	h.instruments.maxConnections.Record(ctx, count, opts...)
+}
+
+func (h *otlpConnectionMetrics) Flush(ctx context.Context) error {
+	return h.meterProvider.ForceFlush(ctx)
+}
+
+func (h *otlpConnectionMetrics) Shutdown() error {
+	var err error
+
+	for _, reg := range h.instrumentRegistrations {
+		if regErr := reg.Unregister(); regErr != nil {
+			err = errors.Join(regErr)
+		}
+	}
+
+	return err
+}
diff --git a/router/pkg/metric/prom_connection_metric_store.go b/router/pkg/metric/prom_connection_metric_store.go
new file mode 100644
index 0000000000..32ecfc342e
--- /dev/null
+++ b/router/pkg/metric/prom_connection_metric_store.go
@@ -0,0 +1,104 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+	"go.opentelemetry.io/otel/attribute"
+
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+const (
+	cosmoRouterConnectionPrometheusMeterName    = "cosmo.router.connections.prometheus"
+	cosmoRouterConnectionPrometheusMeterVersion = "0.0.1"
+)
+
+type promConnectionMetrics struct {
+	instruments             *connectionInstruments
+	meterProvider           *metric.MeterProvider
+	logger                  *zap.Logger
+	meter                   otelmetric.Meter
+	instrumentRegistrations []otelmetric.Registration
+}
+
+func newPromConnectionMetrics(logger *zap.Logger, meterProvider *metric.MeterProvider, stats *ConnectionPoolStats, attributes []attribute.KeyValue) (*promConnectionMetrics, error) {
+	meter := meterProvider.Meter(
+		cosmoRouterConnectionPrometheusMeterName,
+		otelmetric.WithInstrumentationVersion(cosmoRouterConnectionPrometheusMeterVersion),
+	)
+
+	instruments, err := newConnectionInstruments(meter)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create prometheus connection instruments: %w", err)
+	}
+
+	metrics := &promConnectionMetrics{
+		instruments:   instruments,
+		meterProvider: meterProvider,
+		meter:         meter,
+		logger:        logger,
+	}
+
+	metrics.startInitMetrics(stats, attributes)
+	return metrics, nil
+}
+
+func (h *promConnectionMetrics) startInitMetrics(connStats *ConnectionPoolStats, attributes []attribute.KeyValue) error {
+	for subgraph, maxConns := range connStats.MaxConnsPerSubgraph {
+		attrs := make([]attribute.KeyValue, 0, 1)
+		if subgraph != "" {
+			attrs = append(attrs, otel.WgSubgraphName.String(subgraph))
+		}
+		h.MeasureMaxConnections(context.Background(), maxConns, otelmetric.WithAttributes(attrs...))
+	}
+
+	rc, err := h.meter.RegisterCallback(func(_ context.Context, o otelmetric.Observer) error {
+		stats := connStats.GetStats()
+		for key, activeConnections := range stats {
+			attrs := otel.GetServerAttributes(key.Host)
+			if key.Subgraph != "" {
+				attrs = append(attrs, otel.WgSubgraphName.String(key.Subgraph))
+			}
+			o.ObserveInt64(h.instruments.connectionsActive, activeConnections,
+				otelmetric.WithAttributes(attributes...),
+				otelmetric.WithAttributes(attrs...),
+			)
+		}
+
+		return nil
+	}, h.instruments.connectionsActive)
+	if err != nil {
+		return err
+	}
+
+	h.instrumentRegistrations = append(h.instrumentRegistrations, rc)
+	return nil
+}
+
+func (m *promConnectionMetrics) MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption) {
+	m.instruments.connectionAcquireDuration.Record(ctx, duration, opts...)
+}
+
+func (m *promConnectionMetrics) MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption) {
+	m.instruments.maxConnections.Record(ctx, count, opts...)
+}
+
+func (m *promConnectionMetrics) Flush(ctx context.Context) error {
+	return m.meterProvider.ForceFlush(ctx)
+}
+
+func (h *promConnectionMetrics) Shutdown() error {
+	var err error
+
+	for _, reg := range h.instrumentRegistrations {
+		if regErr := reg.Unregister(); regErr != nil {
+			err = errors.Join(regErr)
+		}
+	}
+
+	return err
+}
diff --git a/router/pkg/metric/prometheus_server.go b/router/pkg/metric/prometheus_server.go
index 6bd8894352..1bebe1d536 100644
--- a/router/pkg/metric/prometheus_server.go
+++ b/router/pkg/metric/prometheus_server.go
@@ -45,7 +45,7 @@ func NewPrometheusServer(logger *zap.Logger, listenAddr string, path string, reg
 	return svr
 }
 
-func sanitizeName(name string) string {
+func SanitizeName(name string) string {
 	return strings.Map(sanitizeRune, name)
 }
 
diff --git a/router/pkg/otel/attributes.go b/router/pkg/otel/attributes.go
index e20a7bfe89..d1ee5db243 100644
--- a/router/pkg/otel/attributes.go
+++ b/router/pkg/otel/attributes.go
@@ -2,6 +2,7 @@ package otel
 
 import (
 	"go.opentelemetry.io/otel/attribute"
+	"net"
 )
 
 const (
@@ -46,10 +47,11 @@ const (
 	WgIsBatchingOperation              = attribute.Key("wg.operation.batching.is_batched")
 	WgBatchingOperationsCount          = attribute.Key("wg.operation.batching.operations_count")
 	WgBatchingOperationIndex           = attribute.Key("wg.operation.batching.operation_index")
-
 	// HTTPRequestUploadFileCount is the number of files uploaded in a request (Not specified in the OpenTelemetry specification)
 	HTTPRequestUploadFileCount = attribute.Key("http.request.upload.file_count")
 
+	WgClientReusedConnection = attribute.Key("wg.http.client.reused_connection")
+
 	// Prometheus Schema Field Usage Attrs
 
 	WgOperationSha256   = attribute.Key("wg.operation.sha256")
@@ -76,3 +78,25 @@ var (
 	RouterServerAttribute    = WgComponentName.String("router-server")
 	EngineTransportAttribute = WgComponentName.String("engine-transport")
 )
+
+const (
+	ServerAddress = attribute.Key("server.address")
+	ServerPort    = attribute.Key("server.port")
+)
+
+func GetServerAttributes(host string) []attribute.KeyValue {
+	parsedHost, parsedPort, err := net.SplitHostPort(host)
+	if err != nil {
+		// If we are unable to parse the host string
+		// (e.g. :- if there was no port attached at all)
+		// we skip the server port and just return the host as is
+		return []attribute.KeyValue{
+			ServerAddress.String(host),
+		}
+	}
+
+	return []attribute.KeyValue{
+		ServerAddress.String(parsedHost),
+		ServerPort.String(parsedPort),
+	}
+}


Reviews:
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T15:10:49Z
user: login: StarpTech
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T16:36:06Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T08:54:04Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T08:59:24Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T09:00:53Z
user: login: StarpTech
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T09:11:26Z
user: login: StarpTech
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T09:12:36Z
user: login: StarpTech
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T11:19:38Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:39:34Z
user: login: StarpTech
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T08:25:52Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T10:42:54Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T10:47:02Z
user: login: SkArchon
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T14:53:54Z
user: login: StarpTech
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T15:11:57Z

Review comments:
diff_hunk: @@ -0,0 +1,53 @@
+package core
+
+import (
+	"context"
+	"github.com/wundergraph/cosmo/router/pkg/metric"
+	"net"
+)
+
+type TraceDialer struct {
+	connectionPoolStats *metric.ConnectionPoolStats
+}
+
+func NewTraceDialer() *TraceDialer {
+	return &TraceDialer{
+		connectionPoolStats: metric.NewConnectionPoolStats(),
+	}
+}
+
+type dialerFunc func(ctx context.Context, network, address string) (net.Conn, error)
+
+func (t *TraceDialer) WrapDial(base dialerFunc) dialerFunc {
+	return func(ctx context.Context, network, address string) (net.Conn, error) {
path: router/core/trace_dialer.go
user: login: SkArchon
body: Used JMeter to load test this and get it over one, as otherwise counter doesnt go over one by simple api calls, counter seems to resets back to 0 eventually from my testing
created_at: 2025-05-15T15:10:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,133 @@
+package metric
+
+import (
+	"fmt"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// Connection metric constants
+const (
+	connectionTotal   = "router.connection.total"
+	connectionsActive = "router.connection.active"
+
+	dnsDuration               = "router.connection.dns_duration"
path: router/pkg/metric/connection_measurements.go
user: login: StarpTech
body: The naming is not clear. We should prefix them with e.g. `router.http.client.connection`. Inn the future, we might also add connection metrics about the server
created_at: 2025-05-16T07:35:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -99,6 +100,8 @@ func (rt *RetryHTTPTransport) RoundTrip(req *http.Request) (*http.Response, erro
 		// drain the previous response before retrying
 		rt.drainBody(resp, requestLogger)
 
+		httpclient.IncrementRetryCount(req.Context())
path: router/internal/retrytransport/retry_transport.go
user: login: StarpTech
body: Is this called anywhere else? A user could have retry disabled.
created_at: 2025-05-18T16:12:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+package httpclient
+
+import (
+	"context"
+	"crypto/tls"
+	"net/http"
+	"net/http/httptrace"
+	"sync"
+	"time"
+)
+
+type ClientTraceContextKey struct{}
+
+type TraceInjectingRoundTripper struct {
+	base http.RoundTripper
+}
+
+func NewTraceInjectingRoundTripper(base http.RoundTripper) *TraceInjectingRoundTripper {
+	return &TraceInjectingRoundTripper{
+		base: base,
+	}
+}
+
+func GetClientTraceFromContext(ctx context.Context) *ClientTraceInfo {
+	value := ctx.Value(ClientTraceContextKey{})
+	// Return no-op context if the subgraph context key was never set
+	if value == nil {
+		return &ClientTraceInfo{}
+	}
+	return value.(*ClientTraceInfo)
+}
+
+func (t *TraceInjectingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	trace := t.getClientTrace(req.Context())
+
+	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+	trip, err := t.base.RoundTrip(req)
+
+	return trip, err
+}
+
+func IncrementRetryCount(ctx context.Context) {
+	fromContext := GetClientTraceFromContext(ctx)
+	fromContext.RetryCountLatestIndex++
+	fromContext.ClientTraces = append(fromContext.ClientTraces, &ClientTrace{})
path: router/internal/httpclient/traceclient.go
user: login: StarpTech
body: Why do we need to collect them?
created_at: 2025-05-18T16:13:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+package httpclient
+
+import (
+	"sort"
+	"time"
+)
+
+type DNSStart struct {
+	Time time.Time
+	Host string
+}
+
+type DNSDone struct {
+	Time      time.Time
+	Coalesced bool
+	Error     error
+}
+
+type TLSStart struct {
+	Time time.Time
+}
+
+type TLSDone struct {
+	Time      time.Time
+	Complete  bool
+	DidResume bool
+	Error     error
+}
+
+type DialStart struct {
+	Time    time.Time
+	Network string
+	Address string
+}
+
+type DialDone struct {
+	Time    time.Time
+	Network string
+	Address string
+	Error   error
+}
+
+type WroteHeaders struct {
+	Time time.Time
+}
+
+type WroteRequest struct {
+	Time  time.Time
+	Error error
+}
+
+type FirstByte struct {
+	Time time.Time
+}
+
+type AcquiredConnection struct {
+	Time     time.Time
+	Reused   bool
+	WasIdle  bool
+	IdleTime time.Duration
+}
+
+type GetConnection struct {
+	Time     time.Time
+	HostPort string
+}
+
+type ClientTrace struct {
+	ConnectionGet      *GetConnection
+	ConnectionAcquired *AcquiredConnection
+	DNSStart           *DNSStart
+	DNSDone            *DNSDone
+	TLSStart           *TLSStart
+	TLSDone            *TLSDone
+	DialStart          []DialStart
+	DialDone           []DialDone
+	WroteHeaders       *WroteHeaders
+	WroteRequest       *WroteRequest
+	FirstByte          *FirstByte
+}
+
+type ClientTraceInfo struct {
+	ClientTraces          []*ClientTrace
+	RetryCountLatestIndex int
+}
+
+type DialCombined struct {
+	DialStartTime time.Time
+	DialDoneTime  *time.Time
+	Error         error
+	Network       string
+	Address       string
+}
+
+// GetGroupedDials returns a slice of DialCombined structs
+// Which are combined based on the network (tcp/udp/...) and address
+// We sort by the first dial that was successful, so that the most likely
+// used dial is first in the list
+func (r ClientTrace) GetGroupedDials() []DialCombined {
path: router/internal/httpclient/types.go
user: login: StarpTech
body: This can be simplified. We can take the first DialStart event and ConnectStart to calculate the duration.

```go
var (
        mu            sync.Mutex
        startAll      time.Time
    )

    trace := &httptrace.ClientTrace{
        ConnectStart: func(_, _ string) {
            mu.Lock()
            if startAll.IsZero() {
                startAll = time.Now()
            }
            mu.Unlock()
        },
        GotConn: func(info httptrace.GotConnInfo) {
              mu.Lock()
              total := time.Since(startAll)
              mu.Unlock()
              fmt.Printf("Full connect (TCP+TLS) took %v\n", total)
        },
    }
```
created_at: 2025-05-18T16:24:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -99,6 +100,8 @@ func (rt *RetryHTTPTransport) RoundTrip(req *http.Request) (*http.Response, erro
 		// drain the previous response before retrying
 		rt.drainBody(resp, requestLogger)
 
+		httpclient.IncrementRetryCount(req.Context())
path: router/internal/retrytransport/retry_transport.go
user: login: SkArchon
body: Its only called if retry is enabled.
created_at: 2025-05-19T08:54:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+package httpclient
+
+import (
+	"sort"
+	"time"
+)
+
+type DNSStart struct {
+	Time time.Time
+	Host string
+}
+
+type DNSDone struct {
+	Time      time.Time
+	Coalesced bool
+	Error     error
+}
+
+type TLSStart struct {
+	Time time.Time
+}
+
+type TLSDone struct {
+	Time      time.Time
+	Complete  bool
+	DidResume bool
+	Error     error
+}
+
+type DialStart struct {
+	Time    time.Time
+	Network string
+	Address string
+}
+
+type DialDone struct {
+	Time    time.Time
+	Network string
+	Address string
+	Error   error
+}
+
+type WroteHeaders struct {
+	Time time.Time
+}
+
+type WroteRequest struct {
+	Time  time.Time
+	Error error
+}
+
+type FirstByte struct {
+	Time time.Time
+}
+
+type AcquiredConnection struct {
+	Time     time.Time
+	Reused   bool
+	WasIdle  bool
+	IdleTime time.Duration
+}
+
+type GetConnection struct {
+	Time     time.Time
+	HostPort string
+}
+
+type ClientTrace struct {
+	ConnectionGet      *GetConnection
+	ConnectionAcquired *AcquiredConnection
+	DNSStart           *DNSStart
+	DNSDone            *DNSDone
+	TLSStart           *TLSStart
+	TLSDone            *TLSDone
+	DialStart          []DialStart
+	DialDone           []DialDone
+	WroteHeaders       *WroteHeaders
+	WroteRequest       *WroteRequest
+	FirstByte          *FirstByte
+}
+
+type ClientTraceInfo struct {
+	ClientTraces          []*ClientTrace
+	RetryCountLatestIndex int
+}
+
+type DialCombined struct {
+	DialStartTime time.Time
+	DialDoneTime  *time.Time
+	Error         error
+	Network       string
+	Address       string
+}
+
+// GetGroupedDials returns a slice of DialCombined structs
+// Which are combined based on the network (tcp/udp/...) and address
+// We sort by the first dial that was successful, so that the most likely
+// used dial is first in the list
+func (r ClientTrace) GetGroupedDials() []DialCombined {
path: router/internal/httpclient/types.go
user: login: SkArchon
body: While we cannot use this, because we need to get the first **successful** "ConnectDone" that finished first. We could still simplify it a bit by only keeping a map for the ConnectStart, and within this function itself. Which I will do.
created_at: 2025-05-19T08:59:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+package httpclient
+
+import (
+	"context"
+	"crypto/tls"
+	"net/http"
+	"net/http/httptrace"
+	"sync"
+	"time"
+)
+
+type ClientTraceContextKey struct{}
+
+type TraceInjectingRoundTripper struct {
+	base http.RoundTripper
+}
+
+func NewTraceInjectingRoundTripper(base http.RoundTripper) *TraceInjectingRoundTripper {
+	return &TraceInjectingRoundTripper{
+		base: base,
+	}
+}
+
+func GetClientTraceFromContext(ctx context.Context) *ClientTraceInfo {
+	value := ctx.Value(ClientTraceContextKey{})
+	// Return no-op context if the subgraph context key was never set
+	if value == nil {
+		return &ClientTraceInfo{}
+	}
+	return value.(*ClientTraceInfo)
+}
+
+func (t *TraceInjectingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	trace := t.getClientTrace(req.Context())
+
+	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+	trip, err := t.base.RoundTrip(req)
+
+	return trip, err
+}
+
+func IncrementRetryCount(ctx context.Context) {
+	fromContext := GetClientTraceFromContext(ctx)
+	fromContext.RetryCountLatestIndex++
+	fromContext.ClientTraces = append(fromContext.ClientTraces, &ClientTrace{})
path: router/internal/httpclient/traceclient.go
user: login: SkArchon
body: We right now record all the information on the retries into the metric store because there could be information (latency, etc) spikes there that might be useful for the user, esp if its a common thing.
created_at: 2025-05-19T09:00:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+package httpclient
+
+import (
+	"sort"
+	"time"
+)
+
+type DNSStart struct {
+	Time time.Time
+	Host string
+}
+
+type DNSDone struct {
+	Time      time.Time
+	Coalesced bool
+	Error     error
+}
+
+type TLSStart struct {
+	Time time.Time
+}
+
+type TLSDone struct {
+	Time      time.Time
+	Complete  bool
+	DidResume bool
+	Error     error
+}
+
+type DialStart struct {
+	Time    time.Time
+	Network string
+	Address string
+}
+
+type DialDone struct {
+	Time    time.Time
+	Network string
+	Address string
+	Error   error
+}
+
+type WroteHeaders struct {
+	Time time.Time
+}
+
+type WroteRequest struct {
+	Time  time.Time
+	Error error
+}
+
+type FirstByte struct {
+	Time time.Time
+}
+
+type AcquiredConnection struct {
+	Time     time.Time
+	Reused   bool
+	WasIdle  bool
+	IdleTime time.Duration
+}
+
+type GetConnection struct {
+	Time     time.Time
+	HostPort string
+}
+
+type ClientTrace struct {
+	ConnectionGet      *GetConnection
+	ConnectionAcquired *AcquiredConnection
+	DNSStart           *DNSStart
+	DNSDone            *DNSDone
+	TLSStart           *TLSStart
+	TLSDone            *TLSDone
+	DialStart          []DialStart
+	DialDone           []DialDone
+	WroteHeaders       *WroteHeaders
+	WroteRequest       *WroteRequest
+	FirstByte          *FirstByte
+}
+
+type ClientTraceInfo struct {
+	ClientTraces          []*ClientTrace
+	RetryCountLatestIndex int
+}
+
+type DialCombined struct {
+	DialStartTime time.Time
+	DialDoneTime  *time.Time
+	Error         error
+	Network       string
+	Address       string
+}
+
+// GetGroupedDials returns a slice of DialCombined structs
+// Which are combined based on the network (tcp/udp/...) and address
+// We sort by the first dial that was successful, so that the most likely
+// used dial is first in the list
+func (r ClientTrace) GetGroupedDials() []DialCombined {
path: router/internal/httpclient/types.go
user: login: StarpTech
body: Also when `GotConn` is called after we successfully connected? Remember, we are only interested in the dial duration.
created_at: 2025-05-19T09:11:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+package httpclient
+
+import (
+	"context"
+	"crypto/tls"
+	"net/http"
+	"net/http/httptrace"
+	"sync"
+	"time"
+)
+
+type ClientTraceContextKey struct{}
+
+type TraceInjectingRoundTripper struct {
+	base http.RoundTripper
+}
+
+func NewTraceInjectingRoundTripper(base http.RoundTripper) *TraceInjectingRoundTripper {
+	return &TraceInjectingRoundTripper{
+		base: base,
+	}
+}
+
+func GetClientTraceFromContext(ctx context.Context) *ClientTraceInfo {
+	value := ctx.Value(ClientTraceContextKey{})
+	// Return no-op context if the subgraph context key was never set
+	if value == nil {
+		return &ClientTraceInfo{}
+	}
+	return value.(*ClientTraceInfo)
+}
+
+func (t *TraceInjectingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	trace := t.getClientTrace(req.Context())
+
+	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+	trip, err := t.base.RoundTrip(req)
+
+	return trip, err
+}
+
+func IncrementRetryCount(ctx context.Context) {
+	fromContext := GetClientTraceFromContext(ctx)
+	fromContext.RetryCountLatestIndex++
+	fromContext.ClientTraces = append(fromContext.ClientTraces, &ClientTrace{})
path: router/internal/httpclient/traceclient.go
user: login: StarpTech
body: Why do we have special handling for retries? We are really only interested in durations. A retry would emit another data point. We don't need to distinguish them yet.
created_at: 2025-05-19T09:12:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -46,10 +46,12 @@ const (
 	WgIsBatchingOperation              = attribute.Key("wg.operation.batching.is_batched")
 	WgBatchingOperationsCount          = attribute.Key("wg.operation.batching.operations_count")
 	WgBatchingOperationIndex           = attribute.Key("wg.operation.batching.operation_index")
-
 	// HTTPRequestUploadFileCount is the number of files uploaded in a request (Not specified in the OpenTelemetry specification)
 	HTTPRequestUploadFileCount = attribute.Key("http.request.upload.file_count")
 
+	WgHost       = attribute.Key("wg.conn.host")
path: router/pkg/otel/attributes.go
user: login: StarpTech
body: Should exist in semconv
created_at: 2025-05-21T11:07:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,106 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+	"go.opentelemetry.io/otel/attribute"
+
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+const (
+	cosmoRouterConnectionMeterName    = "cosmo.router.connection"
path: router/pkg/metric/oltp_connection_metric_store.go
user: login: StarpTech
body: `cosmo.router.connections`
created_at: 2025-05-21T11:09:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,66 @@
+package metric
+
+import (
+	"fmt"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// Connection metric constants
+const (
+	maxConnections            = "router.http.client.connection.max"
path: router/pkg/metric/connection_measurements.go
user: login: StarpTech
body: `router.http.client.max_connections`
`router.http.client.active_connections`
`router.http.client.connections.aquire_duration`
created_at: 2025-05-21T11:16:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4227,6 +4229,19 @@ func TestPrometheus(t *testing.T) {
 
 }
 
+func getHost(connectionTotal *io_prometheus_client.Metric) string {
+	host := ""
+	for _, label := range connectionTotal.Label {
+		if label.Name == nil || label.Value == nil {
+			continue
+		}
+		if *label.Name == "wg_conn_host" {
path: router-tests/prometheus_test.go
user: login: StarpTech
body: Reference to a const
created_at: 2025-05-21T11:18:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -78,3 +78,25 @@ var (
 	RouterServerAttribute    = WgComponentName.String("router-server")
 	EngineTransportAttribute = WgComponentName.String("engine-transport")
 )
+
+const (
+	ServerAddress = attribute.Key("server.address")
+	ServerPort    = attribute.Key("server.port")
+)
+
+func GetServerAttributes(host string) []attribute.KeyValue {
path: router/pkg/otel/attributes.go
user: login: SkArchon
body: According to semconv, there were a bunch of similar attributes that we could have used but all of them seem to have been deprecated in favour of `server.address` and `server.port`, thus we have to split the port and host component.

In case a port is not provided / bad host, we stick to returning only the host component.

cc: @StarpTech 
created_at: 2025-05-21T14:39:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4229,17 +4230,22 @@ func TestPrometheus(t *testing.T) {
 
 }
 
-func getHost(connectionTotal *io_prometheus_client.Metric) string {
-	host := ""
+func getPort(connectionTotal *io_prometheus_client.Metric) string {
+	// We can get the key from the otel attribute, but we need to make an initialization
+	// also convert the key to prometheus which uses _ instead of . like otel
+	serverPortKey := strings.ReplaceAll(string(otel.ServerPort.String("").Key), ".", "_")
path: router-tests/prometheus_test.go
user: login: StarpTech
body: This needs to follow specific rules. We have a `sanitizeName` in the metric package. We could make it public.
created_at: 2025-05-22T07:45:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -78,3 +78,25 @@ var (
 	RouterServerAttribute    = WgComponentName.String("router-server")
 	EngineTransportAttribute = WgComponentName.String("engine-transport")
 )
+
+const (
+	ServerAddress = attribute.Key("server.address")
+	ServerPort    = attribute.Key("server.port")
+)
+
+func GetServerAttributes(host string) []attribute.KeyValue {
path: router/pkg/otel/attributes.go
user: login: StarpTech
body: Looks good.
created_at: 2025-05-22T07:47:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+
+	"go.opentelemetry.io/otel/attribute"
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+// ConnectionMetricProvider is the interface that wraps the basic connection metric methods.
+// We maintain two providers, one for OTEL and one for Prometheus.
+type ConnectionMetricProvider interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption)
+	MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption)
+	Flush(ctx context.Context) error
+	Shutdown() error
+}
+
+// ConnectionMetricStore is the interface for connection and pool metrics only.
+type ConnectionMetricStore interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, attrs ...attribute.KeyValue)
+	Flush(ctx context.Context) error
+	Shutdown(ctx context.Context) error
+}
+
+type ConnectionMetrics struct {
+	baseAttributes []attribute.KeyValue
+	logger         *zap.Logger
+
+	otlpConnectionMetrics ConnectionMetricProvider
+	promConnectionMetrics ConnectionMetricProvider
+}
+
+func NewConnectionMetricStore(
+	logger *zap.Logger,
+	baseAttributes []attribute.KeyValue,
+	otelProvider, promProvider *metric.MeterProvider,
+	metricsConfig *Config,
+	connectionPoolStats *ConnectionPoolStats,
+) (*ConnectionMetrics, error) {
+	connMetrics := &ConnectionMetrics{
+		baseAttributes: baseAttributes,
+		logger:         logger,
+	}
+
+	if metricsConfig.OpenTelemetry.ConnectionStats {
path: router/pkg/metric/connection_metric_store.go
user: login: StarpTech
body: In the current store implementation, we work with `sdkmetric.NewMeterProvider()` as default meter provider which are basically noop. Inn this implementation, we need to check in every method if the provider is not nil. Can we align this? 
created_at: 2025-05-22T07:58:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -46,10 +47,11 @@ const (
 	WgIsBatchingOperation              = attribute.Key("wg.operation.batching.is_batched")
 	WgBatchingOperationsCount          = attribute.Key("wg.operation.batching.operations_count")
 	WgBatchingOperationIndex           = attribute.Key("wg.operation.batching.operation_index")
-
 	// HTTPRequestUploadFileCount is the number of files uploaded in a request (Not specified in the OpenTelemetry specification)
 	HTTPRequestUploadFileCount = attribute.Key("http.request.upload.file_count")
 
+	WgConnReused = attribute.Key("wg.conn.reused")
path: router/pkg/otel/attributes.go
user: login: StarpTech
body: We should avoid shortcuts in attribute names. Proposal `wg.http.client.reused_connection`

Similar metric: https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientopen_connections
created_at: 2025-05-22T08:24:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,66 @@
+package metric
+
+import (
+	"fmt"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// Connection metric constants
+const (
+	maxConnections            = "router.http.client.max_connections"
+	connectionsActive         = "router.http.client.active_connections"
+	connectionAcquireDuration = "router.http.client.connections.acquire_duration"
path: router/pkg/metric/connection_measurements.go
user: login: StarpTech
body: To follow the naming schema we could also use `router.http.client.acquire_connection_duration`

https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpclientopen_connections
created_at: 2025-05-22T08:25:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4229,17 +4230,22 @@ func TestPrometheus(t *testing.T) {
 
 }
 
-func getHost(connectionTotal *io_prometheus_client.Metric) string {
-	host := ""
+func getPort(connectionTotal *io_prometheus_client.Metric) string {
+	// We can get the key from the otel attribute, but we need to make an initialization
+	// also convert the key to prometheus which uses _ instead of . like otel
+	serverPortKey := strings.ReplaceAll(string(otel.ServerPort.String("").Key), ".", "_")
path: router-tests/prometheus_test.go
user: login: SkArchon
body: I based it on another place which `repalced` the string in prometheus test. Will update both
created_at: 2025-05-22T10:42:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,66 @@
+package metric
+
+import (
+	"fmt"
+	otelmetric "go.opentelemetry.io/otel/metric"
+)
+
+// Connection metric constants
+const (
+	maxConnections            = "router.http.client.max_connections"
+	connectionsActive         = "router.http.client.active_connections"
+	connectionAcquireDuration = "router.http.client.connections.acquire_duration"
path: router/pkg/metric/connection_measurements.go
user: login: SkArchon
body: I thought we went aligned on prefixing it with `connections` because for future metrics like `router.http.client.connections.dns_duration` etc everything will be prefixed by connections are you suggesting that for future metrics we also don't prefix, i.e. :- router.http.client.dns_connection_duration ?
created_at: 2025-05-22T10:47:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+package metric
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/wundergraph/cosmo/router/pkg/otel"
+
+	"go.opentelemetry.io/otel/attribute"
+	otelmetric "go.opentelemetry.io/otel/metric"
+	"go.opentelemetry.io/otel/sdk/metric"
+	"go.uber.org/zap"
+)
+
+// ConnectionMetricProvider is the interface that wraps the basic connection metric methods.
+// We maintain two providers, one for OTEL and one for Prometheus.
+type ConnectionMetricProvider interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, opts ...otelmetric.RecordOption)
+	MeasureMaxConnections(ctx context.Context, count int64, opts ...otelmetric.RecordOption)
+	Flush(ctx context.Context) error
+	Shutdown() error
+}
+
+// ConnectionMetricStore is the interface for connection and pool metrics only.
+type ConnectionMetricStore interface {
+	MeasureConnectionAcquireDuration(ctx context.Context, duration float64, attrs ...attribute.KeyValue)
+	Flush(ctx context.Context) error
+	Shutdown(ctx context.Context) error
+}
+
+type ConnectionMetrics struct {
+	baseAttributes []attribute.KeyValue
+	logger         *zap.Logger
+
+	otlpConnectionMetrics ConnectionMetricProvider
+	promConnectionMetrics ConnectionMetricProvider
+}
+
+func NewConnectionMetricStore(
+	logger *zap.Logger,
+	baseAttributes []attribute.KeyValue,
+	otelProvider, promProvider *metric.MeterProvider,
+	metricsConfig *Config,
+	connectionPoolStats *ConnectionPoolStats,
+) (*ConnectionMetrics, error) {
+	connMetrics := &ConnectionMetrics{
+		baseAttributes: baseAttributes,
+		logger:         logger,
+	}
+
+	if metricsConfig.OpenTelemetry.ConnectionStats {
path: router/pkg/metric/connection_metric_store.go
user: login: SkArchon
body: Also for the sdkmetric.NewMeterProvider() this is a meter-provider no-op, not a metric-provider no-op, you mean to have a no-op interface for the meter providers right, so we can remove the nil checks?
created_at: 2025-05-22T14:53:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-14T20:13:37Z
body: <!-- add-pr-comment:add-pr-comment -->

# Router image scan passed
:white_check_mark: No security [vulnerabilities](https://github.com/wundergraph/cosmo/security/code-scanning?query=pr%3A1862+is%3Aopen+tool%3ATrivy) found in image:

```
ghcr.io/wundergraph/cosmo/router:sha-1290fc6ef2d8531db8960c3542431e0ca842411d
```
author_association: NONE
user: login: SkArchon
created_at: 2025-05-15T21:27:16Z
body: TODO: Didn't push the prom tests, need to fix the prometheus tests asserts, as they are failing when the values seem to be equal. And need to add shutdown hook.
author_association: CONTRIBUTOR