Repository: apollographql/router
Pull request #6967: Add apollo.router.pipelines metrics
Author: login: BrynCooke
Labels: name: backport-1.x
description: Backport this PR to 1.x
Created: 2025-03-10T12:16:00Z | Merged: 2025-03-11T15:43:49Z

To enable debugging when request pipelines are being kept around a new gauge metric has been added:

- `apollo.router.pipelines` - The number of request pipelines active in the router
    - `schema.id` - The Apollo Studio schema hash associated with the pipeline.
    - `launch.id` - The Apollo Studio launch id associated with the pipeline (optional).
    - `config.hash` - The hash of the configuration

<!-- start metadata -->
<!-- ROUTER-1129 -->
---

**Checklist**

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

- [x] Changes are compatible[^1]
- [x] Documentation[^2] completed
- [ ] Performance impact assessed and acceptable
- Tests added and passing[^3]
    - [ ] Unit Tests
    - [x] Integration Tests
    - [x] Manual Tests

**Exceptions**

*Note any exceptions here*

**Notes**

[^1]: It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this.
[^2]: Configuration is an important part of many changes. Where applicable please try to document configuration examples.
[^3]: Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions.



Diff:
diff --git a/.changesets/feat_pipeline_metrics.md b/.changesets/feat_pipeline_metrics.md
new file mode 100644
index 0000000000..3e35ecbe49
--- /dev/null
+++ b/.changesets/feat_pipeline_metrics.md
@@ -0,0 +1,14 @@
+### Add apollo.router.pipelines metrics ([PR #6967](https://github.com/apollographql/router/pull/6967))
+
+When the Router reloads either via schema change or config change a new request pipeline is created.
+Existing request pipelines are closed once their requests finish. However, this may not happen if there are ongoing long requests 
+such as Subscriptions that do not finish.
+
+To enable debugging when request pipelines are being kept around a new gauge metric has been added:
+
+- `apollo.router.pipelines` - The number of request pipelines active in the router
+    - `schema.id` - The Apollo Studio schema hash associated with the pipeline.
+    - `launch.id` - The Apollo Studio launch id associated with the pipeline (optional).
+    - `config.hash` - The hash of the configuration
+
+By [@BrynCooke](https://github.com/BrynCooke) in https://github.com/apollographql/router/pull/6967
diff --git a/apollo-router/src/configuration/mod.rs b/apollo-router/src/configuration/mod.rs
index f617fbc4f2..000fe02deb 100644
--- a/apollo-router/src/configuration/mod.rs
+++ b/apollo-router/src/configuration/mod.rs
@@ -1,5 +1,6 @@
 //! Logic for loading configuration in to an object model
 use std::fmt;
+use std::hash::Hash;
 use std::io;
 use std::io::BufReader;
 use std::iter;
@@ -33,6 +34,7 @@ use serde::Deserializer;
 use serde::Serialize;
 use serde_json::Map;
 use serde_json::Value;
+use sha2::Digest;
 use thiserror::Error;
 
 use self::cors::Cors;
@@ -134,7 +136,7 @@ impl From<proteus::parser::Error> for ConfigurationError {
 #[derivative(Debug)]
 // We can't put a global #[serde(default)] here because of the Default implementation using `from_str` which use deserialize
 pub struct Configuration {
-    /// The raw configuration string.
+    /// The raw configuration value.
     #[serde(skip)]
     pub(crate) validated_yaml: Option<Value>,
 
@@ -337,6 +339,18 @@ impl Configuration {
 }
 
 impl Configuration {
+    pub(crate) fn hash(&self) -> String {
+        let mut hasher = sha2::Sha256::new();
+        let defaulted_raw = self
+            .validated_yaml
+            .as_ref()
+            .map(|s| serde_yaml::to_string(s).expect("config was not serializable"))
+            .unwrap_or_default();
+        hasher.update(defaulted_raw);
+        let hash: String = format!("{:x}", hasher.finalize());
+        hash
+    }
+
     fn notify(
         apollo_plugins: &Map<String, Value>,
     ) -> Result<Notify<String, graphql::Response>, ConfigurationError> {
diff --git a/apollo-router/src/plugins/telemetry/config_new/instruments.rs b/apollo-router/src/plugins/telemetry/config_new/instruments.rs
index 44512d7c4c..9fe2832baa 100644
--- a/apollo-router/src/plugins/telemetry/config_new/instruments.rs
+++ b/apollo-router/src/plugins/telemetry/config_new/instruments.rs
@@ -7,6 +7,7 @@ use opentelemetry::KeyValue;
 use opentelemetry::metrics::Counter;
 use opentelemetry::metrics::Histogram;
 use opentelemetry::metrics::MeterProvider;
+use opentelemetry::metrics::ObservableGauge;
 use opentelemetry::metrics::UpDownCounter;
 use opentelemetry_semantic_conventions::trace::HTTP_REQUEST_METHOD;
 use opentelemetry_semantic_conventions::trace::SERVER_ADDRESS;
@@ -33,6 +34,7 @@ use super::graphql::selectors::ListLength;
 use super::selectors::CacheKind;
 use crate::Context;
 use crate::metrics;
+use crate::metrics::meter_provider;
 use crate::plugins::telemetry::config_new::Selectors;
 use crate::plugins::telemetry::config_new::attributes::DefaultAttributeRequirementLevel;
 use crate::plugins::telemetry::config_new::attributes::RouterAttributes;
@@ -59,6 +61,8 @@ use crate::plugins::telemetry::config_new::selectors::SupergraphSelector;
 use crate::plugins::telemetry::config_new::selectors::SupergraphValue;
 use crate::plugins::telemetry::otlp::TelemetryDataKind;
 use crate::services::router;
+use crate::services::router::pipeline_handle::PIPELINE_METRIC;
+use crate::services::router::pipeline_handle::pipelines;
 use crate::services::subgraph;
 use crate::services::supergraph;
 
@@ -335,8 +339,8 @@ impl InstrumentsConfig {
                                     )
                                     .as_histogram()
                                     .cloned().expect(
-                                "cannot convert instrument to histogram for router; this should not happen",
-                            )
+                                    "cannot convert instrument to histogram for router; this should not happen",
+                                )
                             ),
                             attributes: Vec::with_capacity(nb_attributes),
                             selector: Some(Arc::new(RouterSelector::RequestHeader {
@@ -379,7 +383,7 @@ impl InstrumentsConfig {
                                     .as_histogram()
                                     .cloned()
                                     .expect(
-                                    "cannot convert instrument to histogram for router; this should not happen",
+                                        "cannot convert instrument to histogram for router; this should not happen",
                                     )
                             ),
                             attributes: Vec::with_capacity(nb_attributes),
@@ -830,16 +834,16 @@ impl InstrumentsConfig {
                         increment: Increment::FieldCustom(None),
                         condition: Condition::True,
                         histogram: Some(static_instruments
-                                .get(FIELD_LENGTH)
-                                .expect(
-                                    "cannot get static instrument for graphql; this should not happen",
-                                )
-                                .as_histogram()
-                                .cloned()
-                                .expect(
-                                    "cannot convert instrument to counter for graphql; this should not happen",
-                                )
-                            ),
+                            .get(FIELD_LENGTH)
+                            .expect(
+                                "cannot get static instrument for graphql; this should not happen",
+                            )
+                            .as_histogram()
+                            .cloned()
+                            .expect(
+                                "cannot convert instrument to counter for graphql; this should not happen",
+                            )
+                        ),
                         attributes: Vec::with_capacity(nb_attributes),
                         selector: Some(Arc::new(GraphQLSelector::ListLength {
                             list_length: ListLength::Value,
@@ -932,16 +936,16 @@ impl InstrumentsConfig {
                         increment: Increment::Custom(None),
                         condition: Condition::True,
                         counter: Some(static_instruments
-                                .get(CACHE_METRIC)
-                                .expect(
-                                    "cannot get static instrument for cache; this should not happen",
-                                )
-                                .as_counter_f64()
-                                .cloned()
-                                .expect(
-                                    "cannot convert instrument to counter for cache; this should not happen",
-                                )
-                            ),
+                            .get(CACHE_METRIC)
+                            .expect(
+                                "cannot get static instrument for cache; this should not happen",
+                            )
+                            .as_counter_f64()
+                            .cloned()
+                            .expect(
+                                "cannot convert instrument to counter for cache; this should not happen",
+                            )
+                        ),
                         attributes: Vec::with_capacity(nb_attributes),
                         selector: Some(Arc::new(SubgraphSelector::Cache {
                             cache: CacheKind::Hit,
@@ -955,6 +959,34 @@ impl InstrumentsConfig {
             }),
         }
     }
+
+    pub(crate) fn new_pipeline_instruments(&self) -> HashMap<String, StaticInstrument> {
+        let meter = meter_provider().meter("apollo/router");
+        let mut instruments = HashMap::new();
+        instruments.insert(
+            PIPELINE_METRIC.to_string(),
+            StaticInstrument::GaugeU64(
+                meter
+                    .u64_observable_gauge(PIPELINE_METRIC)
+                    .with_description("The number of request pipelines active in the router")
+                    .with_callback(|i| {
+                        for (pipeline, count) in &*pipelines() {
+                            let mut attributes = Vec::with_capacity(3);
+                            attributes.push(KeyValue::new("schema.id", pipeline.schema_id.clone()));
+                            if let Some(launch_id) = &pipeline.launch_id {
+                                attributes.push(KeyValue::new("launch.id", launch_id.clone()));
+                            }
+                            attributes
+                                .push(KeyValue::new("config.hash", pipeline.config_hash.clone()));
+
+                            i.observe(*count, &attributes);
+                        }
+                    })
+                    .init(),
+            ),
+        );
+        instruments
+    }
 }
 
 #[derive(Debug)]
@@ -962,6 +994,9 @@ pub(crate) enum StaticInstrument {
     CounterF64(Counter<f64>),
     UpDownCounterI64(UpDownCounter<i64>),
     Histogram(Histogram<f64>),
+    // Gauges are never read
+    #[allow(dead_code)]
+    GaugeU64(ObservableGauge<u64>),
 }
 
 impl StaticInstrument {
diff --git a/apollo-router/src/plugins/telemetry/mod.rs b/apollo-router/src/plugins/telemetry/mod.rs
index 97ddd1b6c4..9605d298dc 100644
--- a/apollo-router/src/plugins/telemetry/mod.rs
+++ b/apollo-router/src/plugins/telemetry/mod.rs
@@ -201,12 +201,7 @@ pub(crate) struct Telemetry {
     custom_endpoints: MultiMap<ListenAddr, Endpoint>,
     apollo_metrics_sender: apollo_exporter::Sender,
     field_level_instrumentation_ratio: f64,
-    pub(crate) graphql_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
-    router_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
-    supergraph_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
-    subgraph_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
-    connector_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
-    cache_custom_instruments: RwLock<Arc<HashMap<String, StaticInstrument>>>,
+    builtin_instruments: RwLock<BuiltinInstruments>,
     activation: Mutex<TelemetryActivation>,
 }
 
@@ -267,6 +262,7 @@ struct BuiltinInstruments {
     subgraph_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     connector_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     cache_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
+    _pipeline_instruments: Arc<HashMap<String, StaticInstrument>>,
 }
 
 fn create_builtin_instruments(config: &InstrumentsConfig) -> BuiltinInstruments {
@@ -277,6 +273,7 @@ fn create_builtin_instruments(config: &InstrumentsConfig) -> BuiltinInstruments
         subgraph_custom_instruments: Arc::new(config.new_builtin_subgraph_instruments()),
         connector_custom_instruments: Arc::new(config.new_builtin_connector_instruments()),
         cache_custom_instruments: Arc::new(config.new_builtin_cache_instruments()),
+        _pipeline_instruments: Arc::new(config.new_pipeline_instruments()),
     }
 }
 
@@ -308,15 +305,6 @@ impl PluginPrivate for Telemetry {
             );
         }
 
-        let BuiltinInstruments {
-            graphql_custom_instruments,
-            router_custom_instruments,
-            supergraph_custom_instruments,
-            subgraph_custom_instruments,
-            connector_custom_instruments,
-            cache_custom_instruments,
-        } = create_builtin_instruments(&config.instrumentation.instruments);
-
         Ok(Telemetry {
             custom_endpoints: metrics_builder.custom_endpoints,
             apollo_metrics_sender: metrics_builder.apollo_metrics_sender,
@@ -335,12 +323,9 @@ impl PluginPrivate for Telemetry {
                     .map(FilterMeterProvider::public),
                 is_active: false,
             }),
-            graphql_custom_instruments: RwLock::new(graphql_custom_instruments),
-            router_custom_instruments: RwLock::new(router_custom_instruments),
-            supergraph_custom_instruments: RwLock::new(supergraph_custom_instruments),
-            subgraph_custom_instruments: RwLock::new(subgraph_custom_instruments),
-            connector_custom_instruments: RwLock::new(connector_custom_instruments),
-            cache_custom_instruments: RwLock::new(cache_custom_instruments),
+            builtin_instruments: RwLock::new(create_builtin_instruments(
+                &config.instrumentation.instruments,
+            )),
             config: Arc::new(config),
         })
     }
@@ -355,7 +340,11 @@ impl PluginPrivate for Telemetry {
             matches!(config.instrumentation.spans.mode, SpanMode::Deprecated);
         let field_level_instrumentation_ratio = self.field_level_instrumentation_ratio;
         let metrics_sender = self.apollo_metrics_sender.clone();
-        let static_router_instruments = self.router_custom_instruments.read().clone();
+        let static_router_instruments = self
+            .builtin_instruments
+            .read()
+            .router_custom_instruments
+            .clone();
 
         ServiceBuilder::new()
             .map_response(move |response: router::Response| {
@@ -583,8 +572,16 @@ impl PluginPrivate for Telemetry {
         let config_map_res_first = config.clone();
         let config_map_res = config.clone();
         let field_level_instrumentation_ratio = self.field_level_instrumentation_ratio;
-        let static_supergraph_instruments = self.supergraph_custom_instruments.read().clone();
-        let static_graphql_instruments = self.graphql_custom_instruments.read().clone();
+        let static_supergraph_instruments = self
+            .builtin_instruments
+            .read()
+            .supergraph_custom_instruments
+            .clone();
+        let static_graphql_instruments = self
+            .builtin_instruments
+            .read()
+            .graphql_custom_instruments
+            .clone();
         ServiceBuilder::new()
             .instrument(move |supergraph_req: &SupergraphRequest| {
                 span_mode.create_supergraph(
@@ -778,8 +775,16 @@ impl PluginPrivate for Telemetry {
         let conf = self.config.clone();
         let subgraph_name = ByteString::from(name);
         let name = name.to_owned();
-        let static_subgraph_instruments = self.subgraph_custom_instruments.read().clone();
-        let static_cache_instruments = self.cache_custom_instruments.read().clone();
+        let static_subgraph_instruments = self
+            .builtin_instruments
+            .read()
+            .subgraph_custom_instruments
+            .clone();
+        let static_cache_instruments = self
+            .builtin_instruments
+            .read()
+            .cache_custom_instruments
+            .clone();
         ServiceBuilder::new()
             .instrument(move |req: &SubgraphRequest| span_mode.create_subgraph(name.as_str(), req))
             .map_request(move |req: SubgraphRequest| request_ftv1(req))
@@ -884,7 +889,11 @@ impl PluginPrivate for Telemetry {
         let req_fn_config = self.config.clone();
         let res_fn_config = self.config.clone();
         let span_mode = self.config.instrumentation.spans.mode;
-        let static_connector_instruments = self.connector_custom_instruments.read().clone();
+        let static_connector_instruments = self
+            .builtin_instruments
+            .read()
+            .connector_custom_instruments
+            .clone();
         ServiceBuilder::new()
             .instrument(move |_req: &connector::request_service::Request| {
                 span_mode.create_connector(source_name.as_str())
@@ -1000,22 +1009,8 @@ impl PluginPrivate for Telemetry {
 
         activation.reload_metrics();
 
-        let BuiltinInstruments {
-            graphql_custom_instruments,
-            router_custom_instruments,
-            supergraph_custom_instruments,
-            subgraph_custom_instruments,
-            connector_custom_instruments,
-            cache_custom_instruments,
-        } = create_builtin_instruments(&self.config.instrumentation.instruments);
-
-        *self.graphql_custom_instruments.write() = graphql_custom_instruments;
-        *self.router_custom_instruments.write() = router_custom_instruments;
-        *self.supergraph_custom_instruments.write() = supergraph_custom_instruments;
-        *self.subgraph_custom_instruments.write() = subgraph_custom_instruments;
-        *self.connector_custom_instruments.write() = connector_custom_instruments;
-        *self.cache_custom_instruments.write() = cache_custom_instruments;
-
+        *self.builtin_instruments.write() =
+            create_builtin_instruments(&self.config.instrumentation.instruments);
         reload_fmt(create_fmt_layer(&self.config));
         activation.is_active = true;
     }
diff --git a/apollo-router/src/services/router.rs b/apollo-router/src/services/router.rs
index a276085b38..e4dd46cd48 100644
--- a/apollo-router/src/services/router.rs
+++ b/apollo-router/src/services/router.rs
@@ -40,6 +40,7 @@ pub type Body = RouterBody;
 pub type Error = hyper::Error;
 
 pub mod body;
+pub(crate) mod pipeline_handle;
 pub(crate) mod service;
 #[cfg(test)]
 mod tests;
diff --git a/apollo-router/src/services/router/pipeline_handle.rs b/apollo-router/src/services/router/pipeline_handle.rs
new file mode 100644
index 0000000000..0b3f897321
--- /dev/null
+++ b/apollo-router/src/services/router/pipeline_handle.rs
@@ -0,0 +1,58 @@
+use std::collections::HashMap;
+use std::sync::OnceLock;
+
+use parking_lot::Mutex;
+use parking_lot::MutexGuard;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES.get_or_init(Default::default).lock()
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id,
+            launch_id,
+            config_hash,
+        };
+        pipelines()
+            .entry(pipeline_ref.clone())
+            .and_modify(|p| *p += 1)
+            .or_insert(1);
+        PipelineHandle { pipeline_ref }
+    }
+}
+
+impl Drop for PipelineHandle {
+    fn drop(&mut self) {
+        let mut pipelines = pipelines();
+        let value = pipelines
+            .get_mut(&self.pipeline_ref)
+            .expect("pipeline_ref MUST be greater than zero");
+        *value -= 1;
+        if *value == 0 {
+            pipelines.remove(&self.pipeline_ref);
+        }
+    }
+}
+
+pub(crate) const PIPELINE_METRIC: &str = "apollo.router.pipelines";
diff --git a/apollo-router/src/services/router/service.rs b/apollo-router/src/services/router/service.rs
index e6436ba2e6..2204b78d7d 100644
--- a/apollo-router/src/services/router/service.rs
+++ b/apollo-router/src/services/router/service.rs
@@ -76,7 +76,6 @@ use crate::query_planner::InMemoryCachePlanner;
 use crate::router_factory::RouterFactory;
 use crate::services::APPLICATION_JSON_HEADER_VALUE;
 use crate::services::HasPlugins;
-#[cfg(test)]
 use crate::services::HasSchema;
 use crate::services::MULTIPART_DEFER_ACCEPT;
 use crate::services::MULTIPART_DEFER_CONTENT_TYPE;
@@ -95,6 +94,7 @@ use crate::services::layers::query_analysis::QueryAnalysisLayer;
 use crate::services::layers::static_page::StaticPageLayer;
 use crate::services::new_service::ServiceFactory;
 use crate::services::router;
+use crate::services::router::pipeline_handle::PipelineHandle;
 use crate::services::supergraph;
 use crate::spec::query::EXTENSIONS_VALUE_COMPLETION_KEY;
 
@@ -979,6 +979,7 @@ pub(crate) fn process_vary_header(headers: &mut HeaderMap<HeaderValue>) {
 pub(crate) struct RouterCreator {
     pub(crate) supergraph_creator: Arc<SupergraphCreator>,
     sb: Buffer<router::Request, BoxFuture<'static, router::ServiceResult>>,
+    _pipeline_handle: Arc<PipelineHandle>,
 }
 
 impl ServiceFactory<router::Request> for RouterCreator {
@@ -1030,6 +1031,17 @@ impl RouterCreator {
         // For now just call activate to make the gauges work on the happy path.
         apq_layer.activate();
 
+        // Create a handle that will help us keep track of this pipeline.
+        // A metric is exposed that allows the use to see if pipelines are being hung onto.
+        let schema_id = supergraph_creator.schema().schema_id.to_string();
+        let launch_id = supergraph_creator
+            .schema()
+            .launch_id
+            .as_ref()
+            .map(|launch_id| launch_id.to_string());
+        let config_hash = configuration.hash();
+        let pipeline_handle = PipelineHandle::new(schema_id, launch_id, config_hash);
+
         let oltp_error_metrics_mode: OtlpErrorMetricsMode =
             match configuration.apollo_plugins.plugins.get("telemetry") {
                 Some(telemetry_config) => {
@@ -1068,6 +1080,7 @@ impl RouterCreator {
         Ok(Self {
             supergraph_creator,
             sb,
+            _pipeline_handle: Arc::new(pipeline_handle),
         })
     }
 
diff --git a/apollo-router/tests/common.rs b/apollo-router/tests/common.rs
index d5e998d963..75800658d7 100644
--- a/apollo-router/tests/common.rs
+++ b/apollo-router/tests/common.rs
@@ -882,7 +882,18 @@ impl IntegrationTest {
     }
 
     #[allow(dead_code)]
-    pub async fn assert_log_contained(&self, msg: &str) {
+    pub fn print_logs(&mut self) {
+        while let Ok(line) = self.stdio_rx.try_recv() {
+            self.logs.push(line.to_string());
+        }
+
+        for line in &self.logs {
+            println!("{}", line);
+        }
+    }
+
+    #[allow(dead_code)]
+    pub fn assert_log_contained(&self, msg: &str) {
         for line in &self.logs {
             if line.contains(msg) {
                 return;
@@ -925,9 +936,18 @@ impl IntegrationTest {
     }
 
     #[allow(dead_code)]
+    /// Checks the metrics contain the supplied string in prometheus format.
+    /// To allow checking of metrics where the value is not stable the magic tag `<any>` can be used.
+    /// For example:
+    /// ```rust,ignore
+    /// router.assert_metrics_contains(r#"apollo_router_pipelines{config_hash="<any>",schema_id="<any>",otel_scope_name="apollo/router"} 1"#, None)
+    /// ```
+    /// Will allow the metric to be checked even if the config hash and schema id are fluid.
     pub async fn assert_metrics_contains(&self, text: &str, duration: Option<Duration>) {
         let now = Instant::now();
         let mut last_metrics = String::new();
+        let text = regex::escape(text).replace("<any>", ".+");
+        let re = Regex::new(&format!("(?m)^{}", text)).expect("Invalid regex");
         while now.elapsed() < duration.unwrap_or_else(|| Duration::from_secs(15)) {
             if let Ok(metrics) = self
                 .get_metrics_response()
@@ -936,7 +956,7 @@ impl IntegrationTest {
                 .text()
                 .await
             {
-                if metrics.contains(text) {
+                if re.is_match(&metrics) {
                     return;
                 }
                 last_metrics = metrics;
diff --git a/apollo-router/tests/integration/telemetry/fixtures/prometheus_updated.router.yaml b/apollo-router/tests/integration/telemetry/fixtures/prometheus_updated.router.yaml
new file mode 100644
index 0000000000..d620bf738e
--- /dev/null
+++ b/apollo-router/tests/integration/telemetry/fixtures/prometheus_updated.router.yaml
@@ -0,0 +1,53 @@
+limits:
+  http_max_request_bytes: 200
+telemetry:
+  instrumentation:
+    instruments:
+      default_requirement_level: required
+      router:
+        http.server.request.duration:
+          attributes:
+            server.port: false
+            server.address: false
+            http.response.status_code:
+              alias: status
+            error:
+              error: reason
+  exporters:
+    metrics:
+      prometheus:
+        listen: 127.0.0.1:4000
+        enabled: true
+        path: /metrics
+      common:
+        views:
+          - name: apollo_router_http_request_duration_seconds
+            aggregation:
+              histogram:
+                buckets:
+                  - 0.1
+                  - 0.5
+                  - 1
+                  - 2
+                  - 3
+                  - 4
+                  - 5
+                  - 100
+                  - 1000
+headers:
+  all:
+    request:
+      - insert:
+          name: "x-custom-header"
+          value: "test_custom"
+override_subgraph_url:
+  products: http://localhost:4005
+include_subgraph_errors:
+  all: true
+supergraph:
+  introspection: true
+apq:
+  router:
+    cache:
+      in_memory:
+        limit: 1000
\ No newline at end of file
diff --git a/apollo-router/tests/integration/telemetry/metrics.rs b/apollo-router/tests/integration/telemetry/metrics.rs
index f1a4c54fa8..fc901ca8c7 100644
--- a/apollo-router/tests/integration/telemetry/metrics.rs
+++ b/apollo-router/tests/integration/telemetry/metrics.rs
@@ -1,6 +1,8 @@
 use std::time::Duration;
 
+use regex::Regex;
 use serde_json::json;
+use wiremock::ResponseTemplate;
 
 use crate::integration::IntegrationTest;
 use crate::integration::common::Query;
@@ -318,4 +320,46 @@ async fn test_gauges_on_reload() {
             None,
         )
         .await;
+
+    router
+        .assert_metrics_contains(r#"apollo_router_pipelines{config_hash="<any>",schema_id="<any>",otel_scope_name="apollo/router"} 1"#, None)
+        .await;
+}
+
+#[tokio::test(flavor = "multi_thread")]
+async fn test_multi_pipelines() {
+    if !graph_os_enabled() {
+        eprintln!("test skipped");
+        return;
+    }
+    let mut router = IntegrationTest::builder()
+        .config(PROMETHEUS_CONFIG)
+        .responder(ResponseTemplate::new(500).set_delay(Duration::from_secs(10)))
+        .build()
+        .await;
+
+    router.start().await;
+    router.assert_started().await;
+
+    let query = router.execute_default_query();
+    // Long running request 1
+    let _h1 = tokio::task::spawn(query);
+    router
+        .update_config(include_str!("fixtures/prometheus_updated.router.yaml"))
+        .await;
+
+    router.assert_reloaded().await;
+    // Long running request 2
+    let query = router.execute_default_query();
+    let _h2 = tokio::task::spawn(query);
+    let metrics = router
+        .get_metrics_response()
+        .await
+        .expect("metrics")
+        .text()
+        .await
+        .expect("metrics");
+    // There should be two instances of the pipeline metrics
+    let regex = Regex::new(r#"(?m)^apollo_router_pipelines[{].+[}] 1"#).expect("regex");
+    assert_eq!(regex.captures_iter(&metrics).count(), 2);
 }
diff --git a/apollo-router/tests/integration/telemetry/otlp.rs b/apollo-router/tests/integration/telemetry/otlp.rs
index 9c428c88ea..f3731aa2a4 100644
--- a/apollo-router/tests/integration/telemetry/otlp.rs
+++ b/apollo-router/tests/integration/telemetry/otlp.rs
@@ -47,7 +47,7 @@ async fn test_trace_error() -> Result<(), BoxError> {
 
     router.start().await;
     router.assert_started().await;
-    router.assert_log_contained("OpenTelemetry trace error occurred: cannot send message to batch processor 'otlp-tracing' as the channel is full").await;
+    router.assert_log_contained("OpenTelemetry trace error occurred: cannot send message to batch processor 'otlp-tracing' as the channel is full");
     router.assert_metrics_contains(r#"apollo_router_telemetry_batch_processor_errors_total{error="channel full",name="otlp-tracing",otel_scope_name="apollo/router"}"#, None).await;
     router.graceful_shutdown().await;
 
diff --git a/docs/source/routing/observability/telemetry/instrumentation/standard-instruments.mdx b/docs/source/routing/observability/telemetry/instrumentation/standard-instruments.mdx
index 2eba69209f..b44ee2aea3 100644
--- a/docs/source/routing/observability/telemetry/instrumentation/standard-instruments.mdx
+++ b/docs/source/routing/observability/telemetry/instrumentation/standard-instruments.mdx
@@ -120,4 +120,10 @@ The initial call to Uplink during router startup is not reflected in metrics.
 
 - `apollo.router.telemetry.batch_processor.errors` - The number of errors encountered by exporter batch processors.
   - `name`: One of `apollo-tracing`, `datadog-tracing`, `jaeger-collector`, `otlp-tracing`, `zipkin-tracing`.
-  - `error` = One of `channel closed`, `channel full`.
+  - `error`: One of `channel closed`, `channel full`.
+
+### Internals
+- `apollo.router.pipelines` - The number of request pipelines active in the router
+    - `schema.id` - The Apollo Studio schema hash associated with the pipeline.
+    - `launch.id` - The Apollo Studio launch id associated with the pipeline (optional).
+    - `config.hash` - The hash of the configuration


Reviews:
user: login: tninesling
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T16:46:07Z
user: login: goto-bus-stop
body: +1 to Tyler's Clone call-out, we should either remove the impl or adjust the comment 😁 

Do you think it's worthwhile to add a test where we have two pipelines active? It could submit any GraphQL request but not read the response (forcing the connection to stay open server-side), and then trigger a reload.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-11T09:51:21Z
user: login: bnjjj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T09:57:23Z
user: login: goto-bus-stop
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T10:02:49Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:25:35Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:25:48Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:30:08Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:30:21Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:30:33Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:31:11Z
user: login: BrynCooke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:31:24Z
user: login: goto-bus-stop
body: sweet!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-11T14:26:35Z

Review comments:
diff_hunk: @@ -0,0 +1,70 @@
+use std::collections::HashMap;
+use std::sync::Mutex;
+use std::sync::MutexGuard;
+use std::sync::OnceLock;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES
+        .get_or_init(Default::default)
+        .lock()
+        .expect("poisoned")
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: tninesling
body: Is it worth using `parking_lot::Mutex` instead to avoid this call to `.expect()`?
created_at: 2025-03-10T16:39:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,14 @@
+### Add apollo.router.pipelines metrics ([PR #6967](https://github.com/apollographql/router/pull/6967))
+
+When the Router reloads either via schema change or config change a new request pipeline is created.
+Existing request pipelines are closed once their requests finish. However, this may not happen if there are ongoing long requests 
+such as Subscriptions that do not finish.
+
+To enable debugging when request pipelines are being kept around a new gauge metric has been added:
+
+- `apollo.router.pipelines` - The number of request pipelines active in the router
+    - `schema.id` - The Apollo Studio schema hash associated with the pipeline.
path: .changesets/feat_pipeline_metrics.md
user: login: tninesling
body: Nice! This would have been super useful in a couple debugging sessions over the past year. Excited to see this making it in
created_at: 2025-03-10T16:41:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,70 @@
+use std::collections::HashMap;
+use std::sync::Mutex;
+use std::sync::MutexGuard;
+use std::sync::OnceLock;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES
+        .get_or_init(Default::default)
+        .lock()
+        .expect("poisoned")
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id: schema_id.to_string(),
+            launch_id,
+            config_hash: config_hash.to_string(),
+        };
+        pipelines()
+            .entry(pipeline_ref.clone())
+            .and_modify(|p| *p += 1)
+            .or_insert(1);
+        PipelineHandle { pipeline_ref }
+    }
+}
+
+impl Clone for PipelineHandle {
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: tninesling
body: > Clone MUST NOT be implemented for this type.

I was surprised to see this based on the above comment. Was that just meant to indicate we shouldn't derive a naive clone impl on that type?
created_at: 2025-03-10T16:45:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1001,6 +1002,17 @@ impl RouterCreator {
         // For now just call activate to make the gauges work on the happy path.
         apq_layer.activate();
 
+        // Create a handle that will help us keep track of this pipeline.
+        // A metric is exposed that allows the use to see if pipelines are being hung onto.
+        let schema_id = supergraph_creator.schema().schema_id.clone();
+        let launch_id = supergraph_creator.schema().launch_id.clone();
path: apollo-router/src/services/router/service.rs
user: login: goto-bus-stop
body: We first call `.clone()` and then `.to_string()` on the clone... can these two just be references?
created_at: 2025-03-10T16:53:14Z
author_association: MEMBER
diff_hunk: @@ -267,16 +266,38 @@ struct BuiltinInstruments {
     subgraph_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     connector_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     cache_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
+    _gauges: Vec<ObservableGauge<u64>>,
 }
 
 fn create_builtin_instruments(config: &InstrumentsConfig) -> BuiltinInstruments {
path: apollo-router/src/plugins/telemetry/mod.rs
user: login: bnjjj
body: Just a nit but I don't understand why it's not just a method on `BuiltinInstruments` like `BuiltinInstruments::new(config)` 
created_at: 2025-03-11T09:51:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -913,6 +913,8 @@ impl IntegrationTest {
     pub async fn assert_metrics_contains(&self, text: &str, duration: Option<Duration>) {
         let now = Instant::now();
         let mut last_metrics = String::new();
+        let text = regex::escape(text).replace("<any>", ".+");
+        let re = Regex::new(&format!("(?m)^{}", text)).expect("Invalid regex");
path: apollo-router/tests/common.rs
user: login: bnjjj
body: It would be interesting to document this somewhere to know it exists and we can use it
created_at: 2025-03-11T09:56:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+use std::collections::HashMap;
+use std::sync::OnceLock;
+
+use parking_lot::Mutex;
+use parking_lot::MutexGuard;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES.get_or_init(Default::default).lock()
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id: schema_id.to_string(),
+            launch_id,
+            config_hash: config_hash.to_string(),
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: goto-bus-stop
body: ```suggestion
            schema_id,
            launch_id,
            config_hash,
```
already Strings
created_at: 2025-03-11T10:02:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,70 @@
+use std::collections::HashMap;
+use std::sync::Mutex;
+use std::sync::MutexGuard;
+use std::sync::OnceLock;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES
+        .get_or_init(Default::default)
+        .lock()
+        .expect("poisoned")
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id: schema_id.to_string(),
+            launch_id,
+            config_hash: config_hash.to_string(),
+        };
+        pipelines()
+            .entry(pipeline_ref.clone())
+            .and_modify(|p| *p += 1)
+            .or_insert(1);
+        PipelineHandle { pipeline_ref }
+    }
+}
+
+impl Clone for PipelineHandle {
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: BrynCooke
body: Yeah, the comment was correct. I deleted the redundant code.
created_at: 2025-03-11T10:25:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,70 @@
+use std::collections::HashMap;
+use std::sync::Mutex;
+use std::sync::MutexGuard;
+use std::sync::OnceLock;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES
+        .get_or_init(Default::default)
+        .lock()
+        .expect("poisoned")
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id: schema_id.to_string(),
+            launch_id,
+            config_hash: config_hash.to_string(),
+        };
+        pipelines()
+            .entry(pipeline_ref.clone())
+            .and_modify(|p| *p += 1)
+            .or_insert(1);
+        PipelineHandle { pipeline_ref }
+    }
+}
+
+impl Clone for PipelineHandle {
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: BrynCooke
body: [e508ac9](https://github.com/apollographql/router/pull/6967/commits/e508ac9608873057792d0c2b8c84868b656068bc)
created_at: 2025-03-11T10:25:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+use std::collections::HashMap;
+use std::sync::OnceLock;
+
+use parking_lot::Mutex;
+use parking_lot::MutexGuard;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES.get_or_init(Default::default).lock()
+}
+
+impl PipelineHandle {
+    pub(crate) fn new(schema_id: String, launch_id: Option<String>, config_hash: String) -> Self {
+        let pipeline_ref = PipelineRef {
+            schema_id: schema_id.to_string(),
+            launch_id,
+            config_hash: config_hash.to_string(),
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: BrynCooke
body: [4f56071](https://github.com/apollographql/router/pull/6967/commits/4f560710d05fe840f6f7313a3bc879530e65d296)
created_at: 2025-03-11T10:30:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -913,6 +913,8 @@ impl IntegrationTest {
     pub async fn assert_metrics_contains(&self, text: &str, duration: Option<Duration>) {
         let now = Instant::now();
         let mut last_metrics = String::new();
+        let text = regex::escape(text).replace("<any>", ".+");
+        let re = Regex::new(&format!("(?m)^{}", text)).expect("Invalid regex");
path: apollo-router/tests/common.rs
user: login: BrynCooke
body: [749279d](https://github.com/apollographql/router/pull/6967/commits/749279de393327811f7df01d15f00736e60ecfc3)
created_at: 2025-03-11T10:30:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -267,16 +266,38 @@ struct BuiltinInstruments {
     subgraph_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     connector_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     cache_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
+    _gauges: Vec<ObservableGauge<u64>>,
 }
 
 fn create_builtin_instruments(config: &InstrumentsConfig) -> BuiltinInstruments {
path: apollo-router/src/plugins/telemetry/mod.rs
user: login: BrynCooke
body: [6ca71d2](https://github.com/apollographql/router/pull/6967/commits/6ca71d2bb57b2d8172d0cb7ee8830f6a15e30672)
created_at: 2025-03-11T10:30:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -267,16 +266,38 @@ struct BuiltinInstruments {
     subgraph_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     connector_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
     cache_custom_instruments: Arc<HashMap<String, StaticInstrument>>,
+    _gauges: Vec<ObservableGauge<u64>>,
 }
 
 fn create_builtin_instruments(config: &InstrumentsConfig) -> BuiltinInstruments {
path: apollo-router/src/plugins/telemetry/mod.rs
user: login: BrynCooke
body: I brought the code into alignment. We may add config in future for the pipelines metric, but probably not. It's good to have a standard pattern though.
created_at: 2025-03-11T10:31:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,70 @@
+use std::collections::HashMap;
+use std::sync::Mutex;
+use std::sync::MutexGuard;
+use std::sync::OnceLock;
+
+/// A pipeline is used to keep track of how many pipelines we have active. It's associated with an instance of RouterCreator
+/// The telemetry plugin has a gauge to expose this data
+/// Pipeline ref represents a unique pipeline
+#[derive(Clone, Hash, Eq, PartialEq, Debug)]
+pub(crate) struct PipelineRef {
+    pub(crate) schema_id: String,
+    pub(crate) launch_id: Option<String>,
+    pub(crate) config_hash: String,
+}
+
+/// A pipeline handle does the actual tracking of pipelines
+/// Creating a new pipeline handle will insert a PipelineRef into a static map.
+/// Dropping all pipeline handles associated with the internal ref will remove the PipelineRef
+/// Clone MUST NOT be implemented for this type. Cloning will make extra copies that when dropped will throw off the global count.
+pub(crate) struct PipelineHandle {
+    pipeline_ref: PipelineRef,
+}
+
+static PIPELINES: OnceLock<Mutex<HashMap<PipelineRef, u64>>> = OnceLock::new();
+pub(crate) fn pipelines() -> MutexGuard<'static, HashMap<PipelineRef, u64>> {
+    PIPELINES
+        .get_or_init(Default::default)
+        .lock()
+        .expect("poisoned")
path: apollo-router/src/services/router/pipeline_handle.rs
user: login: BrynCooke
body: [61bc841](https://github.com/apollographql/router/pull/6967/commits/61bc8410ba899ed1c958066d89d8b21067288cae)
created_at: 2025-03-11T10:31:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-10T12:16:14Z
body: @BrynCooke, please consider creating a changeset entry in `/.changesets/`. [These instructions](https://github.com/apollographql/router/blob/HEAD/.changesets/README.md) describe the process and tooling.
author_association: CONTRIBUTOR
user: login: svc-apollo-docs
created_at: 2025-03-10T12:16:45Z
body: ### ✅ Docs preview ready

The preview is ready to be viewed. [View the preview](https://www.apollographql.com/docs/deploy-preview/dd215e28d6e5ed73d3402cd7)

File Changes

<details>
<summary>0 new, 1 changed, 0 removed</summary>

```
* graphos/routing/(latest)/observability/telemetry/instrumentation/standard-instruments.mdx
```

</details>

**Build ID**: dd215e28d6e5ed73d3402cd7

**URL**: [https://www.apollographql.com/docs/deploy-preview/dd215e28d6e5ed73d3402cd7](https://www.apollographql.com/docs/deploy-preview/dd215e28d6e5ed73d3402cd7)
author_association: NONE
user: login: BrynCooke
created_at: 2025-03-11T11:36:08Z
body: Trying to write a test where two pipelines are active is proving difficult, but worth doing. Gonna have to pause this for now, but will implement before merging.
author_association: CONTRIBUTOR
user: login: BrynCooke
created_at: 2025-03-11T13:56:38Z
body: @goto-bus-stop  I did it! [9c14c98](https://github.com/apollographql/router/pull/6967/commits/9c14c98a6285cbf5225c48e16723a24debfc7999)

Interestingly I couldn't get it to do the thing by not reading the response. I wonder if there is some buffering somewhere that automagically happens.

author_association: CONTRIBUTOR