Repository: MaterializeInc/materialize
Pull request #31770: storage: add replica_id to mz_source_status_history
Author: login: aljoscha
Created: 2025-03-05T15:17:33Z | Merged: 2025-03-15T06:35:07Z

Implements MaterializeInc/database-issues#8974


Diff:
diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md
index 479ebaa3007a2..8af0db4840e5d 100644
--- a/doc/user/content/sql/system-catalog/mz_internal.md
+++ b/doc/user/content/sql/system-catalog/mz_internal.md
@@ -1048,6 +1048,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the sink: one of `created`, `starting`, `running`, `stalled`, `failed`, or `dropped`.              |
 | `error`        | [`text`]                        | If the sink is in an error state, the error message.                                                             |
 | `details`      | [`jsonb`]                       | Additional metadata provided by the sink. In case of error, may contain a `hint` field with helpful suggestions. |
+| `replica_id`   | [`text`]                        | The ID of the replica that an instance of a sink is running on.                                                  |
 
 <!-- RELATION_SPEC_UNDOCUMENTED mz_internal.mz_source_statistics_raw -->
 
@@ -1168,6 +1169,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the source: one of `created`, `starting`, `running`, `paused`, `stalled`, `failed`, or `dropped`.    |
 | `error`        | [`text`]                        | If the source is in an error state, the error message.                                                             |
 | `details`      | [`jsonb`]                       | Additional metadata provided by the source. In case of error, may contain a `hint` field with helpful suggestions. |
+| `replica_id`   | [`text`]                        | The ID of the replica that an instance of a source is running on.                                                  |
 
 <!--
 ## `mz_statement_execution_history`
diff --git a/misc/python/materialize/checks/all_checks/source_errors.py b/misc/python/materialize/checks/all_checks/source_errors.py
index e0faf01eaedf1..4011ea4134db0 100644
--- a/misc/python/materialize/checks/all_checks/source_errors.py
+++ b/misc/python/materialize/checks/all_checks/source_errors.py
@@ -91,18 +91,35 @@ def manipulate(self) -> list[Testdrive]:
     def validate(self) -> Testdrive:
         return Testdrive(
             dedent(
-                # This could also check that the error propagates to subsources,
-                # but the GlobalId migration that occurrs in v0.98 means that we
-                # lose historical data for source errors and this check is not
-                # so crucial that it's imperative that we correlate the original
-                # IDs to these errors.
+                # We check two things: a) that the expected (sub)sources are
+                # stalled, and b) that they report the expected error. Only
+                # checking the error using bool_and wouldn't work because this
+                # check ignores NULL values, so would succeed if, say, all
+                # sources are in state 'running'.
+                #
+                # TODO(aljoscha): We recently migrated the status history
+                # collection, so all updates are lost when upgrading. This has
+                # the consequence that sources report as 'created'. We therefore
+                # have to accept 'created' below, but should remove this
+                # relaxation once we have enough new releasees that there won't
+                # be a migration between tested versions. Plus, because of this
+                # we have to wrap the error check in a coalesce: when all the
+                # status updates show 'created', we'll have no error and get a
+                # NULL result.
+                #
+                # Additionally, we also have to accept paused, because platform
+                # checks might pause replicas, and these paused status updates
+                # take precedence over errors. To fix this, we might want to
+                # rewrite this test to look at mz_source_status_history
+                # instead, which contains the full history.
                 """
-                > SELECT bool_and(error ~* 'publication .+ does not exist')
+                > SELECT
+                        coalesce(bool_and(error ~* 'publication .+ does not exist'), true) as matches,
+                        bool_and(status IN ('stalled', 'created', 'paused')) as is_stalled
                     FROM mz_internal.mz_source_statuses
                     WHERE
-                    name
-                    IN ('source_errors_sourcea', 'source_errors_sourceb');
-                true
+                        name IN ('source_errors_sourcea', 'source_errors_sourceb', 'source_errors_tablea', 'source_errors_tableb');
+                true true
                 """
             )
         )
diff --git a/src/adapter/src/catalog.rs b/src/adapter/src/catalog.rs
index cc9a5f3071535..a7f3a1b0575a5 100644
--- a/src/adapter/src/catalog.rs
+++ b/src/adapter/src/catalog.rs
@@ -26,7 +26,7 @@ use mz_adapter_types::bootstrap_builtin_cluster_config::{
 };
 use mz_adapter_types::connection::ConnectionId;
 use mz_audit_log::{EventType, FullNameV1, ObjectType, VersionedStorageUsage};
-use mz_build_info::DUMMY_BUILD_INFO;
+use mz_build_info::{BuildInfo, DUMMY_BUILD_INFO};
 use mz_catalog::builtin::{
     BuiltinCluster, BuiltinLog, BuiltinSource, BuiltinTable, BUILTIN_PREFIXES,
     MZ_CATALOG_SERVER_CLUSTER,
@@ -590,6 +590,7 @@ impl Catalog {
             storage,
             now,
             environment_id,
+            &DUMMY_BUILD_INFO,
             system_parameter_defaults,
             bootstrap_args,
             None,
@@ -621,6 +622,7 @@ impl Catalog {
             storage,
             now,
             environment_id,
+            &DUMMY_BUILD_INFO,
             system_parameter_defaults,
             bootstrap_args,
             None,
@@ -637,13 +639,18 @@ impl Catalog {
         now: NowFn,
         environment_id: EnvironmentId,
         system_parameter_defaults: BTreeMap<String, String>,
-        version: semver::Version,
+        build_info: &'static BuildInfo,
         bootstrap_args: &BootstrapArgs,
         enable_expression_cache_override: Option<bool>,
     ) -> Result<Catalog, anyhow::Error> {
         let openable_storage = TestCatalogStateBuilder::new(persist_client.clone())
             .with_organization_id(environment_id.organization_id())
-            .with_version(version)
+            .with_version(
+                build_info
+                    .version
+                    .parse()
+                    .expect("build version is parseable"),
+            )
             .build()
             .await?;
         let storage = openable_storage.open_read_only(bootstrap_args).await?;
@@ -652,6 +659,7 @@ impl Catalog {
             storage,
             now,
             Some(environment_id),
+            build_info,
             system_parameter_defaults,
             bootstrap_args,
             enable_expression_cache_override,
@@ -664,6 +672,7 @@ impl Catalog {
         storage: Box<dyn DurableCatalogState>,
         now: NowFn,
         environment_id: Option<EnvironmentId>,
+        build_info: &'static BuildInfo,
         system_parameter_defaults: BTreeMap<String, String>,
         bootstrap_args: &BootstrapArgs,
         enable_expression_cache_override: Option<bool>,
@@ -675,6 +684,7 @@ impl Catalog {
         let previous_ts = now().into();
         let replica_size = &bootstrap_args.default_cluster_replica_size;
         let read_only = false;
+
         let OpenCatalogResult {
             catalog,
             migrated_storage_collections_0dt: _,
@@ -688,7 +698,7 @@ impl Catalog {
             state: StateConfig {
                 unsafe_mode: true,
                 all_features: false,
-                build_info: &DUMMY_BUILD_INFO,
+                build_info,
                 environment_id: environment_id.unwrap_or(EnvironmentId::for_tests()),
                 read_only,
                 now,
diff --git a/src/catalog/src/builtin.rs b/src/catalog/src/builtin.rs
index 3967ac980b0ed..b2c0260e4e300 100644
--- a/src/catalog/src/builtin.rs
+++ b/src/catalog/src/builtin.rs
@@ -3210,13 +3210,52 @@ pub static MZ_SOURCE_STATUSES: LazyLock<BuiltinView> = LazyLock::new(|| BuiltinV
     column_defs: None,
     sql: "
     WITH
-    -- Get the latest events
+    -- The status history contains per-replica events and source-global events.
+    -- For the latter, replica_id is NULL. We turn these into '<source>', so that
+    -- we can treat them uniformly below.
+    uniform_status_history AS
+    (
+        SELECT
+            s.source_id,
+            COALESCE(s.replica_id, '<source>') as replica_id,
+            s.occurred_at,
+            s.status,
+            s.error,
+            s.details
+        FROM mz_internal.mz_source_status_history s
+    ),
+    -- For getting the latest events, we first determine the latest per-replica
+    -- events here and then apply precedence rules below.
+    latest_per_replica_events AS
+    (
+        SELECT DISTINCT ON (source_id, replica_id)
+            occurred_at, source_id, replica_id, status, error, details
+        FROM uniform_status_history
+        ORDER BY source_id, replica_id, occurred_at DESC
+    ),
+    -- We have a precedence list that determines the overall status in case
+    -- there is differing per-replica (including source-global) statuses. If
+    -- there is no 'dropped' status, and any replica reports 'running', the
+    -- overall status is 'running' even if there might be some replica that has
+    -- errors or is paused.
     latest_events AS
     (
-        SELECT DISTINCT ON (source_id)
-            occurred_at, source_id, status, error, details
-        FROM mz_internal.mz_source_status_history
-        ORDER BY source_id, occurred_at DESC
+       SELECT DISTINCT ON (source_id)
+            source_id,
+            occurred_at,
+            status,
+            error,
+            details
+        FROM latest_per_replica_events
+        ORDER BY source_id, CASE status
+                    WHEN 'dropped' THEN 1
+                    WHEN 'running' THEN 2
+                    WHEN 'stalled' THEN 3
+                    WHEN 'starting' THEN 4
+                    WHEN 'paused' THEN 5
+                    WHEN 'ceased' THEN 6
+                    ELSE 7  -- For any other status values
+                END
     ),
     -- Determine which sources are subsources and which are parent sources
     subsources AS
@@ -3337,10 +3376,53 @@ pub static MZ_SINK_STATUSES: LazyLock<BuiltinView> = LazyLock::new(|| BuiltinVie
     oid: oid::VIEW_MZ_SINK_STATUSES_OID,
     column_defs: None,
     sql: "
-WITH latest_events AS (
-    SELECT DISTINCT ON(sink_id) occurred_at, sink_id, status, error, details
-    FROM mz_internal.mz_sink_status_history
-    ORDER BY sink_id, occurred_at DESC
+WITH
+-- The status history contains per-replica events and sink-global events.
+-- For the latter, replica_id is NULL. We turn these into '<sink>', so that
+-- we can treat them uniformly below.
+uniform_status_history AS
+(
+    SELECT
+        s.sink_id,
+        COALESCE(s.replica_id, '<sink>') as replica_id,
+        s.occurred_at,
+        s.status,
+        s.error,
+        s.details
+    FROM mz_internal.mz_sink_status_history s
+),
+-- For getting the latest events, we first determine the latest per-replica
+-- events here and then apply precedence rules below.
+latest_per_replica_events AS
+(
+    SELECT DISTINCT ON (sink_id, replica_id)
+        occurred_at, sink_id, replica_id, status, error, details
+    FROM uniform_status_history
+    ORDER BY sink_id, replica_id, occurred_at DESC
+),
+-- We have a precedence list that determines the overall status in case
+-- there is differing per-replica (including sink-global) statuses. If
+-- there is no 'dropped' status, and any replica reports 'running', the
+-- overall status is 'running' even if there might be some replica that has
+-- errors or is paused.
+latest_events AS
+(
+    SELECT DISTINCT ON (sink_id)
+        sink_id,
+        occurred_at,
+        status,
+        error,
+        details
+    FROM latest_per_replica_events
+    ORDER BY sink_id, CASE status
+                WHEN 'dropped' THEN 1
+                WHEN 'running' THEN 2
+                WHEN 'stalled' THEN 3
+                WHEN 'starting' THEN 4
+                WHEN 'paused' THEN 5
+                WHEN 'ceased' THEN 6
+                ELSE 7  -- For any other status values
+            END
 )
 SELECT
     mz_sinks.id,
diff --git a/src/storage-client/src/client.proto b/src/storage-client/src/client.proto
index b97cbc8426f48..f1ab5239f07dd 100644
--- a/src/storage-client/src/client.proto
+++ b/src/storage-client/src/client.proto
@@ -131,6 +131,7 @@ message ProtoStorageResponse {
     optional string error = 4;
     repeated string hints = 5;
     map<string, string> namespaced_errors = 6;
+    optional string replica_id = 7;
   }
 
   message ProtoStatusUpdates {
diff --git a/src/storage-client/src/client.rs b/src/storage-client/src/client.rs
index 8bd041798d74b..8468bb406e641 100644
--- a/src/storage-client/src/client.rs
+++ b/src/storage-client/src/client.rs
@@ -22,6 +22,7 @@ use async_trait::async_trait;
 use differential_dataflow::difference::Semigroup;
 use differential_dataflow::lattice::Lattice;
 use mz_cluster_client::client::{ClusterStartupEpoch, TimelyConfig, TryIntoTimelyConfig};
+use mz_cluster_client::ReplicaId;
 use mz_ore::assert_none;
 use mz_persist_client::batch::{BatchBuilder, ProtoBatch};
 use mz_persist_client::write::WriteHandle;
@@ -484,6 +485,7 @@ pub struct StatusUpdate {
     pub error: Option<String>,
     pub hints: BTreeSet<String>,
     pub namespaced_errors: BTreeMap<String, String>,
+    pub replica_id: Option<ReplicaId>,
 }
 
 impl StatusUpdate {
@@ -499,6 +501,7 @@ impl StatusUpdate {
             error: None,
             hints: Default::default(),
             namespaced_errors: Default::default(),
+            replica_id: None,
         }
     }
 }
@@ -539,6 +542,11 @@ impl From<StatusUpdate> for Row {
             packer.push(Datum::Null);
         }
 
+        match update.replica_id {
+            Some(id) => packer.push(Datum::String(&id.to_string())),
+            None => packer.push(Datum::Null),
+        }
+
         row
     }
 }
@@ -585,6 +593,7 @@ impl RustType<proto_storage_response::ProtoStatusUpdate> for StatusUpdate {
             error: self.error.clone(),
             hints: self.hints.iter().cloned().collect(),
             namespaced_errors: self.namespaced_errors.clone(),
+            replica_id: self.replica_id.map(|id| id.to_string().into_proto()),
         }
     }
 
@@ -602,6 +611,9 @@ impl RustType<proto_storage_response::ProtoStatusUpdate> for StatusUpdate {
             error: proto.error,
             hints: proto.hints.into_iter().collect(),
             namespaced_errors: proto.namespaced_errors,
+            replica_id: proto
+                .replica_id
+                .map(|replica_id: String| replica_id.parse().expect("must be a valid replica id")),
         })
     }
 }
diff --git a/src/storage-client/src/healthcheck.rs b/src/storage-client/src/healthcheck.rs
index 5339fbd82960b..8627e66313a75 100644
--- a/src/storage-client/src/healthcheck.rs
+++ b/src/storage-client/src/healthcheck.rs
@@ -112,6 +112,7 @@ pub static MZ_SOURCE_STATUS_HISTORY_DESC: LazyLock<RelationDesc> = LazyLock::new
         .with_column("status", ScalarType::String.nullable(false))
         .with_column("error", ScalarType::String.nullable(true))
         .with_column("details", ScalarType::Jsonb.nullable(true))
+        .with_column("replica_id", ScalarType::String.nullable(true))
         .finish()
 });
 
@@ -125,6 +126,7 @@ pub static MZ_SINK_STATUS_HISTORY_DESC: LazyLock<RelationDesc> = LazyLock::new(|
         .with_column("status", ScalarType::String.nullable(false))
         .with_column("error", ScalarType::String.nullable(true))
         .with_column("details", ScalarType::Jsonb.nullable(true))
+        .with_column("replica_id", ScalarType::String.nullable(true))
         .finish()
 });
 
diff --git a/src/storage-controller/src/collection_mgmt.rs b/src/storage-controller/src/collection_mgmt.rs
index b69c06efca70a..8020ccfaa3b0a 100644
--- a/src/storage-controller/src/collection_mgmt.rs
+++ b/src/storage-controller/src/collection_mgmt.rs
@@ -74,6 +74,7 @@ use differential_dataflow::lattice::Lattice;
 use futures::future::BoxFuture;
 use futures::stream::StreamExt;
 use futures::{Future, FutureExt};
+use mz_cluster_client::ReplicaId;
 use mz_dyncfg::ConfigSet;
 use mz_ore::now::{EpochMillis, NowFn};
 use mz_ore::retry::Retry;
@@ -1045,7 +1046,7 @@ where
     /// We have to shut down when receiving from this.
     shutdown_rx: oneshot::Receiver<()>,
     /// If this collection deduplicates statuses, this map is used to track the previous status.
-    previous_statuses: Option<BTreeMap<GlobalId, Status>>,
+    previous_statuses: Option<BTreeMap<(GlobalId, Option<ReplicaId>), Status>>,
 }
 
 impl<T> AppendOnlyWriteTask<T>
@@ -1070,37 +1071,38 @@ where
         let (tx, rx) = mpsc::unbounded_channel();
         let (shutdown_tx, shutdown_rx) = oneshot::channel();
 
-        let previous_statuses: Option<BTreeMap<GlobalId, Status>> = match introspection_config
-            .as_ref()
-            .map(|config| config.introspection_type)
-        {
-            Some(IntrospectionType::SourceStatusHistory)
-            | Some(IntrospectionType::SinkStatusHistory) => Some(BTreeMap::new()),
-
-            Some(IntrospectionType::ReplicaMetricsHistory)
-            | Some(IntrospectionType::WallclockLagHistory)
-            | Some(IntrospectionType::PrivatelinkConnectionStatusHistory)
-            | Some(IntrospectionType::ReplicaStatusHistory)
-            | Some(IntrospectionType::PreparedStatementHistory)
-            | Some(IntrospectionType::StatementExecutionHistory)
-            | Some(IntrospectionType::SessionHistory)
-            | Some(IntrospectionType::StatementLifecycleHistory)
-            | Some(IntrospectionType::SqlText)
-            | None => None,
-
-            Some(introspection_type @ IntrospectionType::ShardMapping)
-            | Some(introspection_type @ IntrospectionType::Frontiers)
-            | Some(introspection_type @ IntrospectionType::ReplicaFrontiers)
-            | Some(introspection_type @ IntrospectionType::StorageSourceStatistics)
-            | Some(introspection_type @ IntrospectionType::StorageSinkStatistics)
-            | Some(introspection_type @ IntrospectionType::ComputeDependencies)
-            | Some(introspection_type @ IntrospectionType::ComputeOperatorHydrationStatus)
-            | Some(introspection_type @ IntrospectionType::ComputeMaterializedViewRefreshes)
-            | Some(introspection_type @ IntrospectionType::ComputeErrorCounts)
-            | Some(introspection_type @ IntrospectionType::ComputeHydrationTimes) => {
-                unreachable!("not append-only collection: {introspection_type:?}")
-            }
-        };
+        let previous_statuses: Option<BTreeMap<(GlobalId, Option<ReplicaId>), Status>> =
+            match introspection_config
+                .as_ref()
+                .map(|config| config.introspection_type)
+            {
+                Some(IntrospectionType::SourceStatusHistory)
+                | Some(IntrospectionType::SinkStatusHistory) => Some(BTreeMap::new()),
+
+                Some(IntrospectionType::ReplicaMetricsHistory)
+                | Some(IntrospectionType::WallclockLagHistory)
+                | Some(IntrospectionType::PrivatelinkConnectionStatusHistory)
+                | Some(IntrospectionType::ReplicaStatusHistory)
+                | Some(IntrospectionType::PreparedStatementHistory)
+                | Some(IntrospectionType::StatementExecutionHistory)
+                | Some(IntrospectionType::SessionHistory)
+                | Some(IntrospectionType::StatementLifecycleHistory)
+                | Some(IntrospectionType::SqlText)
+                | None => None,
+
+                Some(introspection_type @ IntrospectionType::ShardMapping)
+                | Some(introspection_type @ IntrospectionType::Frontiers)
+                | Some(introspection_type @ IntrospectionType::ReplicaFrontiers)
+                | Some(introspection_type @ IntrospectionType::StorageSourceStatistics)
+                | Some(introspection_type @ IntrospectionType::StorageSinkStatistics)
+                | Some(introspection_type @ IntrospectionType::ComputeDependencies)
+                | Some(introspection_type @ IntrospectionType::ComputeOperatorHydrationStatus)
+                | Some(introspection_type @ IntrospectionType::ComputeMaterializedViewRefreshes)
+                | Some(introspection_type @ IntrospectionType::ComputeErrorCounts)
+                | Some(introspection_type @ IntrospectionType::ComputeHydrationTimes) => {
+                    unreachable!("not append-only collection: {introspection_type:?}")
+                }
+            };
 
         let mut task = Self {
             id,
@@ -1420,7 +1422,12 @@ where
                 .filter(|r| match r {
                     AppendOnlyUpdate::Row(_) => true,
                     AppendOnlyUpdate::Status(update) => {
-                        match (previous_statuses.get(&update.id).as_deref(), &update.status) {
+                        match (
+                            previous_statuses
+                                .get(&(update.id, update.replica_id))
+                                .as_deref(),
+                            &update.status,
+                        ) {
                             (None, _) => true,
                             (Some(old), new) => old.superseded_by(*new),
                         }
@@ -1429,7 +1436,9 @@ where
                 .collect();
             previous_statuses.extend(new.iter().filter_map(|update| match update {
                 AppendOnlyUpdate::Row(_) => None,
-                AppendOnlyUpdate::Status(update) => Some((update.id, update.status)),
+                AppendOnlyUpdate::Status(update) => {
+                    Some(((update.id, update.replica_id), update.status))
+                }
             }));
             new
         } else {
@@ -1782,6 +1791,7 @@ mod tests {
             error: Some(error_message.to_string()),
             hints: BTreeSet::from([hint.to_string()]),
             namespaced_errors: Default::default(),
+            replica_id: None,
         });
 
         for (datum, column_type) in row.iter().zip(MZ_SINK_STATUS_HISTORY_DESC.iter_types()) {
@@ -1826,6 +1836,7 @@ mod tests {
             error: Some(error_message.to_string()),
             hints: Default::default(),
             namespaced_errors: Default::default(),
+            replica_id: None,
         });
 
         for (datum, column_type) in row.iter().zip(MZ_SINK_STATUS_HISTORY_DESC.iter_types()) {
@@ -1854,6 +1865,7 @@ mod tests {
             error: None,
             hints: BTreeSet::from([hint.to_string()]),
             namespaced_errors: Default::default(),
+            replica_id: None,
         });
 
         for (datum, column_type) in row.iter().zip(MZ_SINK_STATUS_HISTORY_DESC.iter_types()) {
@@ -1898,6 +1910,7 @@ mod tests {
             error: Some(error_message.to_string()),
             hints: Default::default(),
             namespaced_errors: BTreeMap::from([("thing".to_string(), "error".to_string())]),
+            replica_id: None,
         });
 
         for (datum, column_type) in row.iter().zip(MZ_SINK_STATUS_HISTORY_DESC.iter_types()) {
@@ -1943,6 +1956,7 @@ mod tests {
             error: Some(error_message.to_string()),
             hints: BTreeSet::from([hint.to_string()]),
             namespaced_errors: BTreeMap::from([("thing".to_string(), "error".to_string())]),
+            replica_id: None,
         });
 
         for (datum, column_type) in row.iter().zip(MZ_SINK_STATUS_HISTORY_DESC.iter_types()) {
diff --git a/src/storage-controller/src/instance.rs b/src/storage-controller/src/instance.rs
index fed52cdf48fba..14cda570d49e9 100644
--- a/src/storage-controller/src/instance.rs
+++ b/src/storage-controller/src/instance.rs
@@ -56,6 +56,12 @@ pub(crate) struct Instance<T> {
     /// list of running ingestions is quite a bit more convenient in the
     /// implementation of `StorageController::active_ingestions`.
     active_ingestions: BTreeSet<GlobalId>,
+    /// The exports currently running on this instance.
+    ///
+    /// While this is derivable from `history` on demand, keeping a denormalized
+    /// list of running exports is quite a bit more convenient for the
+    /// controller.
+    active_exports: BTreeSet<GlobalId>,
     /// The command history, used to replay past commands when introducing new replicas or
     /// reconnecting to existing replicas.
     history: CommandHistory<T>,
@@ -95,6 +101,7 @@ where
         let mut instance = Self {
             replicas: Default::default(),
             active_ingestions: BTreeSet::new(),
+            active_exports: BTreeSet::new(),
             history,
             epoch,
             metrics,
@@ -160,6 +167,11 @@ where
         &self.active_ingestions
     }
 
+    /// Returns the exports running on this instance.
+    pub fn active_exports(&self) -> &BTreeSet<GlobalId> {
+        &self.active_exports
+    }
+
     /// Sets the status to paused for all sources/sinks in the history.
     fn update_paused_statuses(&mut self) {
         let now = mz_ore::now::to_datetime((self.now)());
@@ -172,6 +184,7 @@ where
                 "There is currently no replica running this {object_type}"
             )]),
             namespaced_errors: Default::default(),
+            replica_id: None,
         };
 
         self.history.reduce();
@@ -180,8 +193,24 @@ where
         for command in self.history.iter() {
             match command {
                 StorageCommand::RunIngestions(cmds) => {
-                    let updates = cmds.iter().map(|c| make_update(c.id, "source"));
-                    status_updates.extend(updates);
+                    for ingestion in cmds.iter() {
+                        // NOTE(aljoscha): We filter out the remap collection because we
+                        // don't get any status updates about it from the replica side. So
+                        // we don't want to synthesize a 'paused' status here.
+                        //
+                        // TODO(aljoscha): I think we want to fix this eventually, and make
+                        // sure we get status updates for the remap shard as well. Currently
+                        // its handling in the source status collection is a bit difficult
+                        // because we don't have updates for it in the status history
+                        // collection.
+                        let subsource_ids = ingestion
+                            .description
+                            .collection_ids()
+                            .filter(|id| id != &ingestion.description.remap_collection_id);
+                        for id in subsource_ids {
+                            status_updates.push(make_update(id, "source"));
+                        }
+                    }
                 }
                 StorageCommand::RunSinks(cmds) => {
                     let updates = cmds.iter().map(|c| make_update(c.id, "sink"));
@@ -208,10 +237,16 @@ where
                     self.active_ingestions.insert(ingestion.id);
                 }
             }
+            StorageCommand::RunSinks(sinks) => {
+                for sink in sinks {
+                    self.active_exports.insert(sink.id);
+                }
+            }
             StorageCommand::AllowCompaction(policies) => {
                 for (id, frontier) in policies {
                     if frontier.is_empty() {
                         self.active_ingestions.remove(id);
+                        self.active_exports.remove(id);
                     }
                 }
             }
diff --git a/src/storage-controller/src/lib.rs b/src/storage-controller/src/lib.rs
index 4e0292618b31b..ce378f12a9751 100644
--- a/src/storage-controller/src/lib.rs
+++ b/src/storage-controller/src/lib.rs
@@ -516,10 +516,66 @@ where
             .get_mut(&instance_id)
             .unwrap_or_else(|| panic!("instance {instance_id} does not exist"));
 
+        let status_now = mz_ore::now::to_datetime((self.now)());
+        let mut source_status_updates = vec![];
+        let mut sink_status_updates = vec![];
+
+        let make_update = |id, object_type| StatusUpdate {
+            id,
+            status: Status::Paused,
+            timestamp: status_now,
+            error: None,
+            hints: BTreeSet::from([format!(
+                "The replica running this {object_type} has been dropped"
+            )]),
+            namespaced_errors: Default::default(),
+            replica_id: Some(replica_id),
+        };
+
         for id in instance.active_ingestions() {
+            let ingestion = self
+                .collections
+                .get_mut(id)
+                .expect("instance contains unknown ingestion");
+
+            ingestion.active_replicas.remove(&replica_id);
+
+            if let Some(active_replicas) = self.dropped_objects.get_mut(id) {
+                active_replicas.remove(&replica_id);
+                if active_replicas.is_empty() {
+                    self.dropped_objects.remove(id);
+                }
+            }
+
+            let ingestion_description = match &ingestion.data_source {
+                DataSource::Ingestion(ingestion_description) => ingestion_description.clone(),
+                _ => panic!(
+                    "unexpected data source for ingestion: {:?}",
+                    ingestion.data_source
+                ),
+            };
+
+            // NOTE(aljoscha): We filter out the remap collection because we
+            // don't get any status updates about it from the replica side. So
+            // we don't want to synthesize a 'paused' status here.
+            //
+            // TODO(aljoscha): I think we want to fix this eventually, and make
+            // sure we get status updates for the remap shard as well. Currently
+            // its handling in the source status collection is a bit difficult
+            // because we don't have updates for it in the status history
+            // collection.
+            let subsource_ids = ingestion_description
+                .collection_ids()
+                .filter(|id| id != &ingestion_description.remap_collection_id);
+            for id in subsource_ids {
+                source_status_updates.push(make_update(id, "source"));
+            }
+        }
+
+        for id in instance.active_exports() {
             self.collections
                 .get_mut(id)
-                .expect("instance contains unknown ingestion")
+                .expect("instance contains unknown export")
                 .active_replicas
                 .remove(&replica_id);
 
@@ -529,8 +585,26 @@ where
                     self.dropped_objects.remove(id);
                 }
             }
+
+            sink_status_updates.push(make_update(*id, "sink"));
         }
+
         instance.drop_replica(replica_id);
+
+        if !self.read_only {
+            if !source_status_updates.is_empty() {
+                self.append_status_introspection_updates(
+                    IntrospectionType::SourceStatusHistory,
+                    source_status_updates,
+                );
+            }
+            if !sink_status_updates.is_empty() {
+                self.append_status_introspection_updates(
+                    IntrospectionType::SinkStatusHistory,
+                    sink_status_updates,
+                );
+            }
+        }
     }
 
     async fn evolve_nullability_for_bootstrap(
@@ -2120,8 +2194,8 @@ where
                     }
                 }
             }
-            Some((_replica_id, StorageResponse::StatusUpdates(updates))) => {
-                for status_update in updates.iter() {
+            Some((replica_id, StorageResponse::StatusUpdates(mut updates))) => {
+                for status_update in &mut updates {
                     // NOTE(aljoscha): We sniff out the hydration status for
                     // ingestions from status updates. This is the easiest we
                     // can do right now, without going deeper into changing the
@@ -2137,7 +2211,6 @@ where
                     //
                     // I wouldn't say it's ideal, but it's workable until we
                     // find something better.
-
                     match status_update.status {
                         Status::Running => {
                             let collection = self.collections.get_mut(&status_update.id);
@@ -2166,6 +2239,11 @@ where
                         }
                         _ => (),
                     }
+
+                    // Set replica_id in the status update if available
+                    if let Some(id) = replica_id {
+                        status_update.replica_id = Some(id);
+                    }
                 }
                 self.record_status_updates(updates);
             }
@@ -3529,9 +3607,12 @@ enum StatusHistoryRetentionPolicy {
     TimeWindow(Duration),
 }
 
-fn source_status_history_desc(params: &StorageParameters) -> StatusHistoryDesc<GlobalId> {
+fn source_status_history_desc(
+    params: &StorageParameters,
+) -> StatusHistoryDesc<(GlobalId, Option<ReplicaId>)> {
     let desc = &MZ_SOURCE_STATUS_HISTORY_DESC;
-    let (key_idx, _) = desc.get_by_name(&"source_id".into()).expect("exists");
+    let (source_id_idx, _) = desc.get_by_name(&"source_id".into()).expect("exists");
+    let (replica_id_idx, _) = desc.get_by_name(&"replica_id".into()).expect("exists");
     let (time_idx, _) = desc.get_by_name(&"occurred_at".into()).expect("exists");
 
     StatusHistoryDesc {
@@ -3539,15 +3620,28 @@ fn source_status_history_desc(params: &StorageParameters) -> StatusHistoryDesc<G
             params.keep_n_source_status_history_entries,
         ),
         extract_key: Box::new(move |datums| {
-            GlobalId::from_str(datums[key_idx].unwrap_str()).expect("GlobalId column")
+            (
+                GlobalId::from_str(datums[source_id_idx].unwrap_str()).expect("GlobalId column"),
+                if datums[replica_id_idx].is_null() {
+                    None
+                } else {
+                    Some(
+                        ReplicaId::from_str(datums[replica_id_idx].unwrap_str())
+                            .expect("ReplicaId column"),
+                    )
+                },
+            )
         }),
         extract_time: Box::new(move |datums| datums[time_idx].unwrap_timestamptz()),
     }
 }
 
-fn sink_status_history_desc(params: &StorageParameters) -> StatusHistoryDesc<GlobalId> {
+fn sink_status_history_desc(
+    params: &StorageParameters,
+) -> StatusHistoryDesc<(GlobalId, Option<ReplicaId>)> {
     let desc = &MZ_SINK_STATUS_HISTORY_DESC;
-    let (key_idx, _) = desc.get_by_name(&"sink_id".into()).expect("exists");
+    let (sink_id_idx, _) = desc.get_by_name(&"sink_id".into()).expect("exists");
+    let (replica_id_idx, _) = desc.get_by_name(&"replica_id".into()).expect("exists");
     let (time_idx, _) = desc.get_by_name(&"occurred_at".into()).expect("exists");
 
     StatusHistoryDesc {
@@ -3555,7 +3649,17 @@ fn sink_status_history_desc(params: &StorageParameters) -> StatusHistoryDesc<Glo
             params.keep_n_sink_status_history_entries,
         ),
         extract_key: Box::new(move |datums| {
-            GlobalId::from_str(datums[key_idx].unwrap_str()).expect("GlobalId column")
+            (
+                GlobalId::from_str(datums[sink_id_idx].unwrap_str()).expect("GlobalId column"),
+                if datums[replica_id_idx].is_null() {
+                    None
+                } else {
+                    Some(
+                        ReplicaId::from_str(datums[replica_id_idx].unwrap_str())
+                            .expect("ReplicaId column"),
+                    )
+                },
+            )
         }),
         extract_time: Box::new(move |datums| datums[time_idx].unwrap_timestamptz()),
     }
diff --git a/src/storage/src/healthcheck.rs b/src/storage/src/healthcheck.rs
index 72a0e25fff196..eeb9faa87f652 100644
--- a/src/storage/src/healthcheck.rs
+++ b/src/storage/src/healthcheck.rs
@@ -288,6 +288,7 @@ impl HealthOperator for DefaultWriter {
             } else {
                 BTreeMap::new()
             },
+            replica_id: None,
         });
     }
 
diff --git a/src/testdrive/src/action.rs b/src/testdrive/src/action.rs
index 4767404d634e4..36635cfda09fb 100644
--- a/src/testdrive/src/action.rs
+++ b/src/testdrive/src/action.rs
@@ -358,7 +358,7 @@ impl State {
     pub async fn with_catalog_copy<F, T>(
         &self,
         system_parameter_defaults: BTreeMap<String, String>,
-        version: semver::Version,
+        build_info: &'static BuildInfo,
         bootstrap_args: &BootstrapArgs,
         enable_expression_cache_override: Option<bool>,
         f: F,
@@ -394,7 +394,7 @@ impl State {
                 SYSTEM_TIME.clone(),
                 self.materialize.environment_id.clone(),
                 system_parameter_defaults,
-                version,
+                build_info,
                 bootstrap_args,
                 enable_expression_cache_override,
             )
diff --git a/src/testdrive/src/action/consistency.rs b/src/testdrive/src/action/consistency.rs
index f366874eed04b..91c25082a67f6 100644
--- a/src/testdrive/src/action/consistency.rs
+++ b/src/testdrive/src/action/consistency.rs
@@ -185,11 +185,14 @@ async fn check_catalog_state(state: &State) -> Result<(), anyhow::Error> {
         .and_then(|storage_metadata| storage_metadata.unfinalized_shards);
 
     // Load the on-disk catalog and dump its state.
-    let version: semver::Version = state.build_info.version.parse().expect("invalid version");
+
+    // Make sure the version is parseable.
+    let _: semver::Version = state.build_info.version.parse().expect("invalid version");
+
     let maybe_disk_catalog = state
         .with_catalog_copy(
             system_parameter_defaults,
-            version,
+            state.build_info,
             &state.materialize.bootstrap_args,
             // The expression cache can be taxing on the CPU and is unnecessary for consistency checks.
             Some(false),
diff --git a/test/sqllogictest/autogenerated/mz_internal.slt b/test/sqllogictest/autogenerated/mz_internal.slt
index c66673407250a..58f6b78c81fb1 100644
--- a/test/sqllogictest/autogenerated/mz_internal.slt
+++ b/test/sqllogictest/autogenerated/mz_internal.slt
@@ -581,6 +581,7 @@ SELECT position, name, type FROM objects WHERE schema = 'mz_internal' AND object
 3  status  text
 4  error  text
 5  details  jsonb
+6  replica_id  text
 
 query ITT
 SELECT position, name, type FROM objects WHERE schema = 'mz_internal' AND object = 'mz_source_statistics' ORDER BY position
@@ -618,6 +619,7 @@ SELECT position, name, type FROM objects WHERE schema = 'mz_internal' AND object
 3  status  text
 4  error  text
 5  details  jsonb
+6  replica_id  text
 
 query ITT
 SELECT position, name, type FROM objects WHERE schema = 'mz_internal' AND object = 'mz_statement_lifecycle_history' ORDER BY position
diff --git a/test/sqllogictest/distinct_arrangements.slt b/test/sqllogictest/distinct_arrangements.slt
index dacc675690e1c..71b7276ba1d7e 100644
--- a/test/sqllogictest/distinct_arrangements.slt
+++ b/test/sqllogictest/distinct_arrangements.slt
@@ -1030,7 +1030,7 @@ ArrangeBy[[]]  11
 Arranged␠DistinctBy  47
 Arranged␠MinsMaxesHierarchical␠input  14
 Arranged␠ReduceInaccumulable  3
-Arranged␠TopK␠input  68
+Arranged␠TopK␠input  92
 Distinct␠recursive␠err  4
 DistinctBy  47
 DistinctByErrorCheck  47
@@ -1040,7 +1040,7 @@ ReduceInaccumulable␠Error␠Check  3
 ReduceMinsMaxes  3
 ReduceMinsMaxes␠Error␠Check  1
 Reduced␠Fallibly␠MinsMaxesHierarchical  14
-Reduced␠TopK␠input  68
+Reduced␠TopK␠input  92
 Threshold␠local  10
 
 statement ok
diff --git a/test/sqllogictest/explain/optimized_plan_as_text.slt b/test/sqllogictest/explain/optimized_plan_as_text.slt
index 6ee78b5c0e93e..4bc07a86c95ab 100644
--- a/test/sqllogictest/explain/optimized_plan_as_text.slt
+++ b/test/sqllogictest/explain/optimized_plan_as_text.slt
@@ -139,7 +139,7 @@ query T multiline
 EXPLAIN OPTIMIZED PLAN AS VERBOSE TEXT FOR SELECT * FROM mz_internal.mz_source_status_history
 ----
 Explained Query (fast path):
-  Project (#1, #0, #2..=#4)
+  Project (#1, #0, #2..=#5)
     ReadIndex on=mz_internal.mz_source_status_history mz_source_status_history_ind=[*** full scan ***]
 
 Used Indexes:
diff --git a/test/sqllogictest/explain/optimized_plan_as_text_redacted.slt b/test/sqllogictest/explain/optimized_plan_as_text_redacted.slt
index c0543a9c308c3..7e3c44d81df4e 100644
--- a/test/sqllogictest/explain/optimized_plan_as_text_redacted.slt
+++ b/test/sqllogictest/explain/optimized_plan_as_text_redacted.slt
@@ -87,7 +87,7 @@ query T multiline
 EXPLAIN OPTIMIZED PLAN AS VERBOSE TEXT FOR SELECT * FROM mz_internal.mz_source_status_history
 ----
 Explained Query (fast path):
-  Project (#1, #0, #2..=#4)
+  Project (#1, #0, #2..=#5)
     ReadIndex on=mz_internal.mz_source_status_history mz_source_status_history_ind=[*** full scan ***]
 
 Used Indexes:
diff --git a/test/sqllogictest/mz_catalog_server_index_accounting.slt b/test/sqllogictest/mz_catalog_server_index_accounting.slt
index 8eafe768c33b9..8f7fce225632c 100644
--- a/test/sqllogictest/mz_catalog_server_index_accounting.slt
+++ b/test/sqllogictest/mz_catalog_server_index_accounting.slt
@@ -652,6 +652,7 @@ mz_sink_statistics_raw  messages_staged
 mz_sink_status_history  details
 mz_sink_status_history  error
 mz_sink_status_history  occurred_at
+mz_sink_status_history  replica_id
 mz_sink_status_history  sink_id
 mz_sink_status_history  status
 mz_sink_statuses  details
@@ -718,6 +719,7 @@ mz_source_statistics_with_history  updates_staged
 mz_source_status_history  details
 mz_source_status_history  error
 mz_source_status_history  occurred_at
+mz_source_status_history  replica_id
 mz_source_status_history  source_id
 mz_source_status_history  status
 mz_source_statuses  details
diff --git a/test/sqllogictest/system-cluster.slt b/test/sqllogictest/system-cluster.slt
index aabdd336fc95f..315e8979ba862 100644
--- a/test/sqllogictest/system-cluster.slt
+++ b/test/sqllogictest/system-cluster.slt
@@ -497,7 +497,7 @@ Explained Query:
     With
       cte l0 =
         Project (#0, #3)
-          Filter (#6 <= 100) AND (#6 >= 0) AND (#3) IS NOT NULL
+          Filter (#7 <= 100) AND (#7 >= 0) AND (#3) IS NOT NULL
             Map (timestamp_tz_to_mz_timestamp(#0))
               ReadIndex on=mz_internal.mz_source_status_history mz_source_status_history_ind=[lookup value=("u6")]
     Return
diff --git a/test/testdrive-old-kafka-src-syntax/mzcompose.py b/test/testdrive-old-kafka-src-syntax/mzcompose.py
index 1e2cb9fb913b9..9960e7674977e 100644
--- a/test/testdrive-old-kafka-src-syntax/mzcompose.py
+++ b/test/testdrive-old-kafka-src-syntax/mzcompose.py
@@ -301,7 +301,16 @@ def workflow_migration(c: Composition, parser: WorkflowArgumentParser) -> None:
                 filter, root_dir=MZ_ROOT / "test" / "testdrive-old-kafka-src-syntax"
             )
         )
-    matching_files = [file for file in matching_files if file != "session.td"]
+
+    # Exclude status-history.td because we added the replica_id column. This a)
+    # makes it so we lose history across migrations and b) makes it hard to
+    # write a test because the column is present in the newest version but not
+    # older versions.
+    matching_files = [
+        file
+        for file in matching_files
+        if file != "session.td" and file != "status-history.td"
+    ]
 
     dependencies = [
         "fivetran-destination",
diff --git a/test/testdrive-old-kafka-src-syntax/status-history.td b/test/testdrive-old-kafka-src-syntax/status-history.td
index 187440f83410e..600ea4342c2f2 100644
--- a/test/testdrive-old-kafka-src-syntax/status-history.td
+++ b/test/testdrive-old-kafka-src-syntax/status-history.td
@@ -22,22 +22,25 @@ $ set-from-sql var=load_id
 SELECT id FROM mz_sources WHERE name = 'counter'
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 1;
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > ALTER CLUSTER c SET (REPLICATION FACTOR 1)
 
+$ set-from-sql var=replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'c'
+
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${load_id} running <null> <null>
-"<TIMESTAMP> UTC" ${load_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} running <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} starting <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > ALTER CLUSTER c SET (REPLICATION FACTOR 0)
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 4;
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
-"<TIMESTAMP> UTC" ${load_id} running <null> <null>
-"<TIMESTAMP> UTC" ${load_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"The replica running this source has been dropped\"]}"  ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} running <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} starting <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > DROP CLUSTER c CASCADE
 
@@ -71,9 +74,15 @@ $ kafka-create-topic topic=status-history
 $ set-from-sql var=source_id
 SELECT id FROM mz_sources WHERE name = 'kafka_source'
 
+$ set-from-sql var=source_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_source_cluster'
+
+$ set-from-sql var=sink_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_sink_cluster'
+
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_statuses WHERE id = '${source_id}';
 "${source_id}" kafka_source kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -85,7 +94,7 @@ SELECT id FROM mz_sinks WHERE name = 'kafka_sink'
 # Additionally it can happen that both 'starting' and 'running' are reported on the same millisecond
 # so we filter out any other statuses.
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' AND status = 'starting' ORDER BY occurred_at ASC LIMIT 1;
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 $ kafka-ingest format=bytes topic=status-history
 a
@@ -106,15 +115,15 @@ $ kafka-verify-data format=avro sink=materialize.public.kafka_sink sort-messages
 {"before": null, "after": {"row":{"text": "d"}}}
 
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${sink_id} running <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} running <null> <null> ${sink_replica_id}
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 > SELECT * FROM mz_internal.mz_sink_statuses WHERE id = '${sink_id}';
 "${sink_id}" kafka_sink kafka "<TIMESTAMP> UTC" running <null> <null>
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_statuses WHERE id = '${source_id}';
 "${source_id}" kafka_source kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -141,13 +150,19 @@ SELECT id FROM mz_sources WHERE name = 'kafka_source_2'
 $ set-from-sql var=sink_id_2
 SELECT id FROM mz_sinks WHERE name = 'kafka_sink_2'
 
+$ set-from-sql var=source_2_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_source_2_cluster'
+
+$ set-from-sql var=sink_2_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_sink_2_cluster'
+
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id_2}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${sink_id_2} starting <null> <null>
-"<TIMESTAMP> UTC" ${sink_id_2} running <null> <null>
+"<TIMESTAMP> UTC" ${sink_id_2} starting <null> <null> ${sink_2_replica_id}
+"<TIMESTAMP> UTC" ${sink_id_2} running <null> <null> ${sink_2_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id_2}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id_2} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id_2} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id_2} running <null> <null> ${source_2_replica_id}
+"<TIMESTAMP> UTC" ${source_id_2} starting <null> <null> ${source_2_replica_id}
 
 > SELECT * FROM mz_internal.mz_sink_statuses WHERE id IN ('${sink_id}', '${sink_id_2}') ORDER BY id;
 "${sink_id}" kafka_sink kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -162,13 +177,13 @@ SELECT id FROM mz_sinks WHERE name = 'kafka_sink_2'
 > DROP SINK kafka_sink
 
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${sink_id} dropped <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} running <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} dropped <null> <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} running <null> <null> ${sink_replica_id}
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 > DROP SOURCE kafka_source CASCADE
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${source_id} dropped <null> <null>
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id} dropped <null> <null> <null>
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}
diff --git a/test/testdrive/kafka-recreate-topic.td b/test/testdrive/kafka-recreate-topic.td
index 0d3dbb47457df..acacb3c74c58e 100644
--- a/test/testdrive/kafka-recreate-topic.td
+++ b/test/testdrive/kafka-recreate-topic.td
@@ -121,6 +121,29 @@ $ kafka-create-topic topic=good-topic
   FORMAT TEXT
   ENVELOPE NONE
 
+# After suspending the cluster (by scaling the replication factor to 0), the
+# latest state is paused for sources that previously reported as stalled.
+# Because their last state _is_ in fact paused. Errored sources don't restart,
+# so they will not go through the source lifecycle on restarting the replica,
+# so they remain paused.
+
+# Before suspending the cluster, we see the errors in the status collection.
+# For source0 and source1, we already went through a couple suspension cycles,
+# so they already report as paused. With source2 online, we didn't yet do a
+# suspend-restart cycle.
+
+> SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE type != 'progress'
+name            status    error
+-------------------------------
+good_source     running   <null>
+good_source_tbl running   <null>
+source0         paused    <null>
+source0_tbl     paused    <null>
+source1         paused    <null>
+source1_tbl     paused    <null>
+source2         stalled   "kafka: Source error: source must be dropped and recreated due to failure: topic was recreated"
+source2_tbl     stalled   "kafka: Source error: source must be dropped and recreated due to failure: topic was recreated"
+
 > ALTER CLUSTER to_recreate SET (REPLICATION FACTOR 0)
 > ALTER CLUSTER to_recreate SET (REPLICATION FACTOR 1)
 
@@ -132,17 +155,19 @@ text
 ----
 1
 
+# After another suspend-restart cycle, source2 also reports as paused.
+
 > SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE type != 'progress'
 name            status    error
 -------------------------------
 good_source     running   <null>
 good_source_tbl running   <null>
 source0         paused    <null>
-source0_tbl     stalled   "kafka: Source error: source must be dropped and recreated due to failure: topic was deleted"
+source0_tbl     paused    <null>
 source1         paused    <null>
-source1_tbl     stalled   "kafka: Source error: source must be dropped and recreated due to failure: topic was recreated"
+source1_tbl     paused    <null>
 source2         paused    <null>
-source2_tbl     stalled   "kafka: Source error: source must be dropped and recreated due to failure: topic was recreated"
+source2_tbl     paused    <null>
 
 # Testdrive expects all sources to end in a healthy state, so manufacture that
 # by dropping sources.
diff --git a/test/testdrive/status-history.td b/test/testdrive/status-history.td
index adbbfe4f52011..ff89baa4610c9 100644
--- a/test/testdrive/status-history.td
+++ b/test/testdrive/status-history.td
@@ -22,22 +22,25 @@ $ set-from-sql var=load_id
 SELECT id FROM mz_sources WHERE name = 'counter'
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 1;
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > ALTER CLUSTER c SET (REPLICATION FACTOR 1)
 
+$ set-from-sql var=replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'c'
+
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${load_id} running <null> <null>
-"<TIMESTAMP> UTC" ${load_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} running <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} starting <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > ALTER CLUSTER c SET (REPLICATION FACTOR 0)
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${load_id}' ORDER BY occurred_at DESC LIMIT 4;
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
-"<TIMESTAMP> UTC" ${load_id} running <null> <null>
-"<TIMESTAMP> UTC" ${load_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}"
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"The replica running this source has been dropped\"]}"  ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} running <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} starting <null> <null> ${replica_id}
+"<TIMESTAMP> UTC" ${load_id} paused <null> "{\"hints\":[\"There is currently no replica running this source\"]}" <null>
 
 > DROP CLUSTER c CASCADE
 
@@ -73,9 +76,15 @@ $ kafka-create-topic topic=status-history
 $ set-from-sql var=source_id
 SELECT id FROM mz_sources WHERE name = 'kafka_source'
 
+$ set-from-sql var=source_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_source_cluster'
+
+$ set-from-sql var=sink_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_sink_cluster'
+
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_statuses WHERE id = '${source_id}';
 "${source_id}" kafka_source kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -87,7 +96,7 @@ SELECT id FROM mz_sinks WHERE name = 'kafka_sink'
 # Additionally it can happen that both 'starting' and 'running' are reported on the same millisecond
 # so we filter out any other statuses.
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' AND status = 'starting' ORDER BY occurred_at ASC LIMIT 1;
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 $ kafka-ingest format=bytes topic=status-history
 a
@@ -108,15 +117,15 @@ $ kafka-verify-data format=avro sink=materialize.public.kafka_sink sort-messages
 {"before": null, "after": {"row":{"text": "d"}}}
 
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${sink_id} running <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} running <null> <null> ${sink_replica_id}
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 > SELECT * FROM mz_internal.mz_sink_statuses WHERE id = '${sink_id}';
 "${sink_id}" kafka_sink kafka "<TIMESTAMP> UTC" running <null> <null>
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_statuses WHERE id = '${source_id}';
 "${source_id}" kafka_source kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -145,13 +154,19 @@ SELECT id FROM mz_sources WHERE name = 'kafka_source_2'
 $ set-from-sql var=sink_id_2
 SELECT id FROM mz_sinks WHERE name = 'kafka_sink_2'
 
+$ set-from-sql var=source_2_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_source_2_cluster'
+
+$ set-from-sql var=sink_2_replica_id
+SELECT r.id FROM mz_clusters c JOIN mz_cluster_replicas r ON c.id = r.cluster_id  WHERE c.name = 'kafka_sink_2_cluster'
+
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id_2}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${sink_id_2} starting <null> <null>
-"<TIMESTAMP> UTC" ${sink_id_2} running <null> <null>
+"<TIMESTAMP> UTC" ${sink_id_2} starting <null> <null> ${sink_2_replica_id}
+"<TIMESTAMP> UTC" ${sink_id_2} running <null> <null> ${sink_2_replica_id}
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id_2}' ORDER BY occurred_at DESC LIMIT 2;
-"<TIMESTAMP> UTC" ${source_id_2} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id_2} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id_2} running <null> <null> ${source_2_replica_id}
+"<TIMESTAMP> UTC" ${source_id_2} starting <null> <null> ${source_2_replica_id}
 
 > SELECT * FROM mz_internal.mz_sink_statuses WHERE id IN ('${sink_id}', '${sink_id_2}') ORDER BY id;
 "${sink_id}" kafka_sink kafka "<TIMESTAMP> UTC" running <null> <null>
@@ -166,13 +181,13 @@ SELECT id FROM mz_sinks WHERE name = 'kafka_sink_2'
 > DROP SINK kafka_sink
 
 > SELECT * FROM mz_internal.mz_sink_status_history WHERE sink_id = '${sink_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${sink_id} dropped <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} running <null> <null>
-"<TIMESTAMP> UTC" ${sink_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} dropped <null> <null> <null>
+"<TIMESTAMP> UTC" ${sink_id} running <null> <null> ${sink_replica_id}
+"<TIMESTAMP> UTC" ${sink_id} starting <null> <null> ${sink_replica_id}
 
 > DROP SOURCE kafka_source CASCADE
 
 > SELECT * FROM mz_internal.mz_source_status_history WHERE source_id = '${source_id}' ORDER BY occurred_at DESC LIMIT 3;
-"<TIMESTAMP> UTC" ${source_id} dropped <null> <null>
-"<TIMESTAMP> UTC" ${source_id} running <null> <null>
-"<TIMESTAMP> UTC" ${source_id} starting <null> <null>
+"<TIMESTAMP> UTC" ${source_id} dropped <null> <null> <null>
+"<TIMESTAMP> UTC" ${source_id} running <null> <null> ${source_replica_id}
+"<TIMESTAMP> UTC" ${source_id} starting <null> <null> ${source_replica_id}


Reviews:
user: login: bkirwi
body: Hmm. I think the semantics of the status stuff have become a bit of a mess, and this makes the situation rather more complex. But also I appreciate the argument that there's no way to get a global picture of the system without replica information here, given the current design...

Anyways I'm fine with going ahead on this for now, though it'd be good to find a little time eventually to hash out how we want this stuff to work...
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T20:14:24Z
user: login: aljoscha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T13:26:11Z
user: login: aljoscha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T13:27:38Z
user: login: aljoscha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T13:34:30Z
user: login: bkirwi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:08:33Z
user: login: bkirwi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:10:02Z
user: login: aljoscha
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T20:36:58Z

Review comments:
diff_hunk: @@ -3210,13 +3210,55 @@ pub static MZ_SOURCE_STATUSES: LazyLock<BuiltinView> = LazyLock::new(|| BuiltinV
     column_defs: None,
     sql: "
     WITH
-    -- Get the latest events
+    -- The status history contains per-replica events and source-global events.
+    -- For the latter, replica_id is NULL. We turn these into '<source>', so that
+    -- we can treat them uniformly below.
+    uniform_status_history AS
+    (
+        SELECT
+            s.source_id,
+            CASE s.replica_id
+                WHEN NULL THEN '<source>'
+                ELSE s.replica_id
+            END,
path: src/catalog/src/builtin.rs
user: login: bkirwi
body: This is equivalent to a `coalesce(s.replica_id, 'source')`, yeah?
created_at: 2025-03-12T19:42:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -180,8 +193,24 @@ where
         for command in self.history.iter() {
             match command {
                 StorageCommand::RunIngestions(cmds) => {
-                    let updates = cmds.iter().map(|c| make_update(c.id, "source"));
-                    status_updates.extend(updates);
+                    for ingestion in cmds.iter() {
+                        // NOTE(aljoscha): We filter out the remap collection because we
+                        // don't get any status updates about it from the replica side. So
+                        // we don't want to synthesize a 'paused' status here.
+                        //
+                        // TODO(aljoscha): I think we want to fix this eventually, and make
+                        // sure we get status updates for the remap shard as well. Currently
+                        // its handling in the source status collection is a bit difficult
+                        // because we don't have updates for it in the status history
+                        // collection.
path: src/storage-controller/src/instance.rs
user: login: bkirwi
body: (FWIW I've always thought reporting statuses for individual source tables was pretty suspect, so I think this is fine.)
created_at: 2025-03-12T19:56:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1048,6 +1048,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the sink: one of `created`, `starting`, `running`, `stalled`, `failed`, or `dropped`.              |
path: doc/user/content/sql/system-catalog/mz_internal.md
user: login: bkirwi
body: Is this list still up to date?
created_at: 2025-03-12T19:58:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -516,10 +516,66 @@ where
             .get_mut(&instance_id)
             .unwrap_or_else(|| panic!("instance {instance_id} does not exist"));
 
+        let status_now = mz_ore::now::to_datetime((self.now)());
+        let mut source_status_updates = vec![];
+        let mut sink_status_updates = vec![];
+
+        let make_update = |id, object_type| StatusUpdate {
+            id,
+            status: Status::Paused,
+            timestamp: status_now,
+            error: None,
+            hints: BTreeSet::from([format!(
+                "The replica running this {object_type} has been dropped"
+            )]),
+            namespaced_errors: Default::default(),
+            replica_id: Some(replica_id),
+        };
+
         for id in instance.active_ingestions() {
+            let ingestion = self
+                .collections
+                .get_mut(id)
+                .expect("instance contains unknown ingestion");
path: src/storage-controller/src/lib.rs
user: login: bkirwi
body: Are these two states guaranteed to be updated transactionally? (A log and `continue` here seems less scary if we're not totally sure.) 
created_at: 2025-03-12T20:01:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3210,13 +3210,55 @@ pub static MZ_SOURCE_STATUSES: LazyLock<BuiltinView> = LazyLock::new(|| BuiltinV
     column_defs: None,
     sql: "
     WITH
-    -- Get the latest events
+    -- The status history contains per-replica events and source-global events.
+    -- For the latter, replica_id is NULL. We turn these into '<source>', so that
+    -- we can treat them uniformly below.
+    uniform_status_history AS
+    (
+        SELECT
+            s.source_id,
+            CASE s.replica_id
+                WHEN NULL THEN '<source>'
+                ELSE s.replica_id
+            END,
path: src/catalog/src/builtin.rs
user: login: aljoscha
body: hehe, yes, and I've even used that in this here commit! 🙈 
created_at: 2025-03-13T13:26:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1048,6 +1048,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the sink: one of `created`, `starting`, `running`, `stalled`, `failed`, or `dropped`.              |
path: doc/user/content/sql/system-catalog/mz_internal.md
user: login: aljoscha
body: I think it's not, if this is the list: https://github.com/MaterializeInc/materialize/blob/cc15d88f1159aeb6de8246e09fd7ddfccc77508f/src/storage-client/src/client.rs#L420
created_at: 2025-03-13T13:27:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -516,10 +516,66 @@ where
             .get_mut(&instance_id)
             .unwrap_or_else(|| panic!("instance {instance_id} does not exist"));
 
+        let status_now = mz_ore::now::to_datetime((self.now)());
+        let mut source_status_updates = vec![];
+        let mut sink_status_updates = vec![];
+
+        let make_update = |id, object_type| StatusUpdate {
+            id,
+            status: Status::Paused,
+            timestamp: status_now,
+            error: None,
+            hints: BTreeSet::from([format!(
+                "The replica running this {object_type} has been dropped"
+            )]),
+            namespaced_errors: Default::default(),
+            replica_id: Some(replica_id),
+        };
+
         for id in instance.active_ingestions() {
+            let ingestion = self
+                .collections
+                .get_mut(id)
+                .expect("instance contains unknown ingestion");
path: src/storage-controller/src/lib.rs
user: login: aljoscha
body: Not transactionally, but these parts can't have concurrency (at least today). We've had this `expect()` here for a long time, so I think it's okay. But I can change this and the other one to a `soft_assert_or_log` as a follow-up, if you like!
created_at: 2025-03-13T13:34:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1048,6 +1048,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the sink: one of `created`, `starting`, `running`, `stalled`, `failed`, or `dropped`.              |
path: doc/user/content/sql/system-catalog/mz_internal.md
user: login: bkirwi
body: Rereading, I think the possible reported statuses don't change in this PR, so it's fine to leave this for now. Something for the future holistic revisit possibly!
created_at: 2025-03-13T17:08:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -516,10 +516,66 @@ where
             .get_mut(&instance_id)
             .unwrap_or_else(|| panic!("instance {instance_id} does not exist"));
 
+        let status_now = mz_ore::now::to_datetime((self.now)());
+        let mut source_status_updates = vec![];
+        let mut sink_status_updates = vec![];
+
+        let make_update = |id, object_type| StatusUpdate {
+            id,
+            status: Status::Paused,
+            timestamp: status_now,
+            error: None,
+            hints: BTreeSet::from([format!(
+                "The replica running this {object_type} has been dropped"
+            )]),
+            namespaced_errors: Default::default(),
+            replica_id: Some(replica_id),
+        };
+
         for id in instance.active_ingestions() {
+            let ingestion = self
+                .collections
+                .get_mut(id)
+                .expect("instance contains unknown ingestion");
path: src/storage-controller/src/lib.rs
user: login: bkirwi
body: Mmm... f this is just code movement, I think it's AOK to leave it alone. 
created_at: 2025-03-13T17:10:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1048,6 +1048,7 @@ messages and additional metadata helpful for debugging.
 | `status`       | [`text`]                        | The status of the sink: one of `created`, `starting`, `running`, `stalled`, `failed`, or `dropped`.              |
path: doc/user/content/sql/system-catalog/mz_internal.md
user: login: aljoscha
body: Yep, sorry, I should have explicitly responded that I don't think the PR changes these 👌 
created_at: 2025-03-14T20:36:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: aljoscha
created_at: 2025-03-11T19:26:55Z
body: @bkirwi You're probably not familiar with these parts, but I think no-one is. The closest other options are @teskje and @petrosagg who are both out.

I did check with @sjwiesman and @SangJunBak that adding this new column is okay, especially since it means we do a migration, which wipes out all history. This is especially fine, though, since we prune that history anyways and only keep the latest 5 updates.

Nightlies are currently failing, but I think fixing those is similar to fixing the other tests that I already touched.
author_association: CONTRIBUTOR
user: login: SangJunBak
created_at: 2025-03-11T19:32:32Z
body: @aljoscha Do you have this change in a staging environment by chance? Or even locally? Can easily validate this change to see if breaks anything in the Console if so
author_association: CONTRIBUTOR
user: login: bkirwi
created_at: 2025-03-11T22:42:25Z
body: I will take a look!
author_association: CONTRIBUTOR
user: login: aljoscha
created_at: 2025-03-12T08:56:17Z
body: > @aljoscha Do you have this change in a staging environment by chance? Or even locally? Can easily validate this change to see if breaks anything in the Console if so

Excellent suggestion! I did now spin it up in my staging env, and everything seems to be working. At least the parts I know to check.
author_association: CONTRIBUTOR
user: login: aljoscha
created_at: 2025-03-13T13:50:41Z
body: @bkirwi Thanks for the pragmatic review!

I agree that the semantics of these error collections (and other things around subsources) require a holistic re-think, but I also think that my changes are at least not making it worse (well, the code is more complex, yes) and are actually fixing some things to make status reporting more uniform across the main source and subsources.
author_association: CONTRIBUTOR
user: login: bkirwi
created_at: 2025-03-13T17:10:08Z
body: Yeah, sounds like we're on the same page!
author_association: CONTRIBUTOR