Repository: zio/zio-kafka
Pull request #1452: Preparing for producer diagnostics (3.x)
Author: login: erikvanoosten
Created: 2025-02-05T10:37:39Z | Merged: 2025-03-23T11:19:08Z

It has been a long wish to have producer diagnostics. Although these are not really hard to add, the consumer diagnostics are a bit in the way; these already claim some names and make code sharing awkward. Now that we are moving towards zio-kafka 3.0 we can fix these at the cost of some backward incompatibilities.

This PR has the same changes as #1507 (which will be applied to the series/2.x branch) but then with all deprecated code removed.

Partially fixes #1406.

Linked issue #1406: Producer diagnostics
With `ConsumerDiagnostics` users can listen to important events in the consumer. We would like to have something very similar for the producer.

A non-exhaustive list of events that should be supported:
* record is offered for sending (topic, partition, size of record)
* batch is offered for sending (number of records, total size of records)
* record was sent (topic, partition, size of record)

See also #1039.

Diff:
diff --git a/docs/creating-a-consumer.md b/docs/creating-a-consumer.md
index 3714f8fab..2ce43592f 100644
--- a/docs/creating-a-consumer.md
+++ b/docs/creating-a-consumer.md
@@ -67,7 +67,7 @@ The `provide` section of the main program could look like this:
 ```scala
 .provide(
   consumerSettingsLayer,
-  ZLayer.succeed(Diagnostics.NoOp),
+  ZLayer.succeed(Consumer.NoDiagnostics),
   Consumer.live
 )
 ```
diff --git a/docs/transactions.md b/docs/transactions.md
index b69f0fe23..ddd4fc862 100644
--- a/docs/transactions.md
+++ b/docs/transactions.md
@@ -171,7 +171,6 @@ Putting it all together we get:
 ```scala
 import org.apache.kafka.clients.producer.ProducerRecord
 import zio._
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer._
 import zio.kafka.producer._
 import zio.kafka.serde.Serde
@@ -240,7 +239,7 @@ object Transactional extends ZIOAppDefault {
     runConsumerStream
       .provide(
         consumerSettings,
-        ZLayer.succeed(Diagnostics.NoOp),
+        ZLayer.succeed(Consumer.NoDiagnostics),
         Consumer.live,
         producerSettings,
         TransactionalProducer.live
diff --git a/project/MimaSettings.scala b/project/MimaSettings.scala
index 71cdbafe6..824eb3ce7 100644
--- a/project/MimaSettings.scala
+++ b/project/MimaSettings.scala
@@ -14,7 +14,8 @@ object MimaSettings {
         Seq(
           mimaPreviousArtifacts := Set(organization.value %% name.value % binCompatVersion),
           mimaBinaryIssueFilters ++= Seq(
-            exclude[Problem]("zio.kafka.consumer.internal.*")
+            exclude[Problem]("zio.kafka.consumer.internal.*"),
+            exclude[Problem]("zio.kafka.diagnostics.internal.*")
           ),
           mimaFailOnProblem := failOnProblem
         )
diff --git a/zio-kafka-bench/src/main/scala/zio/kafka/bench/ZioKafkaConsumerBenchmark.scala b/zio-kafka-bench/src/main/scala/zio/kafka/bench/ZioKafkaConsumerBenchmark.scala
index 8f05d7986..5b0fcbc48 100644
--- a/zio-kafka-bench/src/main/scala/zio/kafka/bench/ZioKafkaConsumerBenchmark.scala
+++ b/zio-kafka-bench/src/main/scala/zio/kafka/bench/ZioKafkaConsumerBenchmark.scala
@@ -3,8 +3,8 @@ package zio.kafka.bench
 import org.openjdk.jmh.annotations._
 import zio.kafka.admin.AdminClient.NewTopic
 import zio.kafka.bench.ZioBenchmark.randomThing
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer.{ Consumer, Offset, OffsetBatch, Subscription }
+import zio.kafka.diagnostics.Diagnostics
 import zio.kafka.serde.Serde
 import zio.kafka.testkit.Kafka
 import zio.kafka.testkit.KafkaTestUtils
diff --git a/zio-kafka-example/src/main/scala/zio/kafka/example/Main.scala b/zio-kafka-example/src/main/scala/zio/kafka/example/Main.scala
index a18fe6230..14b8a1765 100644
--- a/zio-kafka-example/src/main/scala/zio/kafka/example/Main.scala
+++ b/zio-kafka-example/src/main/scala/zio/kafka/example/Main.scala
@@ -1,7 +1,6 @@
 package zio.kafka.example
 
 import zio._
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer.{ Consumer, ConsumerSettings, Subscription }
 import zio.kafka.serde.Serde
 import zio.logging.backend.SLF4J
@@ -42,7 +41,7 @@ object Main extends ZIOAppDefault {
       runConsumerStream
         .provide(
           consumerSettings,
-          ZLayer.succeed(Diagnostics.NoOp),
+          ZLayer.succeed(Consumer.NoDiagnostics),
           Consumer.live,
           MyKafka.embedded
         )
diff --git a/zio-kafka-example/src/main/scala/zio/kafka/example/Transactional.scala b/zio-kafka-example/src/main/scala/zio/kafka/example/Transactional.scala
index 8e0204b89..88d315af5 100644
--- a/zio-kafka-example/src/main/scala/zio/kafka/example/Transactional.scala
+++ b/zio-kafka-example/src/main/scala/zio/kafka/example/Transactional.scala
@@ -2,7 +2,6 @@ package zio.kafka.example
 
 import org.apache.kafka.clients.producer.ProducerRecord
 import zio._
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer._
 import zio.kafka.producer._
 import zio.kafka.serde.Serde
@@ -82,7 +81,7 @@ object Transactional extends ZIOAppDefault {
       runConsumerStream
         .provide(
           consumerSettings,
-          ZLayer.succeed(Diagnostics.NoOp),
+          ZLayer.succeed(Consumer.NoDiagnostics),
           Consumer.live,
           producerSettings,
           TransactionalProducer.live,
diff --git a/zio-kafka-test/src/test/scala/zio/kafka/consumer/ConsumerSpec.scala b/zio-kafka-test/src/test/scala/zio/kafka/consumer/ConsumerSpec.scala
index f53f17aa5..a457e2a84 100644
--- a/zio-kafka-test/src/test/scala/zio/kafka/consumer/ConsumerSpec.scala
+++ b/zio-kafka-test/src/test/scala/zio/kafka/consumer/ConsumerSpec.scala
@@ -10,14 +10,10 @@ import org.apache.kafka.clients.producer.ProducerRecord
 import org.apache.kafka.common.TopicPartition
 import zio._
 import zio.kafka.ZIOSpecDefaultSlf4j
-import zio.kafka.consumer.Consumer.{ AutoOffsetStrategy, CommitTimeout, OffsetRetrieval }
-import zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization
-import zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization.{
-  ConsumerFinalized,
-  RunloopFinalized,
-  SubscriptionFinalized
-}
-import zio.kafka.consumer.diagnostics.{ DiagnosticEvent, Diagnostics }
+import zio.kafka.consumer.Consumer.{ AutoOffsetStrategy, CommitTimeout, ConsumerDiagnostics, OffsetRetrieval }
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
+import zio.kafka.consumer.diagnostics.DiagnosticEvent.{ ConsumerFinalized, RunloopFinalized, SubscriptionFinalized }
+import zio.kafka.diagnostics.{ Diagnostics, SlidingDiagnostics }
 import zio.kafka.producer.TransactionalProducer
 import zio.kafka.serde.Serde
 import zio.kafka.testkit.{ Kafka, KafkaRandom, KafkaTestUtils }
@@ -648,64 +644,62 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
         val partitionCount = 6
 
         ZIO.scoped {
-          Diagnostics.SlidingQueue
-            .make()
-            .flatMap { diagnostics =>
-              for {
-                // Produce messages on several partitions
-                topic   <- randomTopic
-                group   <- randomGroup
-                client1 <- randomClient
-                client2 <- randomClient
+          SlidingDiagnostics.make[DiagnosticEvent]().flatMap { diagnostics =>
+            for {
+              // Produce messages on several partitions
+              topic   <- randomTopic
+              group   <- randomGroup
+              client1 <- randomClient
+              client2 <- randomClient
 
-                _        <- KafkaTestUtils.createCustomTopic(topic, partitionCount)
-                producer <- KafkaTestUtils.makeProducer
-                _ <- ZIO.foreachDiscard(1 to nrMessages) { i =>
-                       KafkaTestUtils
-                         .produceMany(producer, topic, partition = i % partitionCount, kvs = List(s"key$i" -> s"msg$i"))
-                     }
-
-                consumer1 <- KafkaTestUtils.makeConsumer(client1, Some(group), diagnostics = diagnostics)
-                consumer2 <- KafkaTestUtils.makeConsumer(client2, Some(group))
-
-                // Consume messages
-                subscription = Subscription.topics(topic)
-                consumer1Ready        <- Promise.make[Nothing, Unit]
-                assignedPartitionsRef <- Ref.make(Set.empty[Int]) // Set of partition numbers
-                consumer1Fib <- consumer1
-                                  .partitionedStream(subscription, Serde.string, Serde.string)
-                                  .flatMapPar(partitionCount) { case (tp, partition) =>
-                                    ZStream
-                                      .fromZIO(
-                                        consumer1Ready
-                                          .succeed(())
-                                          .whenZIO(
-                                            assignedPartitionsRef
-                                              .updateAndGet(_ + tp.partition())
-                                              .map(_.size >= (partitionCount / 2))
-                                          ) *>
-                                          partition.runDrain
-                                      )
-                                      .as(tp)
-                                  }
-                                  .take(partitionCount.toLong / 2)
-                                  .runDrain
-                                  .fork
-                diagnosticStream <- ZStream
-                                      .fromQueue(diagnostics.queue)
-                                      .collect { case rebalance: DiagnosticEvent.Rebalance => rebalance }
-                                      .runCollect
-                                      .fork
-                _ <- consumer1Ready.await
-                consumer2Fib <- consumer2
-                                  .partitionedStream(subscription, Serde.string, Serde.string)
-                                  .take(partitionCount.toLong / 2)
-                                  .runDrain
-                                  .fork
-                _ <- consumer1Fib.join
-                _ <- consumer2Fib.join
-              } yield diagnosticStream.join
-            }
+              _        <- KafkaTestUtils.createCustomTopic(topic, partitionCount)
+              producer <- KafkaTestUtils.makeProducer
+              _ <- ZIO.foreachDiscard(1 to nrMessages) { i =>
+                     KafkaTestUtils
+                       .produceMany(producer, topic, partition = i % partitionCount, kvs = List(s"key$i" -> s"msg$i"))
+                   }
+
+              consumer1 <- KafkaTestUtils.makeConsumer(client1, Some(group), diagnostics = diagnostics)
+              consumer2 <- KafkaTestUtils.makeConsumer(client2, Some(group))
+
+              // Consume messages
+              subscription = Subscription.topics(topic)
+              consumer1Ready        <- Promise.make[Nothing, Unit]
+              assignedPartitionsRef <- Ref.make(Set.empty[Int]) // Set of partition numbers
+              consumer1Fib <- consumer1
+                                .partitionedStream(subscription, Serde.string, Serde.string)
+                                .flatMapPar(partitionCount) { case (tp, partition) =>
+                                  ZStream
+                                    .fromZIO(
+                                      consumer1Ready
+                                        .succeed(())
+                                        .whenZIO(
+                                          assignedPartitionsRef
+                                            .updateAndGet(_ + tp.partition())
+                                            .map(_.size >= (partitionCount / 2))
+                                        ) *>
+                                        partition.runDrain
+                                    )
+                                    .as(tp)
+                                }
+                                .take(partitionCount.toLong / 2)
+                                .runDrain
+                                .fork
+              diagnosticStream <- ZStream
+                                    .fromQueue(diagnostics.queue)
+                                    .collect { case rebalance: DiagnosticEvent.Rebalance => rebalance }
+                                    .runCollect
+                                    .fork
+              _ <- consumer1Ready.await
+              consumer2Fib <- consumer2
+                                .partitionedStream(subscription, Serde.string, Serde.string)
+                                .take(partitionCount.toLong / 2)
+                                .runDrain
+                                .fork
+              _ <- consumer1Fib.join
+              _ <- consumer2Fib.join
+            } yield diagnosticStream.join
+          }
         }.flatten
           .map(diagnosticEvents => assert(diagnosticEvents.size)(isGreaterThanEqualTo(2)))
       },
@@ -1002,7 +996,7 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
           clientId: String,
           groupId: String,
           rebalanceSafeCommits: Boolean,
-          diagnostics: Diagnostics
+          diagnostics: ConsumerDiagnostics
         ): ZIO[Scope with Kafka, Throwable, Consumer] =
           for {
             settings <- KafkaTestUtils.consumerSettings(
@@ -1036,7 +1030,7 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
                  .fork
           _                       <- ZIO.logDebug("Starting consumer 1")
           rebalanceEndTimePromise <- Promise.make[Nothing, Instant]
-          c1Diagnostics = new Diagnostics {
+          c1Diagnostics = new ConsumerDiagnostics {
                             override def emit(event: => DiagnosticEvent): UIO[Unit] = event match {
                               case r: DiagnosticEvent.Rebalance if r.assigned.size == 1 =>
                                 ZIO.logDebug(s"Rebalance finished: $r") *>
@@ -1384,7 +1378,7 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
         test(
           "Booting a Consumer to do something else than consuming should not fail with a timeout exception"
         ) {
-          def test(diagnostics: Diagnostics) =
+          def test(diagnostics: ConsumerDiagnostics) =
             for {
               clientId <- randomClient
               settings <- KafkaTestUtils.consumerSettings(
@@ -1396,18 +1390,18 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
             } yield assertCompletes
 
           for {
-            diagnostics <- Diagnostics.SlidingQueue.make(1000)
+            diagnostics <- SlidingDiagnostics.make[DiagnosticEvent](1000)
             testResult <- ZIO.scoped {
                             test(diagnostics)
                           }
-            finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isInstanceOf[Finalization]))
+            finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isFinalizationEvent))
           } yield testResult && assert(finalizationEvents)(hasSameElements(Chunk(ConsumerFinalized)))
         },
         suite(
           "Ordering of finalizers matters. If subscriptions are finalized after the runloop, it creates a deadlock"
         )(
           test("When not consuming, the Runloop is not started so only the Consumer is finalized") {
-            def test(diagnostics: Diagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
+            def test(diagnostics: ConsumerDiagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
               for {
                 clientId <- randomClient
                 settings <- KafkaTestUtils.consumerSettings(clientId = clientId)
@@ -1415,17 +1409,17 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
               } yield assertCompletes
 
             for {
-              diagnostics <- Diagnostics.SlidingQueue.make(1000)
+              diagnostics <- SlidingDiagnostics.make[DiagnosticEvent](1000)
               testResult <- ZIO.scoped {
                               test(diagnostics)
                             }
-              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isInstanceOf[Finalization]))
+              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isFinalizationEvent))
             } yield testResult && assert(finalizationEvents)(hasSameElements(Chunk(ConsumerFinalized)))
           },
           test("When consuming, the Runloop is started. The finalization orders matters to avoid a deadlock") {
             // This test ensures that we're not inadvertently introducing a deadlock by changing the order of finalizers.
 
-            def test(diagnostics: Diagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
+            def test(diagnostics: ConsumerDiagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
               for {
                 clientId <- randomClient
                 topic    <- randomTopic
@@ -1447,11 +1441,11 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
               } yield assert(consumed0)(equalTo(1L)) && assert(consumed1)(equalTo(1L))
 
             for {
-              diagnostics <- Diagnostics.SlidingQueue.make(1000)
+              diagnostics <- SlidingDiagnostics.make[DiagnosticEvent](1000)
               testResult <- ZIO.scoped {
                               test(diagnostics)
                             }
-              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isInstanceOf[Finalization]))
+              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isFinalizationEvent))
             } yield testResult && assert(finalizationEvents)(
               // The order is very important.
               // The subscription must be finalized before the runloop, otherwise it creates a deadlock.
@@ -1472,7 +1466,7 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
             val messagesToConsumeBeforeStop     = 1000 // Adjust this threshold as needed
             val kvs: Iterable[(String, String)] = Iterable.tabulate(numberOfMessages)(i => (s"key-$i", s"msg-$i"))
 
-            def test(diagnostics: Diagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
+            def test(diagnostics: ConsumerDiagnostics): ZIO[Scope & Kafka, Throwable, TestResult] =
               for {
                 clientId <- randomClient
                 topic    <- randomTopic
@@ -1512,11 +1506,11 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
                 assert(consumed1)(equalTo(numberOfMessages.toLong))
 
             for {
-              diagnostics <- Diagnostics.SlidingQueue.make(1000)
+              diagnostics <- SlidingDiagnostics.make[DiagnosticEvent](1000)
               testResult <- ZIO.scoped {
                               test(diagnostics)
                             }
-              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isInstanceOf[Finalization]))
+              finalizationEvents <- diagnostics.queue.takeAll.map(_.filter(_.isFinalizationEvent))
             } yield testResult && assert(finalizationEvents)(
               // The order is very important.
               // The subscription must be finalized before the runloop, otherwise it creates a deadlock.
@@ -1557,4 +1551,13 @@ object ConsumerSpec extends ZIOSpecDefaultSlf4j with KafkaRandom {
         Kafka.embedded
       ) @@ withLiveClock @@ timeout(2.minutes) @@ TestAspect.timed
 
+  private final implicit class ConsumerDiagnosticsOps(private val event: DiagnosticEvent) extends AnyVal {
+    def isFinalizationEvent: Boolean = event match {
+      case SubscriptionFinalized => true
+      case RunloopFinalized      => true
+      case ConsumerFinalized     => true
+      case _                     => false
+    }
+  }
+
 }
diff --git a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/CommitterSpec.scala b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/CommitterSpec.scala
index 5fea26932..5c78ad2fa 100644
--- a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/CommitterSpec.scala
+++ b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/CommitterSpec.scala
@@ -3,9 +3,9 @@ package zio.kafka.consumer.internal
 import org.apache.kafka.clients.consumer.{ MockConsumer, OffsetAndMetadata, OffsetCommitCallback, OffsetResetStrategy }
 import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.errors.RebalanceInProgressException
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.test._
 import zio._
+import zio.kafka.diagnostics.Diagnostics
 
 import java.util.{ Map => JavaMap }
 import scala.jdk.CollectionConverters.{ MapHasAsJava, MapHasAsScala }
diff --git a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/PartitionStreamControlSpec.scala b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/PartitionStreamControlSpec.scala
index 9fcc3a979..bd288c44d 100644
--- a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/PartitionStreamControlSpec.scala
+++ b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/PartitionStreamControlSpec.scala
@@ -4,8 +4,8 @@ import org.apache.kafka.clients.consumer.ConsumerRecord
 import org.apache.kafka.common.TopicPartition
 import zio._
 import zio.kafka.consumer.CommittableRecord
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer.internal.Runloop.ByteArrayCommittableRecord
+import zio.kafka.diagnostics.Diagnostics
 import zio.test._
 
 import java.util.concurrent.TimeoutException
diff --git a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RebalanceCoordinatorSpec.scala b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RebalanceCoordinatorSpec.scala
index 1ed7a72f7..ed55c6704 100644
--- a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RebalanceCoordinatorSpec.scala
+++ b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RebalanceCoordinatorSpec.scala
@@ -3,7 +3,6 @@ package zio.kafka.consumer.internal
 import org.apache.kafka.clients.consumer._
 import org.apache.kafka.common.TopicPartition
 import zio.kafka.ZIOSpecDefaultSlf4j
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer.internal.Committer.CommitOffsets
 import zio.kafka.consumer.internal.ConsumerAccess.ByteArrayKafkaConsumer
 import zio.kafka.consumer.internal.RebalanceCoordinator._
@@ -11,6 +10,7 @@ import zio.kafka.consumer.internal.Runloop.ByteArrayCommittableRecord
 import zio.kafka.consumer.{ CommittableRecord, ConsumerSettings }
 import zio.test._
 import zio._
+import zio.kafka.diagnostics.Diagnostics
 import zio.stream.ZStream
 
 import java.util.{ Map => JavaMap }
diff --git a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RunloopSpec.scala b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RunloopSpec.scala
index e30c4051e..7023a8147 100644
--- a/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RunloopSpec.scala
+++ b/zio-kafka-test/src/test/scala/zio/kafka/consumer/internal/RunloopSpec.scala
@@ -10,9 +10,11 @@ import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.errors.{ AuthenticationException, AuthorizationException }
 import zio._
 import zio.kafka.ZIOSpecDefaultSlf4j
-import zio.kafka.consumer.diagnostics.{ DiagnosticEvent, Diagnostics }
+import zio.kafka.consumer.Consumer.ConsumerDiagnostics
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
 import zio.kafka.consumer.internal.RunloopAccess.PartitionAssignment
 import zio.kafka.consumer.{ ConsumerSettings, Subscription }
+import zio.kafka.diagnostics.{ Diagnostics, SlidingDiagnostics }
 import zio.metrics.{ MetricState, Metrics }
 import zio.stream.{ Take, ZStream }
 import zio.test.TestAspect.withLiveClock
@@ -63,7 +65,7 @@ object RunloopSpec extends ZIOSpecDefaultSlf4j {
       test(
         "runloop does not starts a new stream for partition which being revoked right after assignment within the same RebalanceEvent"
       ) {
-        Diagnostics.SlidingQueue.make(100).flatMap { diagnostics =>
+        SlidingDiagnostics.make[DiagnosticEvent](100).flatMap { diagnostics =>
           withRunloop(diagnostics) { (mockConsumer, partitionsHub, runloop) =>
             mockConsumer.schedulePollTask { () =>
               mockConsumer.updateEndOffsets(Map(tp10 -> Long.box(0L), tp11 -> Long.box(0L)).asJava)
@@ -103,7 +105,7 @@ object RunloopSpec extends ZIOSpecDefaultSlf4j {
       test(
         "runloop continues polling after a lost partition"
       ) {
-        Diagnostics.SlidingQueue.make(100).flatMap { diagnostics =>
+        SlidingDiagnostics.make[DiagnosticEvent](100).flatMap { diagnostics =>
           var rebalanceListener: ConsumerRebalanceListener = null
 
           // Catches the rebalance listener so we can use it
@@ -188,7 +190,7 @@ object RunloopSpec extends ZIOSpecDefaultSlf4j {
     ) @@ withLiveClock
 
   private def withRunloop(
-    diagnostics: Diagnostics = Diagnostics.NoOp,
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp,
     mockConsumer: BinaryMockConsumer = new BinaryMockConsumer(OffsetResetStrategy.LATEST)
   )(
     f: (BinaryMockConsumer, PartitionsHub, Runloop) => ZIO[Scope, Throwable, TestResult]
diff --git a/zio-kafka-testkit/src/main/scala/zio/kafka/testkit/KafkaTestUtils.scala b/zio-kafka-testkit/src/main/scala/zio/kafka/testkit/KafkaTestUtils.scala
index 8cf98f4a5..0b284e3dc 100644
--- a/zio-kafka-testkit/src/main/scala/zio/kafka/testkit/KafkaTestUtils.scala
+++ b/zio-kafka-testkit/src/main/scala/zio/kafka/testkit/KafkaTestUtils.scala
@@ -4,9 +4,9 @@ import org.apache.kafka.clients.consumer.{ ConsumerConfig, ConsumerRecord }
 import org.apache.kafka.clients.producer.{ ProducerRecord, RecordMetadata }
 import zio._
 import zio.kafka.admin._
-import zio.kafka.consumer.Consumer.{ AutoOffsetStrategy, OffsetRetrieval }
+import zio.kafka.consumer.Consumer.{ AutoOffsetStrategy, ConsumerDiagnostics, OffsetRetrieval }
 import zio.kafka.consumer._
-import zio.kafka.consumer.diagnostics.Diagnostics
+import zio.kafka.diagnostics.Diagnostics
 import zio.kafka.producer._
 import zio.kafka.serde.{ Deserializer, Serde }
 import zio.stream.ZStream
@@ -237,7 +237,7 @@ object KafkaTestUtils {
     clientInstanceId: Option[String] = None,
     offsetRetrieval: OffsetRetrieval = OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest),
     allowAutoCreateTopics: Boolean = true,
-    diagnostics: Diagnostics = Diagnostics.NoOp,
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp,
     rebalanceSafeCommits: Boolean = false,
     maxRebalanceDuration: Duration = 3.minutes,
     commitTimeout: Duration = ConsumerSettings.defaultCommitTimeout,
@@ -266,7 +266,9 @@ object KafkaTestUtils {
    *
    * ℹ️ Instead of using a layer, consider using [[KafkaTestUtils.makeConsumer]] to directly get a consumer.
    */
-  def minimalConsumer(diagnostics: Diagnostics = Diagnostics.NoOp): ZLayer[ConsumerSettings, Throwable, Consumer] =
+  def minimalConsumer(
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp
+  ): ZLayer[ConsumerSettings, Throwable, Consumer] =
     ZLayer.makeSome[ConsumerSettings, Consumer](
       ZLayer.succeed(diagnostics) >>> Consumer.live
     )
@@ -282,7 +284,7 @@ object KafkaTestUtils {
     clientInstanceId: Option[String] = None,
     offsetRetrieval: OffsetRetrieval = OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest),
     allowAutoCreateTopics: Boolean = true,
-    diagnostics: Diagnostics = Diagnostics.NoOp,
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp,
     rebalanceSafeCommits: Boolean = false,
     maxRebalanceDuration: Duration = 3.minutes,
     commitTimeout: Duration = ConsumerSettings.defaultCommitTimeout,
@@ -341,7 +343,7 @@ object KafkaTestUtils {
     clientInstanceId: Option[String] = None,
     offsetRetrieval: OffsetRetrieval = OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest),
     allowAutoCreateTopics: Boolean = true,
-    diagnostics: Diagnostics = Diagnostics.NoOp,
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp,
     rebalanceSafeCommits: Boolean = false,
     properties: Map[String, String] = Map.empty,
     rebalanceListener: RebalanceListener = RebalanceListener.noop
@@ -371,7 +373,7 @@ object KafkaTestUtils {
     clientInstanceId: Option[String] = None,
     offsetRetrieval: OffsetRetrieval = OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest),
     allowAutoCreateTopics: Boolean = true,
-    diagnostics: Diagnostics = Diagnostics.NoOp,
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp,
     rebalanceSafeCommits: Boolean = false,
     properties: Map[String, String] = Map.empty,
     rebalanceListener: RebalanceListener = RebalanceListener.noop
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
index c91a6eb84..ef516ee90 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
@@ -8,9 +8,10 @@ import org.apache.kafka.clients.consumer.{
 }
 import org.apache.kafka.common._
 import zio._
-import zio.kafka.consumer.diagnostics.Diagnostics
-import zio.kafka.consumer.diagnostics.Diagnostics.ConcurrentDiagnostics
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
 import zio.kafka.consumer.internal.{ ConsumerAccess, RunloopAccess }
+import zio.kafka.diagnostics.Diagnostics
+import zio.kafka.diagnostics.internal.ConcurrentDiagnostics
 import zio.kafka.serde.{ Deserializer, Serde }
 import zio.kafka.utils.SslHelper
 import zio.stream._
@@ -178,16 +179,23 @@ trait Consumer {
 }
 
 object Consumer {
+
+  /** A callback for consumer diagnostic events. */
+  type ConsumerDiagnostics = zio.kafka.diagnostics.Diagnostics[DiagnosticEvent]
+
+  /** A diagnostics implementation that does nothing. */
+  val NoDiagnostics: ConsumerDiagnostics = zio.kafka.diagnostics.Diagnostics.NoOp
+
   case object CommitTimeout extends RuntimeException("Commit timeout") with NoStackTrace
 
   val offsetBatches: ZSink[Any, Nothing, Offset, Nothing, OffsetBatch] =
     ZSink.foldLeft[Offset, OffsetBatch](OffsetBatch.empty)(_ add _)
 
-  def live: RLayer[ConsumerSettings & Diagnostics, Consumer] =
+  def live: RLayer[ConsumerSettings & ConsumerDiagnostics, Consumer] =
     ZLayer.scoped {
       for {
         settings    <- ZIO.service[ConsumerSettings]
-        diagnostics <- ZIO.service[Diagnostics]
+        diagnostics <- ZIO.service[ConsumerDiagnostics]
         consumer    <- make(settings, diagnostics)
       } yield consumer
     }
@@ -201,10 +209,10 @@ object Consumer {
    */
   def make(
     settings: ConsumerSettings,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp
   ): ZIO[Scope, Throwable, Consumer] =
     for {
-      wrappedDiagnostics <- ConcurrentDiagnostics.make(diagnostics)
+      wrappedDiagnostics <- makeConcurrentDiagnostics(diagnostics)
       _                  <- SslHelper.validateEndpoint(settings.driverSettings)
       consumerAccess     <- ConsumerAccess.make(settings)
       runloopAccess      <- RunloopAccess.make(settings, consumerAccess, wrappedDiagnostics)
@@ -242,10 +250,10 @@ object Consumer {
     javaConsumer: JConsumer[Array[Byte], Array[Byte]],
     settings: ConsumerSettings,
     access: Semaphore,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp
   ): ZIO[Scope, Throwable, Consumer] =
     for {
-      wrappedDiagnostics <- ConcurrentDiagnostics.make(diagnostics)
+      wrappedDiagnostics <- makeConcurrentDiagnostics(diagnostics)
       consumerAccess = new ConsumerAccess(javaConsumer, access)
       runloopAccess <- RunloopAccess.make(settings, consumerAccess, wrappedDiagnostics)
     } yield new ConsumerLive(consumerAccess, settings, runloopAccess)
@@ -341,6 +349,11 @@ object Consumer {
     case object Latest   extends AutoOffsetStrategy
     case object None     extends AutoOffsetStrategy
   }
+
+  private def makeConcurrentDiagnostics(diagnostics: ConsumerDiagnostics): ZIO[Scope, Nothing, ConsumerDiagnostics] =
+    if (diagnostics == Diagnostics.NoOp) ZIO.succeed(diagnostics)
+    else ConcurrentDiagnostics.make(diagnostics, DiagnosticEvent.ConsumerFinalized)
+
 }
 
 private[consumer] final class ConsumerLive private[consumer] (
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/DiagnosticEvent.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/DiagnosticEvent.scala
index ea97c6f80..811c88673 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/DiagnosticEvent.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/DiagnosticEvent.scala
@@ -4,6 +4,7 @@ import org.apache.kafka.clients.consumer.OffsetAndMetadata
 import org.apache.kafka.common.TopicPartition
 
 sealed trait DiagnosticEvent
+
 object DiagnosticEvent {
 
   final case class Poll(
@@ -11,6 +12,7 @@ object DiagnosticEvent {
     tpWithData: Set[TopicPartition],
     tpWithoutData: Set[TopicPartition]
   ) extends DiagnosticEvent
+
   final case class Request(partition: TopicPartition) extends DiagnosticEvent
 
   sealed trait Commit extends DiagnosticEvent
@@ -27,11 +29,8 @@ object DiagnosticEvent {
     ended: Set[TopicPartition]
   ) extends DiagnosticEvent
 
-  sealed trait Finalization extends DiagnosticEvent
-  object Finalization {
-    case object SubscriptionFinalized extends Finalization
-    case object RunloopFinalized      extends Finalization
-    case object ConsumerFinalized     extends Finalization
-  }
+  case object SubscriptionFinalized extends DiagnosticEvent
+  case object RunloopFinalized      extends DiagnosticEvent
+  case object ConsumerFinalized     extends DiagnosticEvent
 
 }
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/Diagnostics.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/Diagnostics.scala
deleted file mode 100644
index 653ce9239..000000000
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/diagnostics/Diagnostics.scala
+++ /dev/null
@@ -1,43 +0,0 @@
-package zio.kafka.consumer.diagnostics
-
-import zio.stream.ZStream
-import zio._
-import zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization.ConsumerFinalized
-
-trait Diagnostics {
-  def emit(event: => DiagnosticEvent): UIO[Unit]
-}
-object Diagnostics {
-  case object NoOp extends Diagnostics {
-    override def emit(event: => DiagnosticEvent): UIO[Unit] = ZIO.unit
-  }
-
-  final case class SlidingQueue private[Diagnostics] (queue: Queue[DiagnosticEvent]) extends Diagnostics {
-    override def emit(event: => DiagnosticEvent): UIO[Unit] = queue.offer(event).unit
-  }
-
-  object SlidingQueue {
-    def make(queueSize: Int = 16): ZIO[Scope, Nothing, SlidingQueue] =
-      ZIO.acquireRelease(Queue.sliding[DiagnosticEvent](queueSize))(_.shutdown).map(SlidingQueue(_))
-  }
-
-  object ConcurrentDiagnostics {
-
-    /**
-     * @return
-     *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to
-     *   the fiber via an unbounded queue
-     */
-    def make(wrapped: Diagnostics): ZIO[Scope, Nothing, Diagnostics] =
-      if (wrapped == Diagnostics.NoOp) ZIO.succeed(Diagnostics.NoOp)
-      else {
-        for {
-          queue <- ZIO.acquireRelease(Queue.unbounded[DiagnosticEvent])(_.shutdown)
-          fib   <- ZStream.fromQueue(queue).tap(wrapped.emit(_)).takeUntil(_ == ConsumerFinalized).runDrain.forkScoped
-          _     <- ZIO.addFinalizer(queue.offer(ConsumerFinalized) *> fib.await)
-        } yield new Diagnostics {
-          override def emit(event: => DiagnosticEvent): UIO[Unit] = queue.offer(event).unit
-        }
-      }
-  }
-}
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/LiveCommitter.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/LiveCommitter.scala
index 88fad046f..d0d39df12 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/LiveCommitter.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/LiveCommitter.scala
@@ -2,12 +2,12 @@ package zio.kafka.consumer.internal
 import org.apache.kafka.clients.consumer.{ OffsetAndMetadata, OffsetCommitCallback }
 import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.errors.RebalanceInProgressException
-import zio.kafka.consumer.Consumer.CommitTimeout
-import zio.kafka.consumer.diagnostics.{ DiagnosticEvent, Diagnostics }
+import zio.kafka.consumer.Consumer.{ CommitTimeout, ConsumerDiagnostics }
 import zio.kafka.consumer.internal.Committer.CommitOffsets
 import zio.kafka.consumer.internal.ConsumerAccess.ByteArrayKafkaConsumer
 import zio.kafka.consumer.internal.LiveCommitter.Commit
 import zio._
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
 
 import java.util.{ Map => JavaMap }
 import scala.collection.mutable
@@ -16,7 +16,7 @@ import scala.jdk.CollectionConverters._
 private[consumer] final class LiveCommitter(
   commitQueue: Queue[Commit],
   commitTimeout: Duration,
-  diagnostics: Diagnostics,
+  diagnostics: ConsumerDiagnostics,
   consumerMetrics: ConsumerMetrics,
   onCommitAvailable: UIO[Unit],
   committedOffsetsRef: Ref[CommitOffsets],
@@ -172,7 +172,7 @@ private[consumer] final class LiveCommitter(
 private[internal] object LiveCommitter {
   def make(
     commitTimeout: Duration,
-    diagnostics: Diagnostics,
+    diagnostics: ConsumerDiagnostics,
     consumerMetrics: ConsumerMetrics,
     onCommitAvailable: UIO[Unit]
   ): ZIO[Scope, Nothing, LiveCommitter] = for {
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/PartitionStreamControl.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/PartitionStreamControl.scala
index dd93b9c07..83a44938f 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/PartitionStreamControl.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/PartitionStreamControl.scala
@@ -1,12 +1,13 @@
 package zio.kafka.consumer.internal
 
 import org.apache.kafka.common.TopicPartition
+import zio.kafka.consumer.Consumer.ConsumerDiagnostics
 import zio.kafka.consumer.Offset
-import zio.kafka.consumer.diagnostics.{ DiagnosticEvent, Diagnostics }
 import zio.kafka.consumer.internal.PartitionStreamControl.QueueInfo
 import zio.kafka.consumer.internal.Runloop.ByteArrayCommittableRecord
 import zio.stream.{ Take, ZStream }
 import zio._
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
 
 import java.util.concurrent.TimeoutException
 import scala.util.control.NoStackTrace
@@ -127,7 +128,7 @@ object PartitionStreamControl {
   private[internal] def newPartitionStream(
     tp: TopicPartition,
     requestData: UIO[Unit],
-    diagnostics: Diagnostics,
+    diagnostics: ConsumerDiagnostics,
     maxStreamPullInterval: Duration
   ): UIO[PartitionStreamControl] = {
     val maxStreamPullIntervalNanos = maxStreamPullInterval.toNanos
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/Runloop.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/Runloop.scala
index d7aa77b00..fff72f050 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/Runloop.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/Runloop.scala
@@ -4,10 +4,9 @@ import org.apache.kafka.clients.consumer._
 import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.errors.{ AuthenticationException, AuthorizationException }
 import zio._
-import zio.kafka.consumer.Consumer.OffsetRetrieval
+import zio.kafka.consumer.Consumer.{ ConsumerDiagnostics, OffsetRetrieval }
 import zio.kafka.consumer._
-import zio.kafka.consumer.diagnostics.DiagnosticEvent.{ Finalization, Rebalance }
-import zio.kafka.consumer.diagnostics.{ DiagnosticEvent, Diagnostics }
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
 import zio.kafka.consumer.internal.ConsumerAccess.ByteArrayKafkaConsumer
 import zio.kafka.consumer.internal.RebalanceCoordinator._
 import zio.kafka.consumer.internal.Runloop._
@@ -24,7 +23,7 @@ private[consumer] final class Runloop private (
   consumer: ConsumerAccess,
   commandQueue: Queue[RunloopCommand],
   partitionsHub: Hub[Take[Throwable, PartitionAssignment]],
-  diagnostics: Diagnostics,
+  diagnostics: ConsumerDiagnostics,
   maxStreamPullInterval: Duration,
   currentStateRef: Ref[State],
   rebalanceCoordinator: RebalanceCoordinator,
@@ -306,7 +305,7 @@ private[consumer] final class Runloop private (
                                        lostTps.size
                                      )
                                 _ <- diagnostics.emit(
-                                       Rebalance(
+                                       DiagnosticEvent.Rebalance(
                                          revoked = revokedTps,
                                          assigned = assignedTps,
                                          lost = lostTps,
@@ -587,12 +586,12 @@ object Runloop {
     settings: ConsumerSettings,
     maxStreamPullInterval: Duration,
     maxRebalanceDuration: Duration,
-    diagnostics: Diagnostics,
+    diagnostics: ConsumerDiagnostics,
     consumer: ConsumerAccess,
     partitionsHub: Hub[Take[Throwable, PartitionAssignment]]
   ): URIO[Scope, Runloop] =
     for {
-      _                  <- ZIO.addFinalizer(diagnostics.emit(Finalization.RunloopFinalized))
+      _                  <- ZIO.addFinalizer(diagnostics.emit(DiagnosticEvent.RunloopFinalized))
       commandQueue       <- ZIO.acquireRelease(Queue.unbounded[RunloopCommand])(_.shutdown)
       lastRebalanceEvent <- Ref.Synchronized.make[RebalanceEvent](RebalanceEvent.None)
       initialState = State.initial
diff --git a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/RunloopAccess.scala b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/RunloopAccess.scala
index 6b9941b30..3d3955b6b 100644
--- a/zio-kafka/src/main/scala/zio/kafka/consumer/internal/RunloopAccess.scala
+++ b/zio-kafka/src/main/scala/zio/kafka/consumer/internal/RunloopAccess.scala
@@ -2,13 +2,14 @@ package zio.kafka.consumer.internal
 
 import org.apache.kafka.clients.consumer.ConsumerConfig
 import org.apache.kafka.common.TopicPartition
-import zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization
-import zio.kafka.consumer.diagnostics.Diagnostics
 import zio.kafka.consumer.internal.Runloop.ByteArrayCommittableRecord
 import zio.kafka.consumer.internal.RunloopAccess.PartitionAssignment
 import zio.kafka.consumer.{ ConsumerSettings, InvalidSubscriptionUnion, OffsetBatch, Subscription }
 import zio.stream.{ Stream, Take, UStream, ZStream }
 import zio._
+import zio.kafka.consumer.Consumer.ConsumerDiagnostics
+import zio.kafka.consumer.diagnostics.DiagnosticEvent
+import zio.kafka.diagnostics.Diagnostics
 
 private[internal] sealed trait RunloopState
 private[internal] object RunloopState {
@@ -28,7 +29,7 @@ private[consumer] final class RunloopAccess private (
   runloopStateRef: Ref.Synchronized[RunloopState],
   partitionHub: Hub[Take[Throwable, PartitionAssignment]],
   makeRunloop: UIO[Runloop],
-  diagnostics: Diagnostics
+  diagnostics: ConsumerDiagnostics
 ) {
 
   private def withRunloopZIO[E](
@@ -62,7 +63,7 @@ private[consumer] final class RunloopAccess private (
       _ <- withRunloopZIO(requireRunning = true)(_.addSubscription(subscription))
       _ <- ZIO.addFinalizer {
              withRunloopZIO(requireRunning = false)(_.removeSubscription(subscription)) <*
-               diagnostics.emit(Finalization.SubscriptionFinalized)
+               diagnostics.emit(DiagnosticEvent.SubscriptionFinalized)
            }
     } yield stream
 
@@ -77,7 +78,7 @@ private[consumer] object RunloopAccess {
   def make(
     settings: ConsumerSettings,
     consumerAccess: ConsumerAccess,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = Diagnostics.NoOp
   ): ZIO[Scope, Throwable, RunloopAccess] =
     for {
       maxPollInterval <- maxPollIntervalConfig(settings)
diff --git a/zio-kafka/src/main/scala/zio/kafka/diagnostics/Diagnostics.scala b/zio-kafka/src/main/scala/zio/kafka/diagnostics/Diagnostics.scala
new file mode 100644
index 000000000..e343a9a34
--- /dev/null
+++ b/zio-kafka/src/main/scala/zio/kafka/diagnostics/Diagnostics.scala
@@ -0,0 +1,21 @@
+package zio.kafka.diagnostics
+
+import zio.{ UIO, ZIO }
+
+/**
+ * A callback interface for diagnostic events.
+ */
+trait Diagnostics[-DiagnosticEvent] {
+  def emit(event: => DiagnosticEvent): UIO[Unit]
+}
+
+object Diagnostics {
+
+  /**
+   * A diagnostics implementation that does nothing.
+   */
+  val NoOp: Diagnostics[Any] = new Diagnostics[Any] {
+    override def emit(event: => Any): UIO[Unit] = ZIO.unit
+  }
+
+}
diff --git a/zio-kafka/src/main/scala/zio/kafka/diagnostics/SlidingDiagnostics.scala b/zio-kafka/src/main/scala/zio/kafka/diagnostics/SlidingDiagnostics.scala
new file mode 100644
index 000000000..96ce07b26
--- /dev/null
+++ b/zio-kafka/src/main/scala/zio/kafka/diagnostics/SlidingDiagnostics.scala
@@ -0,0 +1,39 @@
+package zio.kafka.diagnostics
+
+import zio.{ Queue, Scope, UIO, ZIO }
+
+object SlidingDiagnostics {
+
+  /**
+   * Create a [[Diagnostics]] implementation that keeps the last `queueSize` emitted events in a queue.
+   *
+   * Useful for testing, or any other case where you don't want to risk high memory usage due to a slow diagnostics
+   * implementation.
+   *
+   * For example, for a Diagnostics that keeps the last 100
+   * [[zio.kafka.consumer.diagnostics.DiagnosticEvent consumer DiagnosticEvent]]s use:
+   * {{{
+   *   import zio.kafka.consumer.diagnostics.{DiagnosticEvent => ConsumerDiagnosticEvent}
+   *   for {
+   *     diagnostics <- SlidingDiagnostics.make[ConsumerDiagnosticEvent](100)
+   *     ... use diagnostics
+   *     events <- diagnostics.queue.takeAll
+   *   } yield {
+   *     assert(events.length == 100)
+   *   }
+   * }}}
+   *
+   * @param queueSize
+   *   number of events to retain in the queue
+   * @tparam DiagnosticEvent
+   *   the type of event to keep
+   */
+  def make[DiagnosticEvent](queueSize: Int = 16): ZIO[Scope, Nothing, QueuingDiagnostics[DiagnosticEvent]] =
+    ZIO.acquireRelease(Queue.sliding[DiagnosticEvent](queueSize))(_.shutdown).map(QueuingDiagnostics(_))
+
+}
+
+final case class QueuingDiagnostics[DiagnosticEvent] private[diagnostics] (queue: Queue[DiagnosticEvent])
+    extends Diagnostics[DiagnosticEvent] {
+  override def emit(event: => DiagnosticEvent): UIO[Unit] = queue.offer(event).unit
+}
diff --git a/zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala b/zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
new file mode 100644
index 000000000..c4a036024
--- /dev/null
+++ b/zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
@@ -0,0 +1,25 @@
+package zio.kafka.diagnostics.internal
+
+import zio._
+import zio.kafka.diagnostics.Diagnostics
+import zio.stream._
+
+object ConcurrentDiagnostics {
+
+  /**
+   * @return
+   *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to the
+   *   fiber via an unbounded queue. Upon finalization, `finalEvent` is emitted as the last event.
+   */
+  def make[DiagnosticEvent](
+    wrapped: Diagnostics[DiagnosticEvent],
+    finalEvent: DiagnosticEvent
+  ): ZIO[Scope, Nothing, Diagnostics[DiagnosticEvent]] =
+    for {
+      queue <- ZIO.acquireRelease(Queue.unbounded[DiagnosticEvent])(_.shutdown)
+      fib   <- ZStream.fromQueue(queue).tap(wrapped.emit(_)).takeUntil(_ == finalEvent).runDrain.forkScoped
+      _     <- ZIO.addFinalizer(queue.offer(finalEvent) *> fib.await)
+    } yield new Diagnostics[DiagnosticEvent] {
+      override def emit(event: => DiagnosticEvent): UIO[Unit] = queue.offer(event).unit
+    }
+}


Reviews:
user: login: vermas7988
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T09:03:00Z
user: login: vermas7988
body: I did a review. Just two suggestions if they are applicable.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T09:08:11Z
user: login: vermas7988
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T09:08:59Z
user: login: erikvanoosten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T09:58:41Z
user: login: erikvanoosten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T10:01:45Z
user: login: vermas7988
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T10:31:49Z
user: login: vermas7988
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T10:34:21Z
user: login: erikvanoosten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T10:38:16Z
user: login: svroonland
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-22T15:13:32Z
user: login: erikvanoosten
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-22T19:24:45Z

Review comments:
diff_hunk: @@ -252,10 +259,10 @@ object Consumer {
     javaConsumer: JConsumer[Array[Byte], Array[Byte]],
     settings: ConsumerSettings,
     access: Semaphore,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = NoDiagnostics
path: zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
user: login: vermas7988
body: If we are already doing a breaking change, I would prefer to use a boolean here in parameter e.g. `diagnosticsEnabled` to hide the details of type from user. It would be beginner friendly and also, It would prevent breaking changes even if the types are changed internally like now or some other reshuffling happens.
created_at: 2025-03-01T09:02:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,25 @@
+package zio.kafka.diagnostics.internal
+
+import zio._
+import zio.kafka.diagnostics.Diagnostics
+import zio.stream._
+
+object ConcurrentDiagnostics {
+
+  /**
+   * @return
+   *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to the
+   *   fiber via an unbounded queue. Upon finalization, `finalEvent` is emitted as the last event.
+   */
+  def make[DiagnosticEvent](
path: zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
user: login: vermas7988
body: Personally I would have had a parent sealed trait for the `DiagnosticEvent` if we know that we only have `ConsumerDiagnosticEvent` and `ProducerDiagnosticEvent` implementaions. Or do we plan flexibility to have some other implementation from user? If we do that we can also drop explicit need of `finalEvent` as it can be decided by the type of event. 
created_at: 2025-03-01T09:07:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,10 +259,10 @@ object Consumer {
     javaConsumer: JConsumer[Array[Byte], Array[Byte]],
     settings: ConsumerSettings,
     access: Semaphore,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = NoDiagnostics
path: zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
user: login: vermas7988
body: or `enableDiagnostics`
created_at: 2025-03-01T09:08:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,10 +259,10 @@ object Consumer {
     javaConsumer: JConsumer[Array[Byte], Array[Byte]],
     settings: ConsumerSettings,
     access: Semaphore,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = NoDiagnostics
path: zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
user: login: erikvanoosten
body: Unfortunately, it doesn't work that way. There is no 'enable', as a user you have to implement the `Diagnostics` trait and in that way provide a callback.
created_at: 2025-03-01T09:58:41Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,25 @@
+package zio.kafka.diagnostics.internal
+
+import zio._
+import zio.kafka.diagnostics.Diagnostics
+import zio.stream._
+
+object ConcurrentDiagnostics {
+
+  /**
+   * @return
+   *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to the
+   *   fiber via an unbounded queue. Upon finalization, `finalEvent` is emitted as the last event.
+   */
+  def make[DiagnosticEvent](
path: zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
user: login: erikvanoosten
body: Theoretically we might have more `DiagnosticEvent`s. For example for the admin client.

However, here the `DiagnosticEvent` is fully unspecified because `ConcurrentDiagnostics` does not need to make any assumptions about it. The only downside is that the type can not be inferred (that downside does not go away with your proposal).
created_at: 2025-03-01T10:01:45Z
author_association: COLLABORATOR
diff_hunk: @@ -252,10 +259,10 @@ object Consumer {
     javaConsumer: JConsumer[Array[Byte], Array[Byte]],
     settings: ConsumerSettings,
     access: Semaphore,
-    diagnostics: Diagnostics = Diagnostics.NoOp
+    diagnostics: ConsumerDiagnostics = NoDiagnostics
path: zio-kafka/src/main/scala/zio/kafka/consumer/Consumer.scala
user: login: vermas7988
body: okay understood..
created_at: 2025-03-01T10:31:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,25 @@
+package zio.kafka.diagnostics.internal
+
+import zio._
+import zio.kafka.diagnostics.Diagnostics
+import zio.stream._
+
+object ConcurrentDiagnostics {
+
+  /**
+   * @return
+   *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to the
+   *   fiber via an unbounded queue. Upon finalization, `finalEvent` is emitted as the last event.
+   */
+  def make[DiagnosticEvent](
path: zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
user: login: vermas7988
body: just curious why we cant infer the type once we have sealed trait and then we start to have type bound on the param? But it makes sense that `ConcurrentDiagnostics` doesnt need to make assumptions about the Event type.
created_at: 2025-03-01T10:34:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,25 @@
+package zio.kafka.diagnostics.internal
+
+import zio._
+import zio.kafka.diagnostics.Diagnostics
+import zio.stream._
+
+object ConcurrentDiagnostics {
+
+  /**
+   * @return
+   *   a `Diagnostics` that runs the wrapped `Diagnostics` concurrently in a separate fiber. Events are emitting to the
+   *   fiber via an unbounded queue. Upon finalization, `finalEvent` is emitted as the last event.
+   */
+  def make[DiagnosticEvent](
path: zio-kafka/src/main/scala/zio/kafka/diagnostics/internal/ConcurrentDiagnostics.scala
user: login: erikvanoosten
body: Sorry, I mixed it up with `SlidingDiagnostics` which is also not making any assumptions and therefore needs an annotation:
```scala
SlidingDiagnostics. make[ConsumerDiagnosticEvent](100)
```
created_at: 2025-03-01T10:38:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,21 @@
+package zio.kafka.diagnostics
+
+import zio.{ UIO, ZIO }
+
+/**
+ * A callback interface for diagnostic events.
+ */
+trait Diagnostics[-DiagnosticEvent] {
+  def emit(event: => DiagnosticEvent): UIO[Unit]
+}
+
+object Diagnostics {
+
+  /**
+   * A diagnostics implementation that does nothing.
+   */
+  val noOp: Diagnostics[Any] = new Diagnostics[Any] {
path: zio-kafka/src/main/scala/zio/kafka/diagnostics/Diagnostics.scala
user: login: erikvanoosten
body: Should be named `NoOp`.
created_at: 2025-03-22T19:24:45Z
author_association: COLLABORATOR

PR comments:
user: login: erikvanoosten
created_at: 2025-02-08T11:57:32Z
body: Hint for the reviewer: enable 'hide whitespace' to minimize the diff.
author_association: COLLABORATOR
user: login: svroonland
created_at: 2025-03-08T10:18:53Z
body: I can see the use in generalizing SlidingDiagnostics and ConcurrentDiagnostics by parameterizing them with the type of diagnostic event. But I'd like to see if we can keep the rest of the stuff in the same package and with the same name, just to minimize changes.

What would be the downside of having `zio.kafka.consumer.diagnostics.DiagnosticEvent` and `zio.kafka.producer.diagnostics.DiagnosticEvent` for example?
author_association: COLLABORATOR
user: login: erikvanoosten
created_at: 2025-03-08T11:44:56Z
body: > What would be the downside of having `zio.kafka.consumer.diagnostics.DiagnosticEvent` and `zio.kafka.producer.diagnostics.DiagnosticEvent` for example?

One downside is that its very annoying when you have a scala file that needs both classes.
But your proposal is not unreasonable. I'll try it out.

author_association: COLLABORATOR
user: login: erikvanoosten
created_at: 2025-03-16T15:09:38Z
body: **UPDATE: this comment is no longer accurate.**

@svroonland the last commit reduces the number of changes one needs for migrating from zio-kafka 2.x to 3.x:

The following changes are possible/needed:

| Required? | Old | New |
| --- | --- | --- |
| optional | `zio.kafka.consumer.diagnostics.Diagnostics` | `zio.kafka.consumer.Consumer.ConsumerDiagnostics` |
| optional | `zio.kafka.consumer.diagnostics.Diagnostics.NoOp` | `zio.kafka.consumer.Consumer.NoDiagnostics` |
| required | `zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization.*` | `zio.kafka.consumer.ConsumerDiagnosticEvent.*` |

Different invocation:
old: `zio.kafka.consumer.diagnostics.Diagnostics.SligingQueue.make()`
new: `zio.kafka.diagnostics.SlidingDiagnostics.make[zio.kafka.consumer.diagnostics.DiagnosticEvent]()`

Move to internal or test package:

| Old | New | New location |
| --- | --- | --- |
| `zio.kafka.consumer.diagnostics.Diagnostics.ConcurrentDiagnostics` | `zio.kafka.diagnostics.internal.ConcurrentDiagnostics` | internal |
| `zio.kafka.consumer.diagnostics.DiagnosticEvent.Finalization` | removed | - |

Notes:

- The intermediate trait `DiagnosticEvent.Finalization` was removed from the type hierarchy because it doesn't serve any user purpose (it was introduced to make it easier to test the shutdown sequence of the consumer).
author_association: COLLABORATOR