Repository: zio/zio
Pull request #9697: Add `ZStream.mapZIOChunked`
Author: login: erikvanoosten
Created: 2025-03-14T14:10:20Z | Merged: 2025-03-28T05:24:53Z

Add `ZStream.mapZIOChunked` as an alternative to `mapZIO` that does not destroy the chunking structure.

Co-authored-by: Matthias Berndt <matthias_berndt@gmx.de>
Co-authored-by: Paul Daniels <paulpdaniels@gmail.com>
Co-authored-by: Regis Kuckaertz <platen-porter0e@icloud.com>


Diff:
diff --git a/streams-tests/shared/src/test/scala/zio/stream/ZStreamSpec.scala b/streams-tests/shared/src/test/scala/zio/stream/ZStreamSpec.scala
index 038c1a0c90f5..98a722109237 100644
--- a/streams-tests/shared/src/test/scala/zio/stream/ZStreamSpec.scala
+++ b/streams-tests/shared/src/test/scala/zio/stream/ZStreamSpec.scala
@@ -2735,6 +2735,39 @@ object ZStreamSpec extends ZIOBaseSpec {
             )(equalTo(Chunk(Right(1), Right(2), Left("boom"))))
           }
         ),
+        suite("mapZIOChunked")(
+          test("ZIO#foreach equivalence when error-free") {
+            check(Gen.small(Gen.listOfN(_)(Gen.byte)), Gen.function(Gen.successes(Gen.byte))) { (data, f) =>
+              val s = ZStream.fromIterable(data)
+
+              for {
+                l <- s.mapZIOChunked(f).runCollect
+                r <- ZIO.foreach(data)(f)
+              } yield assert(l.toList)(equalTo(r))
+            }
+          },
+          test("retains chunks") {
+            ZStream
+              .fromChunks(Chunk(1, 2))
+              .mapZIOChunked(ZIO.succeed(_))
+              .chunks
+              .runCount
+              .map(count => assertTrue(count == 1))
+          },
+          test("retains chunks up to error") {
+            assertZIO(
+              ZStream
+                .fromChunks(Chunk(1, 2), Chunk(3, 4, 5))
+                .mapZIOChunked {
+                  case 4 => ZIO.fail("boom")
+                  case x => ZIO.succeed(x)
+                }
+                .either
+                .chunks
+                .runCollect
+            )(equalTo(Chunk(Chunk(Right(1), Right(2)), Chunk(Right(3)), Chunk(Left("boom")))))
+          }
+        ),
         suite("mapZIOPar")(
           test("foreachParN equivalence") {
             checkN(10)(Gen.small(Gen.listOfN(_)(Gen.byte)), Gen.function(Gen.successes(Gen.byte))) { (data, f) =>
diff --git a/streams/shared/src/main/scala/zio/stream/ZPipeline.scala b/streams/shared/src/main/scala/zio/stream/ZPipeline.scala
index 0a0289605589..9393d4b11cfe 100644
--- a/streams/shared/src/main/scala/zio/stream/ZPipeline.scala
+++ b/streams/shared/src/main/scala/zio/stream/ZPipeline.scala
@@ -482,6 +482,25 @@ final class ZPipeline[-Env, +Err, -In, +Out] private (
   )(implicit trace: Trace): ZPipeline[Env2, Err2, In, Out2] =
     self >>> ZPipeline.mapZIO(f)
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env2 <: Env, Err2 >: Err, Out2](
+    f: Out => ZIO[Env2, Err2, Out2]
+  )(implicit trace: Trace): ZPipeline[Env2, Err2, In, Out2] =
+    self >>> ZPipeline.mapZIOChunked(f)
+
   /**
    * Maps over elements of the stream with the specified effectful function,
    * executing up to `n` invocations of `f` concurrently. Transformed elements
@@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
+            val builder = ChunkBuilder.make[Out](chunk.size)
+            chunk
+              .mapZIODiscard(f(_).map(builder += _))
+              .foldCause(
+                cause => writeWithNext(builder, ZChannel.refailCause(cause)),
+                _ => writeWithNext(builder, reader)
+              )
+          },
+        err => ZChannel.refailCause(err),
+        done => ZChannel.succeed(done)
+      )
+    new ZPipeline(reader)
+  }
+
   /**
    * Maps over elements of the stream with the specified effectful function,
    * executing up to `n` invocations of `f` concurrently. Transformed elements
diff --git a/streams/shared/src/main/scala/zio/stream/ZStream.scala b/streams/shared/src/main/scala/zio/stream/ZStream.scala
index 189e26099059..335b77cebd78 100644
--- a/streams/shared/src/main/scala/zio/stream/ZStream.scala
+++ b/streams/shared/src/main/scala/zio/stream/ZStream.scala
@@ -1922,6 +1922,8 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
    *
    * @note
    *   This combinator destroys the chunking structure.
+   * @see
+   *   [[mapZIOChunked]] for a version that preserves the chunking structure
    */
   def mapZIO[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] = {
 
@@ -1943,6 +1945,23 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    self >>> ZPipeline.mapZIOChunked(f)
+
   /**
    * Maps over elements of the stream with the specified effectful function,
    * executing up to `n` invocations of `f` concurrently. Transformed elements


Reviews:
user: login: calvinlfer
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T14:40:15Z
user: login: calvinlfer
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-14T15:11:33Z
user: login: mberndt123
body: I think this not good:
>   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
>   * element in the middle of a Chunk, the other elements of the chunk are lost.

If processing fails in the middle of a chunk, the stream should first emit the elements where processing succeeded and then fail the stream. 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T18:36:55Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T08:10:00Z
user: login: paulpdaniels
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T09:42:35Z
user: login: calvinlfer
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-15T14:05:40Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T14:55:03Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T14:57:48Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T15:58:21Z
user: login: paulpdaniels
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T16:27:53Z
user: login: mberndt123
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T16:31:09Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T16:34:36Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T16:38:55Z
user: login: mberndt123
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T23:13:03Z
user: login: regiskuckaertz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T07:13:23Z
user: login: regiskuckaertz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T07:38:42Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T12:02:05Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T12:23:12Z
user: login: regiskuckaertz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T18:44:28Z
user: login: regiskuckaertz
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-16T18:44:47Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T18:59:22Z
user: login: paulpdaniels
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T03:36:43Z
user: login: paulpdaniels
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T03:41:02Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T08:09:52Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T08:12:33Z
user: login: regiskuckaertz
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-18T08:30:39Z
user: login: paulpdaniels
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T16:44:21Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:11:33Z
user: login: paulpdaniels
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T02:31:32Z
user: login: kyri-petrou
body: Thanks this is amazing! Just a minor comment as I think it'd be good to increase awareness that this method exists
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T09:28:38Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:55:11Z
user: login: kyri-petrou
body: Love this 🚀 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T05:17:53Z
user: login: kyri-petrou
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T05:24:32Z
user: login: erikvanoosten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T11:08:54Z

Review comments:
diff_hunk: @@ -1927,6 +1927,15 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the other elements of the chunk are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunksWith(stream => stream.mapZIO(f))
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: calvinlfer
body: is this the same as doing
```scala
this.mapChunksZIO(chunkA => ZIO.foreach(chunkA)(f))
```
created_at: 2025-03-14T14:40:15Z
author_association: MEMBER
diff_hunk: @@ -1927,6 +1927,15 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the other elements of the chunk are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunksWith(stream => stream.mapZIO(f))
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: erikvanoosten
body: Yes it is.
created_at: 2025-03-15T08:10:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
+      }
+    }
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: paulpdaniels
body: This should actually be implemented within `ZPipeline`, then the implementation here should just reference that:

```scala
def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] = 
  self >>> ZPipeline.mapZIOChunked(f)
```
created_at: 2025-03-15T09:29:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: paulpdaniels
body: This is pretty compact to be sure, but I think for "core" methods like this we should consider using the lower level primitives, especially to make this usable as a pipeline operator:

```scala
  def mapZIOChunked[Env, Err, In, Out](
    f: In => ZIO[Env, Err, Out]
  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
    def loop(
      chunkIterator: Chunk.ChunkIterator[In],
      index: Int,
      builder: ChunkBuilder[Out]
    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
      if (chunkIterator.hasNextAt(index)) {
        ZChannel.unwrap {
          val a = chunkIterator.nextAt(index)
          f(a).foldCause(
            cause =>
              if (builder ne null) ZChannel.write(builder.result()) *> ZChannel.refailCause(cause)
              else ZChannel.refailCause(cause),
            a1 => {
              val b = if (builder ne null) builder else ChunkBuilder.make[Out](chunkIterator.length)
              builder += a1
              loop(chunkIterator, index + 1, b)
            }
          )
        }
      } else {
        if (builder ne null) ZChannel.write(builder.result()) *> reader
        else reader
      }

    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = ZChannel.readWithCause(
      elem => loop(elem.chunkIterator, 0, null),
      err => ZChannel.refailCause(err),
      done => ZChannel.succeed(done)
    )

    new ZPipeline(reader)
  }
```
created_at: 2025-03-15T09:42:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: calvinlfer
body: hey @paulpdaniels, I had a question about ChunkIterator, does calling `.length` consume the iterator or does it leave it intact?
created_at: 2025-03-15T14:05:39Z
author_association: MEMBER
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: erikvanoosten
body: > hey @paulpdaniels, I had a question about ChunkIterator, does calling `.length` consume the iterator or does it leave it intact?

`.length` gives the size of the underlying chunk.
created_at: 2025-03-15T14:55:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: erikvanoosten
body: @paulpdaniels Thanks for that.

The statement `builder += a1`, should that be `b += a1`?
created_at: 2025-03-15T14:57:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1927,6 +1927,28 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Maps over elements of the stream with the specified effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
+   * element in the middle of a Chunk, the following elements of the chunk are
+   * consumed but not processed; they are lost.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
+    this.chunks.flatMap { chunk =>
+      val buffer = Chunk.newBuilder[A1]
+      ZStream.unwrap {
+        ZIO
+          .foreachDiscard(chunk) { a =>
+            f(a).tap(a1 => ZIO.succeed(buffer += a1))
+          }
+          .fold(
+            err => ZStream.fromChunk(buffer.result()) ++ ZStream.fail(err),
+            _ => ZStream.fromChunk(buffer.result())
+          )
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: erikvanoosten
body: Never mind. That code is no longer present.
created_at: 2025-03-15T15:58:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,52 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
+            cause => ZChannel.write(builder.result()) *> ZChannel.refailCause(cause),
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: paulpdaniels
body: This needs an index check, if this is the first value in the chunk we should skip writing an empty chunk and immediately write the error
created_at: 2025-03-15T16:26:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,52 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
+            cause => ZChannel.write(builder.result()) *> ZChannel.refailCause(cause),
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: mberndt123
body: Why? At this point the stream is failed anyway, so it's not worth optimizing this. 
created_at: 2025-03-15T16:31:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,52 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
+            cause => ZChannel.write(builder.result()) *> ZChannel.refailCause(cause),
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: I agree. Emitting empty chunks doesn't make sense. I'll update it.
created_at: 2025-03-15T16:34:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,52 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
+            cause => ZChannel.write(builder.result()) *> ZChannel.refailCause(cause),
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: Fixed in https://github.com/zio/zio/pull/9697/commits/cf506a767a0f2eeec78529ba40b25632340afd29
created_at: 2025-03-15T16:38:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: regiskuckaertz
body: recursive effectful code like that often gets reported as slow, I suggest to change `loop` like this instead:
```scala
def loop(
  chunkIterator: Chunk.ChunkIterator[In],
  index: Int,
  builder: ChunkBuilder[Out]
): IO[Err, Unit] =
  if (chunkIterator.hasNextAt(index)) {
    val in = chunkIterator.nextAt(index)
    f(in).flatMap { out =>
      builder += out
      loop(chunkIterator, index + 1, builder)
    }
  } else ZIO.unit
```
and then fold only once:
```scala
ZChannel.unwrap {
  val builder = ChunkBuilder.make[Out](elem.size)
  loop(elem.chunkIterator, 0, builder).foldCause(
    err => {
      val out = builder.result()
      if (out.nonEmpty) 
        ZChannel.write(out) *> ZChannel.refailCause(cause)
      else ZChannel.refailCause(cause)
    },
    _ => ZChannel.write(builder.result()) *> reader
  )
}
```
created_at: 2025-03-16T07:13:23Z
author_association: MEMBER
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: regiskuckaertz
body: maybe one up is to get rid of loop entirely:
```
val builder = ChunkBuilder.make[Out](elem.size)
elem.mapZIODiscard(f(_).map(builder += _)).foldCause(...)
```
created_at: 2025-03-16T07:38:42Z
author_association: MEMBER
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: Nice idea!
Done in https://github.com/zio/zio/pull/9697/commits/351a19bde3e336bd5931ea351ff4715b2635a8d5
created_at: 2025-03-16T12:02:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: @regiskuckaertz BTW, your suggestion is a bit confusing.

The second implementation in this PR (I think we're at no. 5 now 🙂) was `mapZIODiscard` based (although it did so inside `ZStream`, not in `ZChannel`). I also noticed that `ZChannel.mapZIO` is loop based.

Anyways, I like the current `mapZIODiscard` based implementation. Can you review again please?
created_at: 2025-03-16T12:23:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: regiskuckaertz
body: by loop do you mean the recursive `readWithCause` pattern? tbh I just noticed `mapZIODiscard` uses `foreachDiscard` under the hood, in effectual code idk if that makes a difference.
created_at: 2025-03-16T18:44:27Z
author_association: MEMBER
diff_hunk: @@ -1798,6 +1817,56 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def loop(
+      chunkIterator: Chunk.ChunkIterator[In],
+      index: Int,
+      builder: ChunkBuilder[Out]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      if (chunkIterator.hasNextAt(index)) {
+        ZChannel.unwrap {
+          val a = chunkIterator.nextAt(index)
+          f(a).foldCause(
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: > by loop do you mean the recursive `readWithCause` pattern?

Yes.

> in effectual code idk if that makes a difference.

In a good way I assume?
created_at: 2025-03-16T18:59:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: => ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: paulpdaniels
body: Was the by-name parameter required for some reason? We materialize it in both cases so it seems unnecessary.
created_at: 2025-03-18T03:36:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: => ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: I am not sure about this. I didn't want to change the order in which the stream elements are evaluated. On the other hand, in both cases they are just values so I guess it should be fine.
created_at: 2025-03-18T08:09:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: => ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: Change it to pass by value.
created_at: 2025-03-18T08:12:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: paulpdaniels
body: Super Nit: I realized while doing the benchmarks, that there is a _small_ optimization here if the chunk is empty, we can just immediately recurse, rather than creating all the machinery within the `ZChannel.wrap`. 
created_at: 2025-03-18T16:44:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: So zstreams can emit empty chunks? Is that a common thing?
created_at: 2025-03-18T19:11:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: paulpdaniels
body: They can yes, I don't think it's terrible common, and an argument could be made that we don't want to change the semantics of the stream if we _receive_ an empty chunk since it could be part of some protocol the user is implementing.

- [ ] Actually ignore this nit, I convinced myself just now that it's not great to potentially affect the semantics, for a tiny potential gain.
created_at: 2025-03-19T02:31:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1943,6 +1943,23 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: kyri-petrou
body: Can we add a `@ see [[mapZIOChunked]]` to the scaladoc of `mapZIO` as well? This way users are more likely to be aware of this method (and use it)
created_at: 2025-03-24T09:27:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1943,6 +1943,23 @@ final class ZStream[-R, +E, +A] private (val channel: ZChannel[R, Any, Any, Any,
     new ZStream(self.channel >>> loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[R1 <: R, E1 >: E, A1](f: A => ZIO[R1, E1, A1])(implicit trace: Trace): ZStream[R1, E1, A1] =
path: streams/shared/src/main/scala/zio/stream/ZStream.scala
user: login: erikvanoosten
body: Sure!
Done in https://github.com/zio/zio/pull/9697/commits/16ded37b7e6d95354f43d53c041ce7bfada3b2a6
created_at: 2025-03-24T17:55:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
+            val builder = ChunkBuilder.make[Out](chunk.size)
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: kyri-petrou
body: As a side-note: I wonder if we could reuse the ChunkBuilder after calling `result()` on it but maybe we can re-visit that in a followup PR.
created_at: 2025-03-28T05:24:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1798,6 +1817,49 @@ object ZPipeline extends ZPipelinePlatformSpecificConstructors {
     new ZPipeline(loop(Chunk.ChunkIterator.empty, 0))
   }
 
+  /**
+   * Creates a pipeline that maps over elements of the stream with the specified
+   * effectful function.
+   *
+   * Unlike `mapZIO` processing is done chunk by chunk. This means that
+   * `mapZIOChunked` provides weaker guarantees than `mapZIO`. While
+   * `stream.mapZIO(f).mapZIO(g)` is guaranteed to be equivalent to
+   * `stream.mapZIO(x => f(x).flatMap(g))`, the same is not true for
+   * `mapZIOChunked`. For example, `mapZIO` guarantees that the first element of
+   * a stream will first be processed with `f` and then `g` before the second
+   * element is processed with `f`. `mapZIOChunked` may process the first two
+   * elements with `f` and only then move on to process the first element with
+   * `g`.
+   */
+  def mapZIOChunked[Env, Err, In, Out](
+    f: In => ZIO[Env, Err, Out]
+  )(implicit trace: Trace): ZPipeline[Env, Err, In, Out] = {
+    def writeWithNext(
+      builder: ChunkBuilder[Out],
+      next: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any]
+    ): ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] = {
+      val out = builder.result()
+      if (out.nonEmpty) ZChannel.write(out) *> next else next
+    }
+
+    lazy val reader: ZChannel[Env, Err, Chunk[In], Any, Err, Chunk[Out], Any] =
+      ZChannel.readWithCause(
+        chunk =>
+          ZChannel.unwrap {
+            val builder = ChunkBuilder.make[Out](chunk.size)
path: streams/shared/src/main/scala/zio/stream/ZPipeline.scala
user: login: erikvanoosten
body: I looked into this a bit and I think it is not going to help. The thing is, as long as the stream is not failing or ending, we give the chunk builder a 'sizeHint' that is spot-on; the backing array will be filled to exactlty that size. You should know that if the `ArrayBuilder` fills up the array completely, it will hand out that array in `result` and then builds a new array when the builder is re-used. In other words, the array is always instantiated anew and we might as well also construct a new `ChunkBuilder` so that we get the small benefit of using memory that is local to the CPU core that this is running on.
created_at: 2025-03-30T11:08:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: calvinlfer
created_at: 2025-03-14T22:35:47Z
body: > I think this not good:
> 
> > * Unlike `mapZIO` processing is done chunk by chunk. When `f` fails for an
> > * element in the middle of a Chunk, the other elements of the chunk are lost.
> 
> If processing fails in the middle of a chunk, the stream should first emit the elements where processing succeeded and then fail the stream.

the act of doing this breaks chunking structure though
author_association: MEMBER
user: login: mberndt123
created_at: 2025-03-14T23:57:07Z
body: > the act of doing this breaks chunking structure though

No, not if you implement it right. 
author_association: CONTRIBUTOR
user: login: mberndt123
created_at: 2025-03-15T00:05:26Z
body: I've written an implementation that will retain chunking while also emitting every element that was processed:
https://github.com/mberndt123/zio/tree/mapziochunked

I'd open a merge request to @erikvanoosten's repo, but for whatever reason, I can't create a PR targeting his ZIO fork. 
author_association: CONTRIBUTOR
user: login: erikvanoosten
created_at: 2025-03-15T08:09:25Z
body: > I've written an implementation that will retain chunking while also emitting every element that was processed: https://github.com/mberndt123/zio/tree/mapziochunked
> 
> I'd open a merge request to @erikvanoosten's repo, but for whatever reason, I can't create a PR targeting his ZIO fork.

Thanks @mberndt123 ! I have cherry picked your implementation and updated the scaladoc and unit test. I was playing around with foldLeft to get the same result, but your solution is simpler.
author_association: CONTRIBUTOR
user: login: erikvanoosten
created_at: 2025-03-16T19:00:10Z
body: @regiskuckaertz Thanks for your approval. Can you merge as well please?
author_association: CONTRIBUTOR
user: login: erikvanoosten
created_at: 2025-03-17T18:05:06Z
body: For the ZIO contributor that is reviewing this. @mberndt123 proposed another solution:

```scala
    ZPipeline.chunks.mapStream { (chunk: Chunk[In]) =>
      val builder = ChunkBuilder.make[Out](chunk.size)
      def writeNonEmpty = {
        val out = builder.result()
        ZStream.when(out.nonEmpty)(ZStream.fromChunk(out))
      }
      ZStream.unwrap {
        chunk
          .mapZIODiscard(f(_).map(builder += _))
          .foldCause(
            cause => writeNonEmpty ++ ZStream.failCause(cause),
            _ => writeNonEmpty
          )
      }
 ```

Please let me know if that has preference and I'll swap it in.
author_association: CONTRIBUTOR
user: login: regiskuckaertz
created_at: 2025-03-17T19:17:34Z
body: @kyri-petrou I feel like I'm not in a position to merge anymore but I feel like @erikvanoosten has exercised all due diligence and this has my vote, for what it's worth.
author_association: MEMBER
user: login: mberndt123
created_at: 2025-03-18T10:22:35Z
body: @paulpdaniels, a thumbs-down emoji without further explanation is a bit disappointing. My implementation is significantly terser and relies on higher level combinators (i. e. `mapStream`), both of which are significant maintainability advantages. If you're going to argue performance, then I'd like to see some evidence. Using low-level `ZChannel` primitives doesn't magically make code faster. 
author_association: CONTRIBUTOR
user: login: mberndt123
created_at: 2025-03-18T11:03:41Z
body: I've done some cursory benchmarks with this thing in `StreamBenchmarks.scala`:
```scala
  @Benchmark
  def zioMapZIOChunked: Unit = {
    val stream = ZStream
      .fromChunks(zioChunks: _*)
      .mapZIOChunked(i => ZIO.succeed((i * i).toLong))
    
    val result = stream.runDrain

    unsafeRun(result)
  }

// run with `benchmarks/Jmh/run zio.StreamBenchmarks.zioMapZIOChunked`
```
I didn't see any meaningful (i. e. beyond noise) performance difference between the different implementations.
author_association: CONTRIBUTOR
user: login: paulpdaniels
created_at: 2025-03-18T16:34:57Z
body: @mberndt123 

> then I'd like to see some evidence

Running your above example with the two implementations (current vs proposed) side-by-side

```
// Using  17.0.12-amzn
// benchmarks/jmh:run .*StreamBenchmarks.zioPipelineMapChunked
chunkCount=10000 chunkSize=5000: current impl faster ~15% 
chunkCount=5000 chunkSize=5000: current impl faster ~12%
chunkCount=5000 chunkSize=50: current impl faster ~450%
chunkCount=5000 chunkSize=5: current impl faster ~1000% (10x)
chunkCount=5000 chunkSize=1: current impl faster ~1300% (13x)
```
🪄 

I haven't run a memory profile on these, but I would expect it would lean toward the current implementation as well.

> and relies on higher level combinators

This is a mistake for low-level operators, for instance `mapStream` is meant as an interop operator for user-land code that has a `Stream` but wants it to work in a pipeline without having to unpack it into a `ZChannel`. You can very quickly check these operators implementations to see the amount of extra work the operator would have to do if they were included.

> both of which are significant maintainability advantages

The current implementation is written in line with the rest of the library, consistency is just as important in maintainability.

This does open up some questions in my mind about _why_ the performance is so bad in some of the cases above (perhaps they are too naively reliant on other operators), certainly some fruitful avenues to explore on how to optimize those!

**Edit**

There is a case where the proposed is faster than the current:

```
// Seems consistently faster, though for some reason current implementation generates a lot variance with this one.
chunkCount=5000 chunkSize=0: proposed impl faster ~2%
```


author_association: CONTRIBUTOR
user: login: erikvanoosten
created_at: 2025-03-19T17:37:24Z
body: @kyri-petrou this PR is ready for merging!
author_association: CONTRIBUTOR
user: login: erikvanoosten
created_at: 2025-03-24T17:58:08Z
body: @kyri-petrou 
> Can we add a @ see [[mapZIOChunked]] to the scaladoc of mapZIO as well?

We could introduce some more documentation around these chunk-breakers on https://zio.dev as well. [This zio-kafka page](https://github.com/zio/zio-kafka/blob/master/docs/avoiding-chunk-breakers.md) could be used as a start (see the page source for hidden sections).
author_association: CONTRIBUTOR
user: login: kyri-petrou
created_at: 2025-03-28T05:18:58Z
body: > We could introduce some more documentation around these chunk-breakers on https://zio.dev/ as well

Yeah I agree. I'll create an issue for it

author_association: CONTRIBUTOR