Repository: sindresorhus/execa (JavaScript)

Issue:
Ability to prefix child process stdio lines with a custom string
Proposal: add a new option `prefix` (optional, string). If given, the child's stdout and stderr streams would be transformed so **each line** is prefixed with the given string.

Use case: when running several child processes in parallel and using `inherit` to pipe each child's stdio to the parent, prefixing would make it easier to see what's going on:

```js
Promise.all(jobs.map((job, i) => execa(job.cmd, job.args, {
  stdio: 'inherit',
  prefix: `[job ${i + 1} of jobs.length] `,
}))).then(() => {
  // ...
});
```

```
[job 1 of 4] blah blah
[job 3 of 4] blah blah
[job 3 of 4] blah blah blah
[job 2 of 4] blah
...
```

I think the best API would be to have a `prefix` option which would apply to both stdout and stderr streams, and also `prefixStdout` and `prefixStderr` in case you want to override them individually.

Happy to implement this if interested.
Line-wise transforms
## Background

This feature was suggested by both #21 and #121.
Also mentioned by @sindresorhus in https://github.com/sindresorhus/execa/pull/693#discussion_r1453380836.

## Rationale

Users can now transform `stdin`/`stdout`/`stderr`/`stdio[*]` by passing generators to those options.

The vast majority of processes' input and output is text-based. When transforming text-based data, it is very convenient to map one line at a time. For example, this is required for #121.

Currently, each chunk being transformed might be broken in the middle of a line. For example, the following does not currently work because the word `secret` might be split into 2 different chunks.

```js
const transform = async function * (chunks) {
	for await (const chunk of chunks) {
		if (!chunk.includes('secret')) {
			yield chunk;
		}
	}
};

const {stdout} = await execa('echo', ['This is a secret.'], {stdout: transform});
```

## Solution

Transforms should iterate over chunks line by line. We would use `\n` as an end delimiter (which works for `\r\n` too).

However, I think we should still allow users to opt out of this behavior because, although less common, some processes' input/output is in fact binary. Splitting binary input/output line by line would result in chunks of varying size, some potentially very large if the binary input/output does not contain any newline. This would take away the main benefits of streaming (incremental CPU load + low memory).

## API

The users would get this behavior by default:

```js
execa('...', {stdout: transform});
```

To allow users to opt-out, I would suggest the following format.

```js
execa('...', {stdout: { transform, binary: true }});
```

In other words, a transform can be either `AsyncGeneratorFunction` or `{ transform: AsyncGeneractorFunction, binary: boolean }`. 

This format is also extensible to other options.

## Alternatives

One other possible API would be:

```js
execa('...', {stdout: ['binary', transform]});
```

However, some users might want to use multiple transforms for `stdout`, with some of them being binary, others not. For example, one transform might convert a binary `stdout` to a text-based one. Then a second transform might operate line-wise. The above approach would not allow this.

```js
execa('...', {stdout: ['binary', binaryToTextTransform, secretFilterTransform]});
```

---

Yet another possible API would be:

```js
execa('...', {stdout: transform, binary: true});
```

However, this has the same problem. It also has an additional one: it would apply to all streams (`stdin`/`stdout`/`stderr`/`stdio[*]`). For example, some processes might produce a binary `stdout` but a text-based `stderr`.

```js
execa('...', {stdout: binaryTransform, stderr: textTransform, binary: true});
```

---

Implementing this should be rather simple, since splitting chunks line-wise can be done using the same kind of generator users can now use themselves. In other words, we just need to add an internal generator, like we already do for encoding the chunks.

https://github.com/sindresorhus/execa/blob/e8f6bbd65547a3b19d001fbe4c34d7ae140a1d33/lib/stdio/generator.js#L74-L85

What do you think @sindresorhus?


Discussion: It's not possible to intercept the stdout/stderr when you're using `inherit` (Really wish it was though). We could support it on `pipe`, but not sure it belongs here.
Can't we just reimplement `inherit` so it doesn't actually use the underlying cp's `inherit`? AFAICT inherit is just shorthand for `[process.stdin, process.stdout, process.stderr]`.
Relevant Node.js issue: https://github.com/nodejs/node/issues/8033





Fix (patch):
diff --git a/docs/transform.md b/docs/transform.md
index 885031e424..3305f0bf33 100644
--- a/docs/transform.md
+++ b/docs/transform.md
@@ -7,9 +7,10 @@ Transforms map or filter the input or output of a child process. They are define
 ```js
 import {execa} from 'execa';
 
-const transform = async function * (chunks) {
-	for await (const chunk of chunks) {
-		yield chunk.toUpperCase();
+const transform = async function * (lines) {
+	for await (const line of lines) {
+		const prefix = line.includes('error') ? 'ERROR' : 'INFO'
+		yield `${prefix}: ${line}`
 	}
 };
 
@@ -19,21 +20,21 @@ console.log(stdout); // HELLO
 
 ## Encoding
 
-The `chunks` argument passed to the transform is an [`AsyncIterable<string>`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols). If the [`encoding`](../readme.md#encoding) option is `buffer`, it is an `AsyncIterable<Uint8Array>` instead.
+The `lines` argument passed to the transform is an [`AsyncIterable<string>`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols). If the [`encoding`](../readme.md#encoding) option is `buffer`, it is an `AsyncIterable<Uint8Array>` instead.
 
-The transform can `yield` either a `string` or an `Uint8Array`, regardless of the `chunks` argument's type.
+The transform can `yield` either a `string` or an `Uint8Array`, regardless of the `lines` argument's type.
 
 ## Filtering
 
-`yield` can be called 0, 1 or multiple times. Not calling `yield` enables filtering a specific chunk.
+`yield` can be called 0, 1 or multiple times. Not calling `yield` enables filtering a specific line.
 
 ```js
 import {execa} from 'execa';
 
-const transform = async function * (chunks) {
-	for await (const chunk of chunks) {
-		if (!chunk.includes('secret')) {
-			yield chunk;
+const transform = async function * (lines) {
+	for await (const line of lines) {
+		if (!line.includes('secret')) {
+			yield line;
 		}
 	}
 };
@@ -42,6 +43,19 @@ const {stdout} = await execa('echo', ['This is a secret.'], {stdout: transform})
 console.log(stdout); // ''
 ```
 
+## Binary data
+
+The transform iterates over lines by default.\
+However, if a `{transform, binary: true}` plain object is passed, it iterates over arbitrary chunks of data instead.
+
+```js
+await execa('./binary.js', {stdout: {transform, binary: true}});
+```
+
+This is more efficient and recommended if the data is either:
+	- Binary: Which does not have lines.
+	- Text: But the transform works even if a line or word is split across multiple chunks.
+
 ## Combining
 
 The [`stdin`](../readme.md#stdin), [`stdout`](../readme.md#stdout-1), [`stderr`](../readme.md#stderr-1) and [`stdio`](../readme.md#stdio-1) options can accept an array of values. While this is not specific to transforms, this can be useful with them too. For example, the following transform impacts the value printed by `inherit`.
diff --git a/index.d.ts b/index.d.ts
index b53507388a..6cba55b979 100644
--- a/index.d.ts
+++ b/index.d.ts
@@ -18,6 +18,10 @@ type BaseStdioOption =
 	| 'ignore'
 	| 'inherit';
 
+// @todo Use either `Iterable<string>` or `Iterable<Uint8Array>` based on whether `encoding: 'buffer'` is used.
+// See https://github.com/sindresorhus/execa/issues/694
+type StdioTransform = ((chunks: Iterable<string | Uint8Array>) => AsyncGenerator<string | Uint8Array, void, void>);
+
 type CommonStdioOption<IsSync extends boolean = boolean> =
 	| BaseStdioOption
 	| 'ipc'
@@ -25,9 +29,12 @@ type CommonStdioOption<IsSync extends boolean = boolean> =
 	| undefined
 	| URL
 	| {file: string}
-	// TODO: Use either `Iterable<string>` or `Iterable<Uint8Array>` based on whether `encoding: 'buffer'` is used.
-	// See https://github.com/sindresorhus/execa/issues/694
-	| IfAsync<IsSync, ((chunks: Iterable<string | Uint8Array>) => AsyncGenerator<string | Uint8Array, void, void>)>;
+	| IfAsync<IsSync,
+	| StdioTransform
+	| {
+		transform: StdioTransform;
+		binary?: boolean;
+	}>;
 
 type InputStdioOption<IsSync extends boolean = boolean> =
 	| Uint8Array
diff --git a/lib/stdio/encoding.js b/lib/stdio/encoding.js
index 06802a3de0..5462e24dae 100644
--- a/lib/stdio/encoding.js
+++ b/lib/stdio/encoding.js
@@ -7,8 +7,16 @@ export const handleStreamsEncoding = (stdioStreams, {encoding}, isSync) => {
 		return stdioStreams.map(stdioStream => ({...stdioStream, encoding}));
 	}
 
-	const value = encodingEndGenerator.bind(undefined, encoding);
-	return [...stdioStreams, {...stdioStreams[0], type: 'generator', value, encoding: 'buffer'}];
+	const transform = encodingEndGenerator.bind(undefined, encoding);
+	return [
+		...stdioStreams,
+		{
+			...stdioStreams[0],
+			type: 'generator',
+			value: {transform, binary: true},
+			encoding: 'buffer',
+		},
+	];
 };
 
 // eslint-disable-next-line unicorn/text-encoding-identifier-case
diff --git a/lib/stdio/generator.js b/lib/stdio/generator.js
index fa5e3b4c51..d97227f69a 100644
--- a/lib/stdio/generator.js
+++ b/lib/stdio/generator.js
@@ -1,5 +1,7 @@
 import {generatorsToDuplex} from './duplex.js';
 import {getEncodingStartGenerator} from './encoding.js';
+import {getLinesGenerator} from './lines.js';
+import {isGeneratorOptions} from './type.js';
 
 /*
 Generators can be used to transform/filter standard streams.
@@ -20,7 +22,12 @@ We ensure `objectMode` is `false` for better buffering.
 Chunks are currently processed serially. We could add a `concurrency` option to parallelize in the future.
 */
 export const generatorToDuplexStream = ({value, encoding}) => {
-	const generators = [getEncodingStartGenerator(encoding), value];
+	const {transform, binary} = isGeneratorOptions(value) ? value : {transform: value};
+	const generators = [
+		getEncodingStartGenerator(encoding),
+		getLinesGenerator(encoding, binary),
+		transform,
+	].filter(Boolean);
 	const duplexStream = generatorsToDuplex(generators, {objectMode: false});
 	return {value: duplexStream};
 };
diff --git a/lib/stdio/lines.js b/lib/stdio/lines.js
new file mode 100644
index 0000000000..04fba4102d
--- /dev/null
+++ b/lib/stdio/lines.js
@@ -0,0 +1,57 @@
+// Split chunks line-wise
+export const getLinesGenerator = (encoding, binary) => {
+	if (binary) {
+		return;
+	}
+
+	return encoding === 'buffer' ? linesUint8ArrayGenerator : linesStringGenerator;
+};
+
+const linesUint8ArrayGenerator = async function * (chunks) {
+	yield * linesGenerator(chunks, new Uint8Array(0), 0x0A, concatUint8Array);
+};
+
+const concatUint8Array = (firstChunk, secondChunk) => {
+	const chunk = new Uint8Array(firstChunk.length + secondChunk.length);
+	chunk.set(firstChunk, 0);
+	chunk.set(secondChunk, firstChunk.length);
+	return chunk;
+};
+
+const linesStringGenerator = async function * (chunks) {
+	yield * linesGenerator(chunks, '', '\n', concatString);
+};
+
+const concatString = (firstChunk, secondChunk) => `${firstChunk}${secondChunk}`;
+
+// This imperative logic is much faster than using `String.split()` and uses very low memory.
+// Also, it allows sharing it with `Uint8Array`.
+const linesGenerator = async function * (chunks, emptyValue, newline, concat) {
+	let previousChunks = emptyValue;
+
+	for await (const chunk of chunks) {
+		let start = -1;
+
+		for (let end = 0; end < chunk.length; end += 1) {
+			if (chunk[end] === newline) {
+				let line = chunk.slice(start + 1, end + 1);
+
+				if (previousChunks.length > 0) {
+					line = concat(previousChunks, line);
+					previousChunks = emptyValue;
+				}
+
+				yield line;
+				start = end;
+			}
+		}
+
+		if (start !== chunk.length - 1) {
+			previousChunks = concat(previousChunks, chunk.slice(start + 1));
+		}
+	}
+
+	if (previousChunks.length > 0) {
+		yield previousChunks;
+	}
+};
diff --git a/lib/stdio/type.js b/lib/stdio/type.js
index 53c87ad746..28cd75b691 100644
--- a/lib/stdio/type.js
+++ b/lib/stdio/type.js
@@ -35,11 +35,30 @@ export const getStdioOptionType = (stdioOption, optionName) => {
 		return 'iterable';
 	}
 
+	if (isGeneratorOptions(stdioOption)) {
+		return getGeneratorObjectType(stdioOption, optionName);
+	}
+
 	return 'native';
 };
 
+const getGeneratorObjectType = ({transform, binary}, optionName) => {
+	if (!isAsyncGenerator(transform)) {
+		throw new TypeError(`The \`${optionName}.transform\` option must use an asynchronous generator.`);
+	}
+
+	if (binary !== undefined && typeof binary !== 'boolean') {
+		throw new TypeError(`The \`${optionName}.binary\` option must use a boolean.`);
+	}
+
+	return 'generator';
+};
+
 const isAsyncGenerator = stdioOption => Object.prototype.toString.call(stdioOption) === '[object AsyncGeneratorFunction]';
 const isSyncGenerator = stdioOption => Object.prototype.toString.call(stdioOption) === '[object GeneratorFunction]';
+export const isGeneratorOptions = stdioOption => typeof stdioOption === 'object'
+	&& stdioOption !== null
+	&& stdioOption.transform !== undefined;
 
 export const isUrl = stdioOption => Object.prototype.toString.call(stdioOption) === '[object URL]';
 export const isRegularUrl = stdioOption => isUrl(stdioOption) && stdioOption.protocol !== 'file:';
diff --git a/readme.md b/readme.md
index 5d0ba7839c..5b79233e6c 100644
--- a/readme.md
+++ b/readme.md
@@ -47,6 +47,9 @@ This package improves [`child_process`](https://nodejs.org/api/child_process.htm
 - Improved [Windows support](https://github.com/IndigoUnited/node-cross-spawn#why), including [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) binaries.
 - Executes [locally installed binaries](#preferlocal) without `npx`.
 - [Cleans up](#cleanup) child processes when the parent process ends.
+- Redirect [`stdin`](#stdin)/[`stdout`](#stdout-1)/[`stderr`](#stderr-1) to files, streams, iterables, strings or `Uint8Array`.
+- [Transform](docs/transform.md) `stdin`/`stdout`/`stderr` with simple functions.
+- Iterate over [each text line](docs/transform.md#binary-data) output by the process.
 - [Graceful termination](#optionsforcekillaftertimeout).
 - Get [interleaved output](#all) from `stdout` and `stderr` similar to what is printed on the terminal.
 - [Strips the final newline](#stripfinalnewline) from the output so you don't have to do `stdout.trim()`.