Repository: cloudflare/workers-sdk
Pull request #8753: Update flags for Pipelines create and update
Author: login: cmackenzie1
Created: 2025-04-01T20:10:14Z | Merged: 2025-04-02T19:41:28Z

Transforms are not publicly available yet so we are hiding this from the help output. Private beta users will still be able to provide this flag.

This commit also introduces a new flag `--shard-count` for power users to hand tune their pipeline.

Fixes https://jira.cfdata.org/browse/PIPE-241, https://jira.cfdata.org/browse/PIPE-242

---

<!--
Please don't delete the checkboxes <3
The following selections do not need to be completed if this PR only contains changes to .md files
-->

- Tests
  - [ ] TODO (before merge)
  - [X] Tests included
  - [ ] Tests not necessary because:
- Wrangler / Vite E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
  - [ ] I don't know
  - [ ] Required
  - [X] Not required because: not needed
- Public documentation
  - [ ] TODO (before merge)
  - [X] Cloudflare docs PR(s): https://github.com/cloudflare/cloudflare-docs/pull/18595
  - [ ] Documentation not necessary because:
- Wrangler V3 Backport
  - [ ] TODO (before merge)
  - [ ] Wrangler PR: <!--e.g. <https://github.com/cloudflare/workers-sdk/pull/>...-->
  - [X] Not necessary because: Not public yet. 

<!--
Have you read our [Contributing guide](https://github.com/cloudflare/workers-sdk/blob/main/CONTRIBUTING.md)?
In particular, for non-trivial changes, please always engage on the issue or create a discussion or feature request issue first before writing your code.
-->



Diff:
diff --git a/.changeset/wet-memes-mix.md b/.changeset/wet-memes-mix.md
new file mode 100644
index 0000000000..d19a26b5fd
--- /dev/null
+++ b/.changeset/wet-memes-mix.md
@@ -0,0 +1,7 @@
+---
+"wrangler": patch
+---
+
+- Hide `--transform-worker` flag on `wrangler pipelines <create|update>` during private beta.
+- Add `--shard-count` option for `wrangler pipelines <create|update>` for more control over Pipeline throughput or file
+  size
diff --git a/packages/wrangler/src/__tests__/pipelines.test.ts b/packages/wrangler/src/__tests__/pipelines.test.ts
index 3db3002a11..85728a0f62 100644
--- a/packages/wrangler/src/__tests__/pipelines.test.ts
+++ b/packages/wrangler/src/__tests__/pipelines.test.ts
@@ -296,18 +296,18 @@ describe("pipelines", () => {
 				      --batch-max-rows     Maximum number of rows per batch before flushing  [number]
 				      --batch-max-seconds  Maximum age of batch in seconds before flushing  [number]
 
-				Transformations
-				      --transform-worker  Pipeline transform Worker and entrypoint (<worker>.<entrypoint>)  [string]
-
 				Destination settings
 				      --r2-bucket             Destination R2 bucket name  [string] [required]
 				      --r2-access-key-id      R2 service Access Key ID for authentication. Leave empty for OAuth confirmation.  [string]
 				      --r2-secret-access-key  R2 service Secret Access Key for authentication. Leave empty for OAuth confirmation.  [string]
 				      --r2-prefix             Prefix for storing files in the destination bucket  [string] [default: \\"\\"]
 				      --compression           Compression format for output files  [string] [choices: \\"none\\", \\"gzip\\", \\"deflate\\"] [default: \\"gzip\\"]
-				      --file-template         Template for individual file names (must include \${slug})  [string]
+				      --file-template         Template for individual file names (must include \${slug}). For example: \\"\${slug}.log.gz\\"  [string]
 				      --partition-template    Path template for partitioned files in the bucket. If not specified, the default will be used  [string]
 
+				Pipeline settings
+				      --shard-count  Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files  [number]
+
 				GLOBAL FLAGS
 				  -c, --config   Path to Wrangler configuration file  [string]
 				      --cwd      Run as if Wrangler was started in the specified directory instead of the current working directory  [string]
diff --git a/packages/wrangler/src/pipelines/cli/create.ts b/packages/wrangler/src/pipelines/cli/create.ts
index dbb5e17f31..d430a3aa79 100644
--- a/packages/wrangler/src/pipelines/cli/create.ts
+++ b/packages/wrangler/src/pipelines/cli/create.ts
@@ -96,6 +96,7 @@ export function addCreateOptions(yargs: Argv<CommonYargsOptions>) {
 				describe:
 					"Pipeline transform Worker and entrypoint (<worker>.<entrypoint>)",
 				demandOption: false,
+				hidden: true, // TODO: Remove once transformations launch
 			})
 
 			// Destination options
@@ -163,7 +164,7 @@ export function addCreateOptions(yargs: Argv<CommonYargsOptions>) {
 			})
 			.option("file-template", {
 				type: "string",
-				describe: "Template for individual file names (must include ${slug})",
+				describe: `Template for individual file names (must include $\{slug}). For example: "$\{slug}.log.gz"`,
 				demandOption: false,
 				coerce: (val) => {
 					if (!val.includes("${slug}")) {
@@ -172,6 +173,15 @@ export function addCreateOptions(yargs: Argv<CommonYargsOptions>) {
 					return val;
 				},
 			})
+
+			// Pipeline settings
+			.group(["shard-count"], `${chalk.bold("Pipeline settings")}`)
+			.option("shard-count", {
+				type: "number",
+				describe:
+					"Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files",
+				demandOption: false,
+			})
 	);
 }
 
@@ -281,6 +291,10 @@ export async function createPipelineHandler(
 		pipelineConfig.destination.path.filename = args.fileTemplate;
 	}
 
+	if (args.shardCount) {
+		pipelineConfig.metadata.shards = args.shardCount;
+	}
+
 	logger.log(`🌀 Creating Pipeline named "${name}"`);
 	const pipeline = await createPipeline(accountId, pipelineConfig);
 
diff --git a/packages/wrangler/src/pipelines/cli/update.ts b/packages/wrangler/src/pipelines/cli/update.ts
index 4b42b805ba..a0615212e1 100644
--- a/packages/wrangler/src/pipelines/cli/update.ts
+++ b/packages/wrangler/src/pipelines/cli/update.ts
@@ -106,6 +106,7 @@ export function addUpdateOptions(yargs: Argv<CommonYargsOptions>) {
 				describe:
 					'Pipeline transform Worker and entrypoint, to transform ingested records. Specified as <worker-name>.<entrypoint>, or "none".',
 				demandOption: false,
+				hidden: true, // TODO: Remove once transformations launch
 			})
 
 			// Destination options
@@ -174,6 +175,15 @@ export function addUpdateOptions(yargs: Argv<CommonYargsOptions>) {
 					return val;
 				},
 			})
+
+			// Pipeline settings
+			.group(["shard-count"], `${chalk.bold("Pipeline settings")}`)
+			.option("shard-count", {
+				type: "number",
+				describe:
+					"Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files",
+				demandOption: false,
+			})
 	);
 }
 
@@ -300,6 +310,10 @@ export async function updatePipelineHandler(
 		pipelineConfig.destination.path.filename = args.fileTemplate;
 	}
 
+	if (args.shardCount) {
+		pipelineConfig.metadata.shards = args.shardCount;
+	}
+
 	logger.log(`🌀 Updating Pipeline "${name}"`);
 	const pipeline = await updatePipeline(accountId, name, pipelineConfig);
 
diff --git a/packages/wrangler/src/pipelines/client.ts b/packages/wrangler/src/pipelines/client.ts
index 0d3413ffb6..2fc889eb12 100644
--- a/packages/wrangler/src/pipelines/client.ts
+++ b/packages/wrangler/src/pipelines/client.ts
@@ -11,9 +11,12 @@ import type { R2BucketInfo } from "../r2/helpers";
 
 // ensure this is in sync with:
 //   https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6
-type RecursivePartial<T> = {
-	[P in keyof T]?: RecursivePartial<T[P]>;
-};
+type RecursivePartial<T> = T extends object
+	? {
+			[P in keyof T]?: RecursivePartial<T[P]>;
+		}
+	: T;
+
 export type PartialExcept<T, K extends keyof T> = RecursivePartial<T> &
 	Pick<T, K>;
 
@@ -21,6 +24,7 @@ export type TransformConfig = {
 	script: string;
 	entrypoint: string;
 };
+
 export type HttpSource = {
 	type: "http";
 	format: string;
@@ -30,15 +34,23 @@ export type HttpSource = {
 		origins: ["*"] | string[];
 	};
 };
+
 export type BindingSource = {
 	type: "binding";
 	format: string;
 	schema?: string;
 };
+
+export type Metadata = {
+	shards?: number;
+	[x: string]: unknown;
+};
+
 export type Source = HttpSource | BindingSource;
+
 export type PipelineUserConfig = {
 	name: string;
-	metadata: { [x: string]: string };
+	metadata: Metadata;
 	source: Source[];
 	transforms: TransformConfig[];
 	destination: {


Reviews:
user: login: oliy
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T22:23:47Z
user: login: petebacondarwin
body: One question about default values for shard count. Othewise LGTM.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:08:18Z
user: login: cmackenzie1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T16:01:39Z
user: login: cmackenzie1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T16:02:38Z
user: login: cmackenzie1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T16:04:45Z
user: login: petebacondarwin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T19:41:21Z

Review comments:
diff_hunk: @@ -281,6 +291,10 @@ export async function createPipelineHandler(
 		pipelineConfig.destination.path.filename = args.fileTemplate;
 	}
 
+	if (args.shardCount) {
path: packages/wrangler/src/pipelines/cli/create.ts
user: login: petebacondarwin
body: This will treat `undefined` and `0` the same. I assume this is what you want?
created_at: 2025-04-02T15:06:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +173,15 @@ export function addCreateOptions(yargs: Argv<CommonYargsOptions>) {
 					return val;
 				},
 			})
+
+			// Pipeline settings
+			.group(["shard-count"], `${chalk.bold("Pipeline settings")}`)
+			.option("shard-count", {
+				type: "number",
+				describe:
+					"Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files",
path: packages/wrangler/src/pipelines/cli/create.ts
user: login: petebacondarwin
body: What is the default number of shards? One?
created_at: 2025-04-02T15:06:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -11,16 +11,20 @@ import type { R2BucketInfo } from "../r2/helpers";
 
 // ensure this is in sync with:
 //   https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6
path: packages/wrangler/src/pipelines/client.ts
user: login: petebacondarwin
body: Is this still bitbucket or gitlab?
created_at: 2025-04-02T15:07:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,6 +291,10 @@ export async function createPipelineHandler(
 		pipelineConfig.destination.path.filename = args.fileTemplate;
 	}
 
+	if (args.shardCount) {
path: packages/wrangler/src/pipelines/cli/create.ts
user: login: cmackenzie1
body: Yeah, this is fine. 0 is not an allowed value for number of shards.
created_at: 2025-04-02T16:01:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +173,15 @@ export function addCreateOptions(yargs: Argv<CommonYargsOptions>) {
 					return val;
 				},
 			})
+
+			// Pipeline settings
+			.group(["shard-count"], `${chalk.bold("Pipeline settings")}`)
+			.option("shard-count", {
+				type: "number",
+				describe:
+					"Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files",
path: packages/wrangler/src/pipelines/cli/create.ts
user: login: cmackenzie1
body: I think 2, but that may change. We are trying to avoid baking defaults into wrangler and let the API set them instead. 
created_at: 2025-04-02T16:02:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -11,16 +11,20 @@ import type { R2BucketInfo } from "../r2/helpers";
 
 // ensure this is in sync with:
 //   https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6
path: packages/wrangler/src/pipelines/client.ts
user: login: cmackenzie1
body: GitLab now. What is the policy on linking to internal codebases in a public project?
created_at: 2025-04-02T16:04:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -11,16 +11,20 @@ import type { R2BucketInfo } from "../r2/helpers";
 
 // ensure this is in sync with:
 //   https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6
path: packages/wrangler/src/pipelines/client.ts
user: login: petebacondarwin
body: It's generally fine I believe since it is all locked behind Access.
But no need to update if you don't want to.
created_at: 2025-04-02T19:41:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-04-01T20:10:19Z
body: ###  🦋  Changeset detected

Latest commit: 08b6f8d0cd729b07afc79ad29af98d724cdc02c0

**The changes in this PR will be included in the next version bump.**

<details><summary>This PR includes changesets to release 2 packages</summary>

  | Name                            | Type  |
| ------------------------------- | ----- |
| wrangler                        | Patch |
| @cloudflare/vitest-pool-workers | Patch |

</details>

Not sure what this means? [Click here  to learn what changesets are](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add another changeset to this PR](https://github.com/cloudflare/workers-sdk/new/cole/pipelines-dev-week?filename=.changeset/heavy-avocados-protect.md&value=---%0A%22wrangler%22%3A%20patch%0A---%0A%0AUpdate%20flags%20for%20Pipelines%20create%20and%20update%0A)


author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T20:59:28Z
body: A wrangler prerelease is available for testing. You can install this latest build in your project with:

```sh
npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-wrangler-8753
```

You can reference the automatically updated head of this PR with:

```sh
npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/8753/npm-package-wrangler-8753
```

Or you can use `npx` with this latest build directly:

```sh
npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-wrangler-8753 dev path/to/script.js
```

<details><summary>Additional artifacts:</summary>


cloudflare-workers-bindings-extension:
```sh
wget https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-bindings-extension-8753 -O ./cloudflare-workers-bindings-extension.0.0.0-v705539487.vsix && code --install-extension ./cloudflare-workers-bindings-extension.0.0.0-v705539487.vsix
```



create-cloudflare:
```sh
npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-create-cloudflare-8753 --no-auto-update
```



@cloudflare/kv-asset-handler:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-kv-asset-handler-8753
```



miniflare:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-miniflare-8753
```



@cloudflare/pages-shared:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-pages-shared-8753
```



@cloudflare/unenv-preset:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-unenv-preset-8753
```



@cloudflare/vite-plugin:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-vite-plugin-8753
```



@cloudflare/vitest-pool-workers:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-vitest-pool-workers-8753
```



@cloudflare/workers-editor-shared:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-editor-shared-8753
```



@cloudflare/workers-shared:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-shared-8753
```



@cloudflare/workflows-shared:
```sh
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workflows-shared-8753
```


Note that these links will no longer work once [the GitHub Actions artifact expires](https://docs.github.com/en/organizations/managing-organization-settings/configuring-the-retention-period-for-github-actions-artifacts-and-logs-in-your-organization).

</details>

---

`wrangler@4.6.0` includes the following runtime dependencies:

|Package|Constraint|Resolved|
|-------|----------|--------|
|`miniflare`|workspace:*|4.20250321.1|
|`workerd`|1.20250321.0|1.20250321.0|
|`workerd --version`|1.20250321.0|2025-03-21|

Please ensure constraints are pinned, and `miniflare`/`workerd` minor versions match.
<!-- Sticky Pull Request Comment -->
author_association: CONTRIBUTOR