Repository: openzfs/zfs
Pull request #17246: Introduce zfs rewrite subcommand
Author: login: amotin
Labels: name: Status: Code Review Needed
description: Ready for review and testing
Created: 2025-04-15T20:33:49Z | Merged: 2025-05-12T17:22:17Z

### Motivation and Context
For years users were asking for an ability to re-balance pool after vdev addition, de-fragment randomly written files, change some properties for already written files, etc.  The closest option would be to either copy and rename a file or send/receive/rename the dataset.  Unfortunately all of those options have some downsides.

### Description
This change introduces new `zfs rewrite` subcommand, that allows to rewrite content of specified file(s) as-is without modifications, but at a different location, compression, checksum, dedup, copies and other parameter values.  It is faster than read plus write, since it does not require data copying to user-space. It is also faster for sync=always datasets, since without data modification it does not require ZIL writing.  Also since it is protected by normal range range locks, it can be done under any other load.  Also it does not affect file's modification time or other properties.

### How Has This Been Tested?
Manually tested it on FreeBSD.  Linux-specific code is not yet tested.

### Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Performance enhancement (non-breaking change which improves efficiency)
- [ ] Code cleanup (non-breaking change which makes code smaller or more readable)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Library ABI change (libzfs, libzfs\_core, libnvpair, libuutil and libzfsbootenv)
- [ ] Documentation (a change to man pages or other documentation)

### Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the OpenZFS [code style requirements](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#coding-conventions).
- [ ] I have updated the documentation accordingly.
- [ ] I have read the [**contributing** document](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md).
- [ ] I have added [tests](https://github.com/openzfs/zfs/tree/master/tests) to cover my changes.
- [ ] I have run the ZFS Test Suite with this change applied.
- [x] All commit messages are properly formatted and contain [`Signed-off-by`](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#signed-off-by).



Diff:
diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c
index d706ef5fc38c..665e183485f0 100644
--- a/cmd/zfs/zfs_main.c
+++ b/cmd/zfs/zfs_main.c
@@ -37,6 +37,7 @@
 #include <assert.h>
 #include <ctype.h>
 #include <sys/debug.h>
+#include <dirent.h>
 #include <errno.h>
 #include <getopt.h>
 #include <libgen.h>
@@ -121,6 +122,7 @@ static int zfs_do_change_key(int argc, char **argv);
 static int zfs_do_project(int argc, char **argv);
 static int zfs_do_version(int argc, char **argv);
 static int zfs_do_redact(int argc, char **argv);
+static int zfs_do_rewrite(int argc, char **argv);
 static int zfs_do_wait(int argc, char **argv);
 
 #ifdef __FreeBSD__
@@ -193,6 +195,7 @@ typedef enum {
 	HELP_CHANGE_KEY,
 	HELP_VERSION,
 	HELP_REDACT,
+	HELP_REWRITE,
 	HELP_JAIL,
 	HELP_UNJAIL,
 	HELP_WAIT,
@@ -227,7 +230,7 @@ static zfs_command_t command_table[] = {
 	{ "promote",	zfs_do_promote,		HELP_PROMOTE		},
 	{ "rename",	zfs_do_rename,		HELP_RENAME		},
 	{ "bookmark",	zfs_do_bookmark,	HELP_BOOKMARK		},
-	{ "program",    zfs_do_channel_program, HELP_CHANNEL_PROGRAM    },
+	{ "diff",	zfs_do_diff,		HELP_DIFF		},
 	{ NULL },
 	{ "list",	zfs_do_list,		HELP_LIST		},
 	{ NULL },
@@ -249,27 +252,31 @@ static zfs_command_t command_table[] = {
 	{ NULL },
 	{ "send",	zfs_do_send,		HELP_SEND		},
 	{ "receive",	zfs_do_receive,		HELP_RECEIVE		},
+	{ "redact",	zfs_do_redact,		HELP_REDACT		},
 	{ NULL },
 	{ "allow",	zfs_do_allow,		HELP_ALLOW		},
-	{ NULL },
 	{ "unallow",	zfs_do_unallow,		HELP_UNALLOW		},
 	{ NULL },
 	{ "hold",	zfs_do_hold,		HELP_HOLD		},
 	{ "holds",	zfs_do_holds,		HELP_HOLDS		},
 	{ "release",	zfs_do_release,		HELP_RELEASE		},
-	{ "diff",	zfs_do_diff,		HELP_DIFF		},
+	{ NULL },
 	{ "load-key",	zfs_do_load_key,	HELP_LOAD_KEY		},
 	{ "unload-key",	zfs_do_unload_key,	HELP_UNLOAD_KEY		},
 	{ "change-key",	zfs_do_change_key,	HELP_CHANGE_KEY		},
-	{ "redact",	zfs_do_redact,		HELP_REDACT		},
+	{ NULL },
+	{ "program",	zfs_do_channel_program,	HELP_CHANNEL_PROGRAM	},
+	{ "rewrite",	zfs_do_rewrite,		HELP_REWRITE		},
 	{ "wait",	zfs_do_wait,		HELP_WAIT		},
 
 #ifdef __FreeBSD__
+	{ NULL },
 	{ "jail",	zfs_do_jail,		HELP_JAIL		},
 	{ "unjail",	zfs_do_unjail,		HELP_UNJAIL		},
 #endif
 
 #ifdef __linux__
+	{ NULL },
 	{ "zone",	zfs_do_zone,		HELP_ZONE		},
 	{ "unzone",	zfs_do_unzone,		HELP_UNZONE		},
 #endif
@@ -432,6 +439,9 @@ get_usage(zfs_help_t idx)
 	case HELP_REDACT:
 		return (gettext("\tredact <snapshot> <bookmark> "
 		    "<redaction_snapshot> ...\n"));
+	case HELP_REWRITE:
+		return (gettext("\trewrite [-rvx] [-o <offset>] [-l <length>] "
+		    "<directory|file ...>\n"));
 	case HELP_JAIL:
 		return (gettext("\tjail <jailid|jailname> <filesystem>\n"));
 	case HELP_UNJAIL:
@@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (lstat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
+			if (err)
+				ret = err;
+		} else { /* DT_DIR */
+			fnvlist_add_uint64(dirs, fullname, dev);
+		}
+
+		free(fullname);
+	}
+
+	closedir(dir);
+	return (ret);
+}
+
+static int
+zfs_rewrite_path(const char *path, boolean_t verbose, boolean_t recurse,
+    boolean_t xdev, zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct stat st;
+	int ret = 0;
+
+	if (lstat(path, &st) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to stat %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (S_ISREG(st.st_mode)) {
+		ret = zfs_rewrite_file(path, verbose, args);
+	} else if (S_ISDIR(st.st_mode) && recurse) {
+		ret = zfs_rewrite_dir(path, verbose, xdev, st.st_dev, args,
+		    dirs);
+	}
+	return (ret);
+}
+
+static int
+zfs_do_rewrite(int argc, char **argv)
+{
+	int ret = 0, err, c;
+	boolean_t recurse = B_FALSE, verbose = B_FALSE, xdev = B_FALSE;
+
+	if (argc < 2)
+		usage(B_FALSE);
+
+	zfs_rewrite_args_t args;
+	memset(&args, 0, sizeof (args));
+
+	while ((c = getopt(argc, argv, "l:o:rvx")) != -1) {
+		switch (c) {
+		case 'l':
+			args.len = strtoll(optarg, NULL, 0);
+			break;
+		case 'o':
+			args.off = strtoll(optarg, NULL, 0);
+			break;
+		case 'r':
+			recurse = B_TRUE;
+			break;
+		case 'v':
+			verbose = B_TRUE;
+			break;
+		case 'x':
+			xdev = B_TRUE;
+			break;
+		default:
+			(void) fprintf(stderr, gettext("invalid option '%c'\n"),
+			    optopt);
+			usage(B_FALSE);
+		}
+	}
+
+	argv += optind;
+	argc -= optind;
+	if (argc == 0) {
+		(void) fprintf(stderr,
+		    gettext("missing file or directory target(s)\n"));
+		usage(B_FALSE);
+	}
+
+	nvlist_t *dirs = fnvlist_alloc();
+	for (int i = 0; i < argc; i++) {
+		err = zfs_rewrite_path(argv[i], verbose, recurse, xdev, &args,
+		    dirs);
+		if (err)
+			ret = err;
+	}
+	nvpair_t *dir;
+	while ((dir = nvlist_next_nvpair(dirs, NULL)) != NULL) {
+		err = zfs_rewrite_dir(nvpair_name(dir), verbose, xdev,
+		    fnvpair_value_uint64(dir), &args, dirs);
+		if (err)
+			ret = err;
+		fnvlist_remove_nvpair(dirs, dir);
+	}
+	fnvlist_free(dirs);
+
+	return (ret);
+}
+
 static int
 zfs_do_wait(int argc, char **argv)
 {
diff --git a/contrib/debian/openzfs-zfsutils.install b/contrib/debian/openzfs-zfsutils.install
index 546745930bff..4573cc77ea74 100644
--- a/contrib/debian/openzfs-zfsutils.install
+++ b/contrib/debian/openzfs-zfsutils.install
@@ -73,6 +73,7 @@ usr/share/man/man8/zfs-recv.8
 usr/share/man/man8/zfs-redact.8
 usr/share/man/man8/zfs-release.8
 usr/share/man/man8/zfs-rename.8
+usr/share/man/man8/zfs-rewrite.8
 usr/share/man/man8/zfs-rollback.8
 usr/share/man/man8/zfs-send.8
 usr/share/man/man8/zfs-set.8
diff --git a/include/sys/fs/zfs.h b/include/sys/fs/zfs.h
index 44d63e8708cb..c8deb5be419e 100644
--- a/include/sys/fs/zfs.h
+++ b/include/sys/fs/zfs.h
@@ -1620,6 +1620,15 @@ typedef enum zfs_ioc {
 
 #endif
 
+typedef struct zfs_rewrite_args {
+	uint64_t	off;
+	uint64_t	len;
+	uint64_t	flags;
+	uint64_t	arg;
+} zfs_rewrite_args_t;
+
+#define	ZFS_IOC_REWRITE		_IOW(0x83, 3, zfs_rewrite_args_t)
+
 /*
  * ZFS-specific error codes used for returning descriptive errors
  * to the userland through zfs ioctls.
diff --git a/include/sys/zfs_vnops.h b/include/sys/zfs_vnops.h
index 21f0da4fe6b4..08cf0e2a6e48 100644
--- a/include/sys/zfs_vnops.h
+++ b/include/sys/zfs_vnops.h
@@ -40,6 +40,7 @@ extern int zfs_clone_range(znode_t *, uint64_t *, znode_t *, uint64_t *,
     uint64_t *, cred_t *);
 extern int zfs_clone_range_replay(znode_t *, uint64_t, uint64_t, uint64_t,
     const blkptr_t *, size_t);
+extern int zfs_rewrite(znode_t *, uint64_t, uint64_t, uint64_t, uint64_t);
 
 extern int zfs_getsecattr(znode_t *, vsecattr_t *, int, cred_t *);
 extern int zfs_setsecattr(znode_t *, vsecattr_t *, int, cred_t *);
diff --git a/man/Makefile.am b/man/Makefile.am
index fde704933764..6a7b2d3e46b7 100644
--- a/man/Makefile.am
+++ b/man/Makefile.am
@@ -50,6 +50,7 @@ dist_man_MANS = \
 	%D%/man8/zfs-redact.8 \
 	%D%/man8/zfs-release.8 \
 	%D%/man8/zfs-rename.8 \
+	%D%/man8/zfs-rewrite.8 \
 	%D%/man8/zfs-rollback.8 \
 	%D%/man8/zfs-send.8 \
 	%D%/man8/zfs-set.8 \
diff --git a/man/man8/zfs-rewrite.8 b/man/man8/zfs-rewrite.8
new file mode 100644
index 000000000000..423d6d439e28
--- /dev/null
+++ b/man/man8/zfs-rewrite.8
@@ -0,0 +1,76 @@
+.\" SPDX-License-Identifier: CDDL-1.0
+.\"
+.\" CDDL HEADER START
+.\"
+.\" The contents of this file are subject to the terms of the
+.\" Common Development and Distribution License (the "License").
+.\" You may not use this file except in compliance with the License.
+.\"
+.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+.\" or https://opensource.org/licenses/CDDL-1.0.
+.\" See the License for the specific language governing permissions
+.\" and limitations under the License.
+.\"
+.\" When distributing Covered Code, include this CDDL HEADER in each
+.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+.\" If applicable, add the following below this CDDL HEADER, with the
+.\" fields enclosed by brackets "[]" replaced with your own identifying
+.\" information: Portions Copyright [yyyy] [name of copyright owner]
+.\"
+.\" CDDL HEADER END
+.\"
+.\" Copyright (c) 2025 iXsystems, Inc.
+.\"
+.Dd May 6, 2025
+.Dt ZFS-REWRITE 8
+.Os
+.
+.Sh NAME
+.Nm zfs-rewrite
+.Nd rewrite specified files without modification
+.Sh SYNOPSIS
+.Nm zfs
+.Cm rewrite
+.Oo Fl rvx Ns Oc
+.Op Fl l Ar length
+.Op Fl o Ar offset
+.Ar file Ns | Ns Ar directory Ns …
+.
+.Sh DESCRIPTION
+Rewrite blocks of specified
+.Ar file
+as is without modification at a new location and possibly with new
+properties, such as checksum, compression, dedup, copies, etc,
+as if they were atomically read and written back.
+.Bl -tag -width "-r"
+.It Fl l Ar length
+Rewrite at most this number of bytes.
+.It Fl o Ar offset
+Start at this offset in bytes.
+.It Fl r
+Recurse into directories.
+.It Fl v
+Print names of all successfully rewritten files.
+.It Fl x
+Don't cross file system mount points when recursing.
+.El
+.Sh NOTES
+Rewrite of cloned blocks and blocks that are part of any snapshots,
+same as some property changes may increase pool space usage.
+Holes that were never written or were previously zero-compressed are
+not rewritten and will remain holes even if compression is disabled.
+.Pp
+Rewritten blocks will be seen as modified in next snapshot and as such
+included into the incremental
+.Nm zfs Cm send
+stream.
+.Pp
+If a
+.Fl l
+or
+.Fl o
+value request a rewrite to regions past the end of the file, then those
+regions are silently ignored, and no error is reported.
+.
+.Sh SEE ALSO
+.Xr zfsprops 7
diff --git a/man/man8/zfs.8 b/man/man8/zfs.8
index 5bdeb7f9e455..e16a3a82b672 100644
--- a/man/man8/zfs.8
+++ b/man/man8/zfs.8
@@ -37,7 +37,7 @@
 .\" Copyright 2018 Nexenta Systems, Inc.
 .\" Copyright 2019 Joyent, Inc.
 .\"
-.Dd May 12, 2022
+.Dd April 18, 2025
 .Dt ZFS 8
 .Os
 .
@@ -299,6 +299,12 @@ Execute ZFS administrative operations
 programmatically via a Lua script-language channel program.
 .El
 .
+.Ss Data rewrite
+.Bl -tag -width ""
+.It Xr zfs-rewrite 8
+Rewrite specified files without modification.
+.El
+.
 .Ss Jails
 .Bl -tag -width ""
 .It Xr zfs-jail 8
diff --git a/module/os/freebsd/zfs/zfs_vnops_os.c b/module/os/freebsd/zfs/zfs_vnops_os.c
index b2080a48c4ad..0fa2003554cc 100644
--- a/module/os/freebsd/zfs/zfs_vnops_os.c
+++ b/module/os/freebsd/zfs/zfs_vnops_os.c
@@ -305,6 +305,18 @@ zfs_ioctl(vnode_t *vp, ulong_t com, intptr_t data, int flag, cred_t *cred,
 		*(offset_t *)data = off;
 		return (0);
 	}
+	case ZFS_IOC_REWRITE: {
+		zfs_rewrite_args_t *args = (zfs_rewrite_args_t *)data;
+		if ((flag & FWRITE) == 0)
+			return (SET_ERROR(EBADF));
+		error = vn_lock(vp, LK_SHARED);
+		if (error)
+			return (error);
+		error = zfs_rewrite(VTOZ(vp), args->off, args->len,
+		    args->flags, args->arg);
+		VOP_UNLOCK(vp);
+		return (error);
+	}
 	}
 	return (SET_ERROR(ENOTTY));
 }
diff --git a/module/os/linux/zfs/zpl_file.c b/module/os/linux/zfs/zpl_file.c
index 33307eedbf86..84ce7cdf0bbe 100644
--- a/module/os/linux/zfs/zpl_file.c
+++ b/module/os/linux/zfs/zpl_file.c
@@ -986,6 +986,27 @@ zpl_ioctl_setdosflags(struct file *filp, void __user *arg)
 	return (err);
 }
 
+static int
+zpl_ioctl_rewrite(struct file *filp, void __user *arg)
+{
+	struct inode *ip = file_inode(filp);
+	zfs_rewrite_args_t args;
+	fstrans_cookie_t cookie;
+	int err;
+
+	if (copy_from_user(&args, arg, sizeof (args)))
+		return (-EFAULT);
+
+	if (unlikely(!(filp->f_mode & FMODE_WRITE)))
+		return (-EBADF);
+
+	cookie = spl_fstrans_mark();
+	err = -zfs_rewrite(ITOZ(ip), args.off, args.len, args.flags, args.arg);
+	spl_fstrans_unmark(cookie);
+
+	return (err);
+}
+
 static long
 zpl_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
@@ -1004,6 +1025,8 @@ zpl_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return (zpl_ioctl_getdosflags(filp, (void *)arg));
 	case ZFS_IOC_SETDOSFLAGS:
 		return (zpl_ioctl_setdosflags(filp, (void *)arg));
+	case ZFS_IOC_REWRITE:
+		return (zpl_ioctl_rewrite(filp, (void *)arg));
 	default:
 		return (-ENOTTY);
 	}
diff --git a/module/zfs/zfs_vnops.c b/module/zfs/zfs_vnops.c
index afd9e61313a9..29143a057cc7 100644
--- a/module/zfs/zfs_vnops.c
+++ b/module/zfs/zfs_vnops.c
@@ -1050,6 +1050,143 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *		arg	- flags-specific argument.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags,
+    uint64_t arg)
+{
+	int error;
+
+	if (flags != 0 || arg != 0)
+		return (SET_ERROR(EINVAL));
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skipping any holes.  This might
+		 * be inaccurate for dirty files, but we don't really care.
+		 */
+		if (noff == off) {
+			/* Find next data in the file. */
+			error = dnode_next_offset(dn, 0, &noff, 1, 1, 0);
+			if (error || noff >= off + len) {
+				if (error == ESRCH)	/* No more data. */
+					error = 0;
+				break;
+			}
+			ASSERT3U(noff, >=, off);
+			len -= noff - off;
+			off = noff;
+
+			/* Find where the data end. */
+			error = dnode_next_offset(dn, DNODE_FIND_HOLE, &noff,
+			    1, 1, 0);
+			if (error != 0)
+				noff = off + len;
+		}
+		ASSERT3U(noff, >, off);
+
+		if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) ||
+		    zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) ||
+		    (projid != ZFS_DEFAULT_PROJID &&
+		    zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
+		    projid))) {
+			error = SET_ERROR(EDQUOT);
+			break;
+		}
+
+		n = MIN(MIN(len, noff - off),
+		    DMU_MAX_ACCESS / 2 - P2PHASE(off, zp->z_blksz));
+
+		dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
+		dmu_tx_hold_write_by_dnode(tx, dn, off, n);
+		error = dmu_tx_assign(tx, DMU_TX_WAIT);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+
+		/* Mark all dbufs within range as dirty to trigger rewrite. */
+		dmu_buf_t **dbp;
+		int numbufs;
+		error = dmu_buf_hold_array_by_dnode(dn, off, n, TRUE, FTAG,
+		    &numbufs, &dbp, DMU_READ_PREFETCH);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+		for (int i = 0; i < numbufs; i++) {
+			nr += dbp[i]->db_size;
+			if (dmu_buf_is_dirty(dbp[i], tx))
+				continue;
+			nw += dbp[i]->db_size;
+			dmu_buf_will_dirty(dbp[i], tx);
+		}
+		dmu_buf_rele_array(dbp, numbufs, FTAG);
+
+		dmu_tx_commit(tx);
+
+		len -= n;
+		off += n;
+
+		if (issig()) {
+			error = SET_ERROR(EINTR);
+			break;
+		}
+	}
+
+	DB_DNODE_EXIT(db);
+
+	dataset_kstats_update_read_kstats(&zfsvfs->z_kstat, nr);
+	dataset_kstats_update_write_kstats(&zfsvfs->z_kstat, nw);
+
+	zfs_rangelock_exit(lr);
+	zfs_exit(zfsvfs, FTAG);
+	return (error);
+}
+
 int
 zfs_getsecattr(znode_t *zp, vsecattr_t *vsecp, int flag, cred_t *cr)
 {
diff --git a/tests/runfiles/common.run b/tests/runfiles/common.run
index d7f3c75c7948..9ea511dec0eb 100644
--- a/tests/runfiles/common.run
+++ b/tests/runfiles/common.run
@@ -306,6 +306,10 @@ tags = ['functional', 'cli_root', 'zfs_rename']
 tests = ['zfs_reservation_001_pos', 'zfs_reservation_002_pos']
 tags = ['functional', 'cli_root', 'zfs_reservation']
 
+[tests/functional/cli_root/zfs_rewrite]
+tests = ['zfs_rewrite']
+tags = ['functional', 'cli_root', 'zfs_rewrite']
+
 [tests/functional/cli_root/zfs_rollback]
 tests = ['zfs_rollback_001_pos', 'zfs_rollback_002_pos',
     'zfs_rollback_003_neg', 'zfs_rollback_004_neg']
diff --git a/tests/runfiles/sanity.run b/tests/runfiles/sanity.run
index 6362a2606260..9483eac40efb 100644
--- a/tests/runfiles/sanity.run
+++ b/tests/runfiles/sanity.run
@@ -194,6 +194,10 @@ tags = ['functional', 'cli_root', 'zfs_rename']
 tests = ['zfs_reservation_001_pos', 'zfs_reservation_002_pos']
 tags = ['functional', 'cli_root', 'zfs_reservation']
 
+[tests/functional/cli_root/zfs_rewrite]
+tests = ['zfs_rewrite']
+tags = ['functional', 'cli_root', 'zfs_rewrite']
+
 [tests/functional/cli_root/zfs_rollback]
 tests = ['zfs_rollback_003_neg', 'zfs_rollback_004_neg']
 tags = ['functional', 'cli_root', 'zfs_rollback']
diff --git a/tests/zfs-tests/tests/Makefile.am b/tests/zfs-tests/tests/Makefile.am
index 4c102b3aa1b8..acff5e57db93 100644
--- a/tests/zfs-tests/tests/Makefile.am
+++ b/tests/zfs-tests/tests/Makefile.am
@@ -862,6 +862,9 @@ nobase_dist_datadir_zfs_tests_tests_SCRIPTS += \
 	functional/cli_root/zfs_reservation/setup.ksh \
 	functional/cli_root/zfs_reservation/zfs_reservation_001_pos.ksh \
 	functional/cli_root/zfs_reservation/zfs_reservation_002_pos.ksh \
+	functional/cli_root/zfs_rewrite/cleanup.ksh \
+	functional/cli_root/zfs_rewrite/setup.ksh \
+	functional/cli_root/zfs_rewrite/zfs_rewrite.ksh \
 	functional/cli_root/zfs_rollback/cleanup.ksh \
 	functional/cli_root/zfs_rollback/setup.ksh \
 	functional/cli_root/zfs_rollback/zfs_rollback_001_pos.ksh \
diff --git a/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/cleanup.ksh b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/cleanup.ksh
new file mode 100755
index 000000000000..5e73dd34936e
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/cleanup.ksh
@@ -0,0 +1,26 @@
+#!/bin/ksh -p
+# SPDX-License-Identifier: CDDL-1.0
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or https://opensource.org/licenses/CDDL-1.0.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+default_cleanup
diff --git a/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/setup.ksh b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/setup.ksh
new file mode 100755
index 000000000000..dddfdf8a4679
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/setup.ksh
@@ -0,0 +1,28 @@
+#!/bin/ksh -p
+# SPDX-License-Identifier: CDDL-1.0
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or https://opensource.org/licenses/CDDL-1.0.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+DISK=${DISKS%% *}
+
+default_setup $DISK
diff --git a/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/zfs_rewrite.ksh b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/zfs_rewrite.ksh
new file mode 100755
index 000000000000..d1c0b3c64c27
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/cli_root/zfs_rewrite/zfs_rewrite.ksh
@@ -0,0 +1,104 @@
+#!/bin/ksh -p
+# SPDX-License-Identifier: CDDL-1.0
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or https://opensource.org/licenses/CDDL-1.0.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2025, iXsystems, Inc.
+#
+
+# DESCRIPTION:
+#	Verify zfs rewrite rewrites specified files blocks.
+#
+# STRATEGY:
+#	1. Create two files, one of which is in a directory.
+#	2. Save the checksums and block pointers.
+#	3. Rewrite part of the files.
+#	4. Verify checksums are the same.
+#	5. Verify block pointers of the rewritten part have changed.
+#	6. Rewrite all the files.
+#	7. Verify checksums are the same.
+#	8. Verify all block pointers have changed.
+
+. $STF_SUITE/include/libtest.shlib
+
+typeset tmp=$(mktemp)
+typeset bps=$(mktemp)
+typeset bps1=$(mktemp)
+typeset bps2=$(mktemp)
+
+function cleanup
+{
+	rm -rf $tmp $bps $bps1 $bps2 $TESTDIR/*
+}
+
+log_assert "zfs rewrite rewrites specified files blocks"
+
+log_onexit cleanup
+
+log_must zfs set recordsize=128k $TESTPOOL/$TESTFS
+
+log_must mkdir $TESTDIR/dir
+log_must dd if=/dev/urandom of=$TESTDIR/file1 bs=128k count=8
+log_must dd if=$TESTDIR/file1 of=$TESTDIR/dir/file2 bs=128k
+log_must sync_pool $TESTPOOL
+typeset orig_hash1=$(xxh128digest $TESTDIR/file1)
+typeset orig_hash2=$(xxh128digest $TESTDIR/dir/file2)
+
+log_must [ "$orig_hash1" = "$orig_hash2" ]
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS file1 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps1"
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS dir/file2 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps2"
+
+log_must zfs rewrite -o 327680 -l 262144 -r -x $TESTDIR/file1 $TESTDIR/dir/file2
+log_must sync_pool $TESTPOOL
+typeset new_hash1=$(xxh128digest $TESTDIR/file1)
+typeset new_hash2=$(xxh128digest $TESTDIR/dir/file2)
+log_must [ "$orig_hash1" = "$new_hash1" ]
+log_must [ "$orig_hash2" = "$new_hash2" ]
+
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS file1 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps"
+typeset same=$(echo $(sort -n $bps $bps1 | uniq -d | cut -f1 -d' '))
+log_must [ "$same" = "0 1 5 6 7" ]
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS dir/file2 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps"
+typeset same=$(echo $(sort -n $bps $bps2 | uniq -d | cut -f1 -d' '))
+log_must [ "$same" = "0 1 5 6 7" ]
+
+log_must zfs rewrite -r $TESTDIR/file1 $TESTDIR/dir/file2
+log_must sync_pool $TESTPOOL
+typeset new_hash1=$(xxh128digest $TESTDIR/file1)
+typeset new_hash2=$(xxh128digest $TESTDIR/dir/file2)
+log_must [ "$orig_hash1" = "$new_hash1" ]
+log_must [ "$orig_hash2" = "$new_hash2" ]
+
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS file1 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps"
+typeset same=$(echo $(sort -n $bps $bps1 | uniq -d | cut -f1 -d' '))
+log_must [ -z "$same" ]
+log_must eval "zdb -Ovv $TESTPOOL/$TESTFS dir/file2 > $tmp"
+log_must eval "awk '/ L0 / { print l++ \" \" \$3 }' < $tmp > $bps"
+typeset same=$(echo $(sort -n $bps $bps2 | uniq -d | cut -f1 -d' '))
+log_must [ -z "$same" ]
+
+log_pass


Reviews:
user: login: gmelikov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T22:54:34Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:16:10Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:20:38Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:28:54Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:32:05Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:37:35Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:40:11Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:48:06Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:50:43Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:51:27Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:53:45Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T01:54:17Z
user: login: adamdmoss
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-27T21:42:35Z
user: login: adamdmoss
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-27T21:44:53Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T01:55:45Z
user: login: tonyhutter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:23:15Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T16:31:13Z
user: login: tonyhutter
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T20:06:45Z
user: login: robn
body: The guts of this looks good to me. I'd like you to consider the change I proposed in `zfs_rewrite_path`, but that's all I've got. Good work!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-07T02:34:56Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:44:21Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:47:12Z
user: login: amotin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:56:02Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T22:04:48Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T22:05:45Z
user: login: robn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T22:14:05Z
user: login: tonyhutter
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T17:21:20Z

Review comments:
diff_hunk: @@ -1050,6 +1050,135 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skeeping any holes.  This might
path: module/zfs/zfs_vnops.c
user: login: gmelikov
body: ```suggestion
		 * Rewrite only actual data, skipping any holes.  This might
```
created_at: 2025-04-15T20:50:45Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
path: module/zfs/zfs_vnops.c
user: login: robn
body: Suggest reserving the flag space to known flags, ie:

```suggestion
	if (flags != 0)
	    return (SET_ERROR(EINVAL));
```
created_at: 2025-04-22T01:16:10Z
author_association: MEMBER
diff_hunk: @@ -1620,6 +1620,14 @@ typedef enum zfs_ioc {
 
 #endif
 
+typedef struct zfs_rewrite_args {
+	uint64_t	off;
+	uint64_t	len;
+	uint64_t	flags;
+} zfs_rewrite_args_t;
path: include/sys/fs/zfs.h
user: login: robn
body: I note that Linux's `FIDEDUPERANGE` allows a list of ranges to be provided. I don't think that's interesting here (I'm not sure it's interesting there either), the majority of uses are going to be the whole object, or a single range within, but I mention just to make sure you've thought about whether or not multiple ranges in a single call is interesting.
created_at: 2025-04-22T01:20:38Z
author_association: MEMBER
diff_hunk: @@ -1620,6 +1620,14 @@ typedef enum zfs_ioc {
 
 #endif
 
+typedef struct zfs_rewrite_args {
+	uint64_t	off;
+	uint64_t	len;
+	uint64_t	flags;
+} zfs_rewrite_args_t;
path: include/sys/fs/zfs.h
user: login: amotin
body: I can't really guess why would I want multiple ranges there.  It would have to be pretty application-specific to know what blocks to rewrite.  I've added the range specification just to allow large file to be rewritten in several calls, if wanted.  In similar calls of other file system I was able to find I don't remember multiple range either.
created_at: 2025-04-22T01:28:54Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
path: module/zfs/zfs_vnops.c
user: login: amotin
body: I have no strong opinion about this.  All this function is quite advisory, so I am not sure we need to be strict here.  But if you'd prefer so, I'm fine with it.
created_at: 2025-04-22T01:32:05Z
author_association: MEMBER
diff_hunk: @@ -1620,6 +1620,14 @@ typedef enum zfs_ioc {
 
 #endif
 
+typedef struct zfs_rewrite_args {
+	uint64_t	off;
+	uint64_t	len;
+	uint64_t	flags;
+} zfs_rewrite_args_t;
path: include/sys/fs/zfs.h
user: login: robn
body: Yeah, it's the only one I know of, and imo it just complicates the interface for no obvious gain. Good!
created_at: 2025-04-22T01:37:35Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
path: module/zfs/zfs_vnops.c
user: login: robn
body: I prefer it as a general API design principle, just because I've seen people do weird things or make mistakes with args that got ignored and so seemed to work, and then later on came to life and broke something. This helps us avoid that from the outset.
created_at: 2025-04-22T01:40:11Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skipping any holes.  This might
+		 * be inaccurate for dirty files, but we don't really care.
+		 */
+		if (noff == off) {
+			/* Find next data in the file. */
+			error = dnode_next_offset(dn, 0, &noff, 1, 1, 0);
+			if (error || noff >= off + len) {
+				if (error == ESRCH)	/* No more data. */
+					error = 0;
+				break;
+			}
+			ASSERT3U(noff, >=, off);
+			len -= noff - off;
+			off = noff;
+
+			/* Find where the data end. */
+			error = dnode_next_offset(dn, DNODE_FIND_HOLE, &noff,
+			    1, 1, 0);
+			if (error != 0)
+				noff = off + len;
+		}
+		ASSERT3U(noff, >, off);
+
+		if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) ||
+		    zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) ||
+		    (projid != ZFS_DEFAULT_PROJID &&
+		    zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
+		    projid))) {
+			error = SET_ERROR(EDQUOT);
+			break;
+		}
+
+		n = MIN(MIN(len, noff - off),
+		    DMU_MAX_ACCESS / 2 - P2PHASE(off, zp->z_blksz));
+
+		dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
+		dmu_tx_hold_write_by_dnode(tx, dn, off, n);
+		error = dmu_tx_assign(tx, DMU_TX_WAIT);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+
+		/* Mark all dbufs within range as dirty to trigger rewrite. */
+		dmu_buf_t **dbp;
+		int numbufs;
+		error = dmu_buf_hold_array_by_dnode(dn, off, n, TRUE, FTAG,
+		    &numbufs, &dbp, DMU_READ_PREFETCH);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+		for (int i = 0; i < numbufs; i++) {
+			nr += dbp[i]->db_size;
+			if (dmu_buf_is_dirty(dbp[i], tx))
+				continue;
+			nw += dbp[i]->db_size;
path: module/zfs/zfs_vnops.c
user: login: robn
body: I wonder if, at least in the first cut, we should skip blocks in the BRT? Rewriting them is going to grow disk usage, potentially by quite a lot, and I feel like that might be pretty surprising to a casual user who doesn't know that they have lots of cloned blocked because it's transparent and enabled by default?

I might say the same of dedup'd blocks, and maybe even blocks that also exist in snapshots, though in that case it should be less surprising to the operator because they've already taken some positive action to get the pool into that state, unlike with cloned blocks.
created_at: 2025-04-22T01:48:06Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
path: module/zfs/zfs_vnops.c
user: login: amotin
body: Done.
created_at: 2025-04-22T01:50:43Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skipping any holes.  This might
+		 * be inaccurate for dirty files, but we don't really care.
+		 */
+		if (noff == off) {
+			/* Find next data in the file. */
+			error = dnode_next_offset(dn, 0, &noff, 1, 1, 0);
+			if (error || noff >= off + len) {
+				if (error == ESRCH)	/* No more data. */
+					error = 0;
+				break;
+			}
+			ASSERT3U(noff, >=, off);
+			len -= noff - off;
+			off = noff;
+
+			/* Find where the data end. */
+			error = dnode_next_offset(dn, DNODE_FIND_HOLE, &noff,
+			    1, 1, 0);
+			if (error != 0)
+				noff = off + len;
+		}
+		ASSERT3U(noff, >, off);
+
+		if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) ||
+		    zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) ||
+		    (projid != ZFS_DEFAULT_PROJID &&
+		    zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
+		    projid))) {
+			error = SET_ERROR(EDQUOT);
+			break;
+		}
+
+		n = MIN(MIN(len, noff - off),
+		    DMU_MAX_ACCESS / 2 - P2PHASE(off, zp->z_blksz));
+
+		dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
+		dmu_tx_hold_write_by_dnode(tx, dn, off, n);
+		error = dmu_tx_assign(tx, DMU_TX_WAIT);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+
+		/* Mark all dbufs within range as dirty to trigger rewrite. */
+		dmu_buf_t **dbp;
+		int numbufs;
+		error = dmu_buf_hold_array_by_dnode(dn, off, n, TRUE, FTAG,
+		    &numbufs, &dbp, DMU_READ_PREFETCH);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+		for (int i = 0; i < numbufs; i++) {
+			nr += dbp[i]->db_size;
+			if (dmu_buf_is_dirty(dbp[i], tx))
+				continue;
+			nw += dbp[i]->db_size;
path: module/zfs/zfs_vnops.c
user: login: robn
body: Possible ways to minimise amount of work and/or surprises:
- explicit flag to also do cloned blocks
- don't rewrite blocks that wouldn't change (if we can tell, at least maybe we can generate the write policy and compare first). (oh, now I think about it rewriting dedup'd blocks with no property changes will be re-dedup'd anyway)
- a dry-run option, showing the number of blocks rewritten and so the size change
created_at: 2025-04-22T01:51:27Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skipping any holes.  This might
+		 * be inaccurate for dirty files, but we don't really care.
+		 */
+		if (noff == off) {
+			/* Find next data in the file. */
+			error = dnode_next_offset(dn, 0, &noff, 1, 1, 0);
+			if (error || noff >= off + len) {
+				if (error == ESRCH)	/* No more data. */
+					error = 0;
+				break;
+			}
+			ASSERT3U(noff, >=, off);
+			len -= noff - off;
+			off = noff;
+
+			/* Find where the data end. */
+			error = dnode_next_offset(dn, DNODE_FIND_HOLE, &noff,
+			    1, 1, 0);
+			if (error != 0)
+				noff = off + len;
+		}
+		ASSERT3U(noff, >, off);
+
+		if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) ||
+		    zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) ||
+		    (projid != ZFS_DEFAULT_PROJID &&
+		    zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
+		    projid))) {
+			error = SET_ERROR(EDQUOT);
+			break;
+		}
+
+		n = MIN(MIN(len, noff - off),
+		    DMU_MAX_ACCESS / 2 - P2PHASE(off, zp->z_blksz));
+
+		dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
+		dmu_tx_hold_write_by_dnode(tx, dn, off, n);
+		error = dmu_tx_assign(tx, DMU_TX_WAIT);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+
+		/* Mark all dbufs within range as dirty to trigger rewrite. */
+		dmu_buf_t **dbp;
+		int numbufs;
+		error = dmu_buf_hold_array_by_dnode(dn, off, n, TRUE, FTAG,
+		    &numbufs, &dbp, DMU_READ_PREFETCH);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+		for (int i = 0; i < numbufs; i++) {
+			nr += dbp[i]->db_size;
+			if (dmu_buf_is_dirty(dbp[i], tx))
+				continue;
+			nw += dbp[i]->db_size;
path: module/zfs/zfs_vnops.c
user: login: robn
body: tbc, I'm fine with doing nothing for now, just so long as we thought about it.
created_at: 2025-04-22T01:53:45Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,140 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	(void) flags;
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
+
+	const uint64_t uid = KUID_TO_SUID(ZTOUID(zp));
+	const uint64_t gid = KGID_TO_SGID(ZTOGID(zp));
+	const uint64_t projid = zp->z_projid;
+
+	dmu_buf_impl_t *db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
+	DB_DNODE_ENTER(db);
+	dnode_t *dn = DB_DNODE(db);
+
+	uint64_t n, noff = off, nr = 0, nw = 0;
+	while (len > 0) {
+		/*
+		 * Rewrite only actual data, skipping any holes.  This might
+		 * be inaccurate for dirty files, but we don't really care.
+		 */
+		if (noff == off) {
+			/* Find next data in the file. */
+			error = dnode_next_offset(dn, 0, &noff, 1, 1, 0);
+			if (error || noff >= off + len) {
+				if (error == ESRCH)	/* No more data. */
+					error = 0;
+				break;
+			}
+			ASSERT3U(noff, >=, off);
+			len -= noff - off;
+			off = noff;
+
+			/* Find where the data end. */
+			error = dnode_next_offset(dn, DNODE_FIND_HOLE, &noff,
+			    1, 1, 0);
+			if (error != 0)
+				noff = off + len;
+		}
+		ASSERT3U(noff, >, off);
+
+		if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) ||
+		    zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) ||
+		    (projid != ZFS_DEFAULT_PROJID &&
+		    zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
+		    projid))) {
+			error = SET_ERROR(EDQUOT);
+			break;
+		}
+
+		n = MIN(MIN(len, noff - off),
+		    DMU_MAX_ACCESS / 2 - P2PHASE(off, zp->z_blksz));
+
+		dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
+		dmu_tx_hold_write_by_dnode(tx, dn, off, n);
+		error = dmu_tx_assign(tx, DMU_TX_WAIT);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+
+		/* Mark all dbufs within range as dirty to trigger rewrite. */
+		dmu_buf_t **dbp;
+		int numbufs;
+		error = dmu_buf_hold_array_by_dnode(dn, off, n, TRUE, FTAG,
+		    &numbufs, &dbp, DMU_READ_PREFETCH);
+		if (error) {
+			dmu_tx_abort(tx);
+			break;
+		}
+		for (int i = 0; i < numbufs; i++) {
+			nr += dbp[i]->db_size;
+			if (dmu_buf_is_dirty(dbp[i], tx))
+				continue;
+			nw += dbp[i]->db_size;
path: module/zfs/zfs_vnops.c
user: login: amotin
body: That is what I was planning flags for.  But after looking closer it appeared to me that it would be quite a layering violation to try figure out all the possible relations between block pointer fields and dataset properties here. They are not exactly trivial sometimes.
created_at: 2025-04-22T01:54:17Z
author_association: MEMBER
diff_hunk: @@ -1050,6 +1050,141 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	if (flags != 0)
+		return (SET_ERROR(EINVAL));
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
path: module/zfs/zfs_vnops.c
user: login: adamdmoss
body: If it's important that mmap()'d data ~hits disk~ is flushed (why?) then is there still a race between this zn_flush_cached_data() and the subsequent rangelock, where new mmap()'d data from this range may have entered cache?
created_at: 2025-04-27T21:42:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1050,6 +1050,141 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	if (flags != 0)
+		return (SET_ERROR(EINVAL));
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
path: module/zfs/zfs_vnops.c
user: login: adamdmoss
body: Ah, I think I see, this is only best-effort and shouldn't actually affect integrity.
created_at: 2025-04-27T21:44:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1050,6 +1050,141 @@ zfs_write(znode_t *zp, zfs_uio_t *uio, int ioflag, cred_t *cr)
 	return (0);
 }
 
+/*
+ * Rewrite a range of file as-is without modification.
+ *
+ *	IN:	zp	- znode of file to be rewritten.
+ *		off	- Offset of the range to rewrite.
+ *		len	- Length of the range to rewrite.
+ *		flags	- Random rewrite parameters.
+ *
+ *	RETURN:	0 if success
+ *		error code if failure
+ */
+int
+zfs_rewrite(znode_t *zp, uint64_t off, uint64_t len, uint64_t flags)
+{
+	int error;
+
+	if (flags != 0)
+		return (SET_ERROR(EINVAL));
+
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	if ((error = zfs_enter_verify_zp(zfsvfs, zp, FTAG)) != 0)
+		return (error);
+
+	if (zfs_is_readonly(zfsvfs)) {
+		zfs_exit(zfsvfs, FTAG);
+		return (SET_ERROR(EROFS));
+	}
+
+	if (off >= zp->z_size) {
+		zfs_exit(zfsvfs, FTAG);
+		return (0);
+	}
+	if (len == 0 || len > zp->z_size - off)
+		len = zp->z_size - off;
+
+	/* Flush any mmap()'d data to disk */
+	if (zn_has_cached_data(zp, off, off + len - 1))
+		zn_flush_cached_data(zp, B_TRUE);
+
+	zfs_locked_range_t *lr;
+	lr = zfs_rangelock_enter(&zp->z_rangelock, off, len, RL_WRITER);
path: module/zfs/zfs_vnops.c
user: login: amotin
body: It is just to avoid double re-write in case one is already on its way.  Not really required.
created_at: 2025-04-28T01:55:45Z
author_association: MEMBER
diff_hunk: @@ -432,6 +439,9 @@ get_usage(zfs_help_t idx)
 	case HELP_REDACT:
 		return (gettext("\tredact <snapshot> <bookmark> "
 		    "<redaction_snapshot> ...\n"));
+	case HELP_REWRITE:
+		return (gettext("\trewrite [-rx] [-o <offset>] [-l <length>] "
path: cmd/zfs/zfs_main.c
user: login: tonyhutter
body: ```diff
- [-rx]
+ [-rvx]
```
created_at: 2025-05-01T16:23:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -432,6 +439,9 @@ get_usage(zfs_help_t idx)
 	case HELP_REDACT:
 		return (gettext("\tredact <snapshot> <bookmark> "
 		    "<redaction_snapshot> ...\n"));
+	case HELP_REWRITE:
+		return (gettext("\trewrite [-rx] [-o <offset>] [-l <length>] "
path: cmd/zfs/zfs_main.c
user: login: amotin
body: Fixed.
created_at: 2025-05-01T16:31:13Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
+			if (err)
+				ret = err;
+		} else { /* DT_DIR */
+			fnvlist_add_uint64(dirs, fullname, dev);
+		}
+
+		free(fullname);
+	}
+
+	closedir(dir);
+	return (ret);
+}
+
+static int
+zfs_rewrite_path(const char *path, boolean_t verbose, boolean_t recurse,
+    boolean_t xdev, zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct stat st;
+	int ret = 0;
+
+	if (stat(path, &st) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to stat %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (S_ISREG(st.st_mode)) {
+		ret = zfs_rewrite_file(path, verbose, args);
+	} else if (S_ISDIR(st.st_mode) && recurse) {
+		ret = zfs_rewrite_dir(path, verbose, xdev, st.st_dev, args,
+		    dirs);
+	}
path: cmd/zfs/zfs_main.c
user: login: robn
body: I messed with this a bit, and came up with:

```suggestion
        if (lstat(path, &st) < 0) {
                ret = errno;
                (void) fprintf(stderr, gettext("failed to stat %s: %s\n"),
                    path, strerror(errno));
                return (ret);
        }

        if (S_ISREG(st.st_mode)) {
                ret = zfs_rewrite_file(path, verbose, args);
        } else if (S_ISDIR(st.st_mode) && recurse) {
                ret = zfs_rewrite_dir(path, verbose, xdev, st.st_dev, args,
                    dirs);
        } else {
                ret = S_ISDIR(st.st_mode) ? EISDIR : EINVAL;
                const char *errstr =
                    (ret == EISDIR) ? "is a directory" : "not a regular file";
                (void) fprintf(stderr, gettext("can't rewrite %s: %s\n"),
                    path, errstr);
        }
```

That is:
- don't allow a rewrite through a symlink (matches directory mode, protects against cross-device, protects multiple links to same file)
- show an error if you try to rewrite something that isn't a file
- show an error if you try to write a directory without `-r`

The symlink thing I think is important; at least, it surprised me that it worked at all (rewrote the target), and not being the same as directory mode bothered me. The other two are just niceties.

I did think about also limiting to things not on ZFS, because it is strange to me that a `zfs` command could even have an opinion about non-ZFS filesystems. But, there isn't really a good cross-platform way to find this out, and the `ioctl()` call will probably fail, so I'm kind of whatever about it.
created_at: 2025-05-07T02:06:21Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
path: cmd/zfs/zfs_main.c
user: login: robn
body: I'd print this before the call to `ioctl()`, so the operator can see where it's up to, particularly if it takes a while, or if they ctrl-c it.
created_at: 2025-05-07T02:11:52Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
path: cmd/zfs/zfs_main.c
user: login: robn
body: I was thinking about hard links, and tracking them to avoid not rewriting the object twice, but it's kind of a colossal pita to do a good job of, so probably not worth it. But I said it at least!
created_at: 2025-05-07T02:15:09Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
+			if (err)
+				ret = err;
+		} else { /* DT_DIR */
+			fnvlist_add_uint64(dirs, fullname, dev);
+		}
+
+		free(fullname);
+	}
path: cmd/zfs/zfs_main.c
user: login: robn
body: Should we break the loop for certain kinds of errors from rewrite, like `ENOSPC` or `EDQUOT`? Probably not I guess, it's hard to tell for sure that it's the right thing and ctrl-c is right there.
created_at: 2025-05-07T02:33:16Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
path: cmd/zfs/zfs_main.c
user: login: amotin
body: I was thinking about it, but my motivation was to print only successfully processed files in case somebody wish to manually implement restart.  But I can see the other side too, if people prefer.
created_at: 2025-05-07T13:44:21Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
path: cmd/zfs/zfs_main.c
user: login: amotin
body: I haven't thought about it.  With a good job I suppose you mean accumulating the list of already processed inode numbers for each file system, or there is some other way?
created_at: 2025-05-07T13:47:12Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
+			if (err)
+				ret = err;
+		} else { /* DT_DIR */
+			fnvlist_add_uint64(dirs, fullname, dev);
+		}
+
+		free(fullname);
+	}
+
+	closedir(dir);
+	return (ret);
+}
+
+static int
+zfs_rewrite_path(const char *path, boolean_t verbose, boolean_t recurse,
+    boolean_t xdev, zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct stat st;
+	int ret = 0;
+
+	if (stat(path, &st) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to stat %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (S_ISREG(st.st_mode)) {
+		ret = zfs_rewrite_file(path, verbose, args);
+	} else if (S_ISDIR(st.st_mode) && recurse) {
+		ret = zfs_rewrite_dir(path, verbose, xdev, st.st_dev, args,
+		    dirs);
+	}
path: cmd/zfs/zfs_main.c
user: login: amotin
body: About `lstat()` is a good point. Thank you. Fixed.

About the reporting errors I am not sure.  I wanted to make possible running it with `*` not worrying about directory content, and when recursing we do not report errors for non-regular files, so it is kind of consistent.

Limiting to ZFS would be good, but I don't know how.  That was actually the first reason I added the `-x` flag, to at least allow correct use of it.
created_at: 2025-05-07T13:56:02Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
path: cmd/zfs/zfs_main.c
user: login: robn
body: Yeah, tracking the inode numbers is the heart of it. I can see it getting fiddly around the edges though. I reckon it's not worth it.
created_at: 2025-05-07T22:04:48Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
path: cmd/zfs/zfs_main.c
user: login: robn
body: Yeah, I think I'm not too bothered.
created_at: 2025-05-07T22:05:45Z
author_association: MEMBER
diff_hunk: @@ -9016,6 +9026,192 @@ zfs_do_project(int argc, char **argv)
 	return (ret);
 }
 
+static int
+zfs_rewrite_file(const char *path, boolean_t verbose, zfs_rewrite_args_t *args)
+{
+	int fd, ret = 0;
+
+	fd = open(path, O_WRONLY);
+	if (fd < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (ioctl(fd, ZFS_IOC_REWRITE, args) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to rewrite %s: %s\n"),
+		    path, strerror(errno));
+	} else if (verbose) {
+		printf("%s\n", path);
+	}
+
+	close(fd);
+	return (ret);
+}
+
+static int
+zfs_rewrite_dir(const char *path, boolean_t verbose, boolean_t xdev, dev_t dev,
+    zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct dirent *ent;
+	DIR *dir;
+	int ret = 0, err;
+
+	dir = opendir(path);
+	if (dir == NULL) {
+		if (errno == ENOENT)
+			return (0);
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to opendir %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	size_t plen = strlen(path) + 1;
+	while ((ent = readdir(dir)) != NULL) {
+		char *fullname;
+		struct stat st;
+
+		if (ent->d_type != DT_REG && ent->d_type != DT_DIR)
+			continue;
+
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		if (plen + strlen(ent->d_name) >= PATH_MAX) {
+			(void) fprintf(stderr, gettext("path too long %s/%s\n"),
+			    path, ent->d_name);
+			ret = ENAMETOOLONG;
+			continue;
+		}
+
+		if (asprintf(&fullname, "%s/%s", path, ent->d_name) == -1) {
+			(void) fprintf(stderr,
+			    gettext("failed to allocate memory\n"));
+			ret = ENOMEM;
+			continue;
+		}
+
+		if (xdev) {
+			if (stat(fullname, &st) < 0) {
+				ret = errno;
+				(void) fprintf(stderr,
+				    gettext("failed to stat %s: %s\n"),
+				    fullname, strerror(errno));
+				free(fullname);
+				continue;
+			}
+			if (st.st_dev != dev) {
+				free(fullname);
+				continue;
+			}
+		}
+
+		if (ent->d_type == DT_REG) {
+			err = zfs_rewrite_file(fullname, verbose, args);
+			if (err)
+				ret = err;
+		} else { /* DT_DIR */
+			fnvlist_add_uint64(dirs, fullname, dev);
+		}
+
+		free(fullname);
+	}
+
+	closedir(dir);
+	return (ret);
+}
+
+static int
+zfs_rewrite_path(const char *path, boolean_t verbose, boolean_t recurse,
+    boolean_t xdev, zfs_rewrite_args_t *args, nvlist_t *dirs)
+{
+	struct stat st;
+	int ret = 0;
+
+	if (stat(path, &st) < 0) {
+		ret = errno;
+		(void) fprintf(stderr, gettext("failed to stat %s: %s\n"),
+		    path, strerror(errno));
+		return (ret);
+	}
+
+	if (S_ISREG(st.st_mode)) {
+		ret = zfs_rewrite_file(path, verbose, args);
+	} else if (S_ISDIR(st.st_mode) && recurse) {
+		ret = zfs_rewrite_dir(path, verbose, xdev, st.st_dev, args,
+		    dirs);
+	}
path: cmd/zfs/zfs_main.c
user: login: robn
body: Ahh, I hadn't thought about `*`, interesting.

The actual thing that confused me originally was `zfs-rewrite somedir/`, which outputs nothing and returns success. And then it seemed like you might want that for other file types too.

I guess if I had to argue, I'd want more error output, not less, because I can redirect errors away, and I can prefilter by file types if I want with `find | xargs`.

(Other programs tackle this with a "quiet" option or, less often, a "noisy" option, but I never remember those sort of things exist)
created_at: 2025-05-07T22:14:04Z
author_association: MEMBER

PR comments:
user: login: amotin
created_at: 2025-04-15T20:53:03Z
body: I've tried to find some kernel APIs to wire this to, but found that plenty of Linux file systems each implement their own IOCTL's for similar purposes.  I did the same, except the IOCTL number I took almost arbitrary, since ZFS seems quite rough in this area.  I am open to any better ideas before this is committed.
author_association: MEMBER
user: login: Momi-V
created_at: 2025-04-15T23:34:16Z
body: This looks amazing! Not having to sift through half a dozen shell scripts every time this comes up to see what currently handles the most edge cases correctly is very much appreciated. Especially with RaidZ expansion, being able to direct users to run a built-in command instead of debating what script to send them to would be very nice.

Also being able to reliably rewrite a live dataset while it's in use without having to worry about skipped files or mtime conflicts would make the whole process much less of a hassle. With the only thing to really worry about being snapshots/space usage this seems as close to perfect as reasonably possible (without diving deep into internals and messing with snapshot immutability). Bravo!
author_association: NONE
user: login: clhedrick
created_at: 2025-04-16T14:20:08Z
body: thank you. Fixes one of the biggest problems with ZFS.

Is there a way to suspend the process? It might be nice to have it run only during off hours. 
author_association: NONE
user: login: amotin
created_at: 2025-04-16T14:41:06Z
body: > Is there a way to suspend the process? It might be nice to have it run only during off hours.

It does one file at a time, and should be killable in between.  Signal handling within one huge file can probably be added.  Though the question of the process restart is on the user.  I didn't plan to go that deep into the area within this PR.
author_association: MEMBER
user: login: clhedrick
created_at: 2025-04-16T20:14:16Z
body: I couldn't find documentation in the files changed, so I have to guess how it actually works. Is it a file at a time? I guess you could feed it with a "find" command. For a system with a billion files, do you have a sense how long this is gong to take? We can do scrubs in a day or two, but rsync is impractically slow. If this is happening at the file system level, that migth be the case here as well.
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-16T20:29:35Z
body: > I guess you could feed it with a "find" command.

This will likely be a good use case for GNU Parallel.
author_association: NONE
user: login: Momi-V
created_at: 2025-04-16T20:49:48Z
body: > I couldn't find documentation in the files changed, so I have to guess how it actually works. Is it a file at a time? I guess you could feed it with a "find" command. For a system with a billion files, do you have a sense how long this is gong to take? We can do scrubs in a day or two, but rsync is impractically slow. If this is happening at the file system level, that migth be the case here as well.

It can take a directory as an argument and there are some recursive functions and iterators in the code so piping find into it should not be necessary. That avoids some userspace file handling overhead, but it still has to go through the contents of each directory one file at a time. I also don't see any parallel execution or threading (though I'm not too familiar with ZFS internals, maybe some of the primitives used here run asynchronously?).

Whether doing parallelism in userspace by just calling it for many files/directories at once or not it should have the required locking to just run in the background and be significantly more elegant than the CP + mtime (or potentially userspace hash) check to make sure files didn't change during the copy process avoiding one of the potential pitfalls of existing solutions.
author_association: NONE
user: login: amotin
created_at: 2025-04-16T21:26:27Z
body: I haven't benchmarked it deep yet, but unless the files are tiny, I don't expect there is a major need for parallelism.  The code in kernel should handle up to 16MB at a time, plus allows ZFS to do read-ahead and write-back on top of that, so there will be quite a lot in the pipeline to saturate the disks and/or the system, especially if there is some compression/checksuming/encryption.  And without need to copy data to/from user-space, the only thread will not be doing too much, I think mostly a decompression from ARC.  Bunch of small files on a wide HDD pool I suspect may indeed suffer from read latency, but that in user-space we can optimize/parallelize all day long.
author_association: MEMBER
user: login: tonyhutter
created_at: 2025-04-16T21:31:18Z
body: I gave this a quick test.  It's very fast and does exactly what it says :+1: 
```
# Copy ZFS source workspace to pool with compression=off
$ time cp -a ~/zfs /tank2

real	0m0.600s
user	0m0.032s
sys	0m0.519s

$ df -h /tank2
Filesystem      Size  Used Avail Use% Mounted on
tank2           9.3G  893M  8.4G  10% /tank2


# Set compression to 'gzip' and rewrite
$ sudo ./zfs set compression=gzip tank2
$ time sudo ./zfs rewrite -r /tank2

real	0m2.272s
user	0m0.005s
sys	0m0.005s

$ df -h /tank2
Filesystem      Size  Used Avail Use% Mounted on
tank2           9.3G  402M  8.9G   5% /tank2


# Set compression to 'lz4' and rewrite
$ sudo ./zfs set compression=lz4 tank2
$ time sudo ./zfs rewrite -r /tank2
real	0m1.947s
user	0m0.002s
sys	0m0.010s

$ df -h /tank2
Filesystem      Size  Used Avail Use% Mounted on
tank2           9.3G  456M  8.8G   5% /tank2


# Set compression to 'zstd' and rewrite
$ sudo ./zfs set compression=zstd tank2
$ time sudo ./zfs rewrite -r /tank2

real	0m0.616s
user	0m0.003s
sys	0m0.006s

$ df -h /tank2
Filesystem      Size  Used Avail Use% Mounted on
tank2           9.3G  366M  8.9G   4% /tank2
```
I can already see people writing scripts that go though every dataset, setting the optimal compression, recordsize, etc, and zfs rewrite-ing them.
author_association: CONTRIBUTOR
user: login: amotin
created_at: 2025-04-16T21:42:51Z
body: Cool!  Though the recordsize is one of things it can't change, since it would requite real byte-level copy, not just marking existing blocks dirty.  I am not sure it can be done under the load in general.  At least it would be much more complicated.
author_association: MEMBER
user: login: snajpa
created_at: 2025-04-17T16:58:22Z
body: Umm this is basically same as doing send | recv, isn't it? I mean, in a way, this is already possible to do without any changes, isn't it? Recv will even respect a lower recordsize, if I'm not mistaken - at least when receiving into a pool without large blocks support, it has to do that.

I'm thinking whether we can do better, in the original sense of ZFS "better", meaning "automagic" - what do you think of using snapshots, send|recv, in a loop with ever decreasing delta size and then when the delta isn't decreasing anymore, we could swap those datasets and use (perhaps slightly modified) `zfs_resume_fs` transparently to the userspace... that way we would get transparent migration into a dataset with different options, that would scratch some itches for people, wouldn't it?

It'd be even cooler if it could coalesce smaller blocks into larger ones, but that potentially implies performance problems with write amplification, I would say if the app writes in smaler chunks that it gets onto disk in such smaller chunks, it's probably for the best to leave them that way. For any practical use-case I could think of though, I would definitely appreciate the ability to split the blocks of a dataset using smaller `recordsize`.

If there's a way how to make `zfs rewrite` more automagical, I think it's at least worth considering.
author_association: CONTRIBUTOR
user: login: Momi-V
created_at: 2025-04-17T17:15:06Z
body: send recv has the huge downside of requiring 2x the space, even if you do the delta size thing since it has to send the entire dataset at least once and old data can't be deleted until the new dataset is complete.
Also recv doesn't increase block sizes, it only splits them if they are larger than the other pool supports (and iirc. there have even been some issues with that).
Also that idea sounds a lot more complex than simply walking the directory tree and iterating through the files to mark their records as dirty to cause a rewrite.

> we would get transparent migration into a dataset with different options, that would scratch some itches for people, wouldn't it?

Isn't this exactly what rewrite does? Change the options, run it and all the blocks are changed in the background. Without an application even seeing a change to the file. And unlike send recv it only needs a few MB of extra space.

Edit: with the only real exception being record size, but recv also solves that only partially at best and it doesn't look like there's a reasonable way to work around that in a wholly transparent fashion.
author_association: NONE
user: login: amotin
created_at: 2025-04-19T01:40:34Z
body:  - Added -x flag to not cross mount points.
 - Added signal handling in kernel.
 - Added man page.
author_association: MEMBER
user: login: stuartthebruce
created_at: 2025-04-19T23:20:48Z
body: Which release is this game changing enhancement likely to land in?
author_association: NONE
user: login: amotin
created_at: 2025-04-20T00:33:04Z
body: @stuartthebruce So far it haven't landed even in master, so anybody who want to speed it up is welcome to test and comment.  In general though, when completed, there is no reason why aside of 2.4.0 it can't be ported back to some 2.3.x of the time.
author_association: MEMBER
user: login: stuartthebruce
created_at: 2025-04-20T00:41:01Z
body: > @stuartthebruce So far it haven't landed even in master, so anybody who want to speed it up is welcome to test and comment. In general though, when completed, there is no reason why aside of 2.4.0 it can't be ported back to some 2.3.x of the time.

Good to know there are no obvious blockers from including in a future 2.3.x. Once this hits master I will help by setting up a test system with 1/2PB of 10^9 small files to see if I can break it. Is there any reason to think the code will be sensitive to Linux vs FreeBSD?
author_association: NONE
user: login: amotin
created_at: 2025-04-20T00:46:04Z
body: > Is there any reason to think the code will be sensitive to Linux vs FreeBSD?

IOCTL interface of the kernels is obviously slightly different, requiring OS-specific shims, as with most of other VFS-related code.  But seems like not a big problem, as Tony confirmed it works on Linux too from the first try.
author_association: MEMBER
user: login: amotin
created_at: 2025-04-20T00:50:37Z
body: > Once this hits master

Since this introduces new IOCTL API, I'd appreciate some feedback before it hit master in case some desired functionality might require API changes aside of the `flags` field I already reserved for later extensions.  I was thinking about some options to not rewrite in some cases, but didn't want to pollute the code until I am convinced it is required.
author_association: MEMBER
user: login: stuartthebruce
created_at: 2025-04-20T01:00:02Z
body: > Since this introduces new IOCTL API, I'd appreciate some feedback before it hit master in case some desired functionality might require API changes aside of the `flags` field I already reserved for later extensions. I was thinking about some options to not rewrite in some cases, but didn't want to pollute the code until I am convinced it is required.

OK, I will see if I can find some time this next week to stress test.

author_association: NONE
user: login: robn
created_at: 2025-04-22T01:14:38Z
body: I like the approach (of course I would, it's the same approach I took in [a prototype](https://github.com/robn/zfs/commits/rewrite-range)). Good stuff!

It's sort of a bikeshed, but I don't like `zfs rewrite`, because `zfs` is supposed to be about dataset management, and I can imagine wanting to use that for some sort of dataset-wide rewriting function in the future, _and_ I can imagine people reading it and thinking its for datasets, not for files. But, I don't really have a better suggestion. It's sort of the same reason I never did a `zfs clonefile` to get around the cloning restrictions on Linux. Do we want to make a `zfile` command? Or a sub-sub-command namespace like `zfs file rewrite`, where we could also put `zfs file clone` and probably other things we can think of? Or do we not offer it at all, and instead offer patches to relevant third-party tools to use the ioctls instead? At some point we should decide something, but as usual, I don't know where or how to have that conversation (I definitely have preferences), and I don't know if this is the PR to do it on. Maybe `zfs rewrite-file` is the safe middle road for the moment?


author_association: MEMBER
user: login: amotin
created_at: 2025-04-22T01:24:33Z
body: @robn I had some of the same thinking myself.  But even more than adding file-oriented sub-command to `zfs` I hate adding new commands.  Patching third-party tools to use the IOCTL does not sound realistic for cross-platform feature.  I found some solace in the fact that we already have `zfs project` sub-command, that is file-oriented too.  I am not particularly locked to the `zfs rewrite` name, haven't used it anywhere yet, but `zfs project` we can't rename already.
author_association: MEMBER
user: login: robn
created_at: 2025-04-22T01:36:07Z
body: @amotin the `zfs project` example is a good tiebreaker for now I think, so I will get behind whatever decision you make.

(I will say that I have some bigger thoughts around design, maintenance and deprecation of interfaces of all kinds within OpenZFS that I am hoping to write down and discuss soon (now that I'm finally getting some time back!) but, I don't want my own preferences on this to get in the way of something now and I would hope that such a discussion would end up giving us guidance in situations like this, and a path to renaming or moving things like `zfs rewrite` and `zfs project` if we decide to. So imo it's not the time to throw up a roadblock out of nowhere).

((This is of course my usual out-of-context half-cooked waffling, please enjoy and ignore at your leisure :loudspeaker: :innocent:))
author_association: MEMBER
user: login: tonyhutter
created_at: 2025-04-23T00:24:54Z
body: Test cases would be good
author_association: CONTRIBUTOR
user: login: stuartthebruce
created_at: 2025-04-26T23:32:07Z
body: > OK, I will see if I can find some time this next week to stress test.

I started testing with a copy of the following home directories on an otherwise idle RL9 test system initially running version 2.3.1,
<pre>
[root@zfsarchive1 ~]# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
jbod17              16.8T   910T   136K  /jbod17
jbod17/cal           478G   910T   473G  /jbod17/cal
jbod17/dqr          4.06T   910T  3.97T  /jbod17/dqr
jbod17/grb.exttrig  4.99T   910T  4.98T  /jbod17/grb.exttrig
jbod17/idq          2.51T   910T  2.51T  /jbod17/idq
jbod17/pe.o4        4.73T   910T  4.63T  /jbod17/pe.o4
</pre>
and then updated to,
<pre>
[root@zfsarchive1 zfs]# git log
commit c5f4413446609a2fd3c4b817f7ad7ebceb915991 (HEAD -> rewrite, origin/rewrite)
Author: Alexander Motin <mav@FreeBSD.org>
Date:   Tue Apr 15 16:07:05 2025 -0400

    Introduce zfs rewrite subcommand
    
    This allows to rewrite content of specified file(s) as-is without
...
</pre>
The initial test is to remove compression from one of these datasets with 2M files,
<pre>
[root@zfsarchive1 ~]# zfs get space jbod17/cal
NAME        PROPERTY              VALUE          SOURCE
jbod17/cal  name                  jbod17/cal     -
jbod17/cal  available             910T           -
jbod17/cal  used                  478G           -
jbod17/cal  usedbysnapshots       4.67G          -
jbod17/cal  usedbydataset         473G           -
jbod17/cal  usedbyrefreservation  0B             -
jbod17/cal  usedbychildren        0B             -

[root@zfsarchive1 ~]# zfs get compression,compressratio jbod17/cal
NAME        PROPERTY       VALUE           SOURCE
jbod17/cal  compression    on              inherited from jbod17
jbod17/cal  compressratio  1.14x           -

[root@zfsarchive1 ~]# df -i /jbod17/cal
Filesystem            Inodes   IUsed         IFree IUse% Mounted on
jbod17/cal     1954605040973 1985661 1954603055312    1% /jbod17/cal

[root@zfsarchive1 ~]# set compression=off jbod17/cal

[root@zfsarchive1 ~]# time zfs rewrite -r /jbod17/cal
...
</pre>
This is currently processing the files at ~100MB/s,
<pre>
[root@zfsarchive1 ~]# zpool iostat jbod17 5
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
jbod17      20.8T  1.12P  7.28K    367  84.3M   103M
jbod17      20.8T  1.12P  7.28K    494  70.4M   181M
jbod17      20.8T  1.12P  7.41K    387  87.1M  84.7M
jbod17      20.8T  1.12P  9.09K    440  91.8M   117M
jbod17      20.8T  1.12P  7.21K    434  75.4M   110M
jbod17      20.8T  1.12P  12.8K    515   137M   133M
jbod17      20.8T  1.12P  18.5K    996   200M   280M
</pre>
with the zfs process using ~15% of a cpu-core
<pre>
[root@zfsarchive1 ~]# top

top - 16:25:06 up 13 min,  3 users,  load average: 2.62, 2.41, 1.85
Tasks: 2395 total,   2 running, 2393 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  1.7 sy,  0.0 ni, 97.5 id,  0.7 wa,  0.1 hi,  0.0 si,  0.0 st
MiB Mem : 385934.4 total, 306918.4 free,  78558.8 used,   3688.4 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 307375.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                 
  56137 root      20   0   16352   5120   5120 D  14.1   0.0   0:54.92 zfs                                                                                                                                                     
  41603 root       0 -20       0      0      0 S   1.6   0.0   0:03.67 z_wr_int_9                                                                                                                                              
  42828 root       0 -20       0      0      0 S   1.6   0.0   0:03.61 z_wr_int_0     
</pre>
and `perf` showing,
<pre>
Samples: 95K of event 'cycles:P', 4000 Hz, Event count (approx.): 46282639308 lost: 0/0 drop: 0/0
  Children      Self  Shared Object               Symbol
+   76.99%     0.72%  [kernel]                    [k] taskq_thread
+   57.02%     0.30%  [kernel]                    [k] zio_execute
+   47.12%     0.25%  [kernel]                    [k] _raw_spin_lock_irqsave
+   46.17%    46.07%  [kernel]                    [k] native_queued_spin_lock_slowpath
+   39.39%     0.00%  [kernel]                    [k] ret_from_fork
+   39.38%     0.00%  [kernel]                    [k] kthread
+   35.26%     0.59%  [kernel]                    [k] zio_done
+   30.81%     0.26%  [kernel]                    [k] taskq_dispatch_ent
+   12.70%     0.07%  [kernel]                    [k] zio_vdev_io_start
+   12.65%     0.08%  [kernel]                    [k] zio_nowait
+   10.83%     0.02%  [kernel]                    [k] vdev_mirror_io_start
+    7.65%     0.01%  [kernel]                    [k] entry_SYSCALL_64_after_hwframe
+    7.64%     0.04%  [kernel]                    [k] do_syscall_64
+    7.61%     0.03%  [kernel]                    [k] vdev_raidz_io_start
+    7.50%     0.04%  [kernel]                    [k] zio_write_compress
+    7.29%     0.02%  [kernel]                    [k] zfs_lz4_compress
...
</pre>
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-27T19:33:33Z
body: > I started testing with a copy of the following home directories on an otherwise idle RL9 test system initially running version 2.3.1,

That finished without crashing,
<pre>
[root@zfsarchive1 ~]# time zfs rewrite -r /jbod17/cal
                                                            
real    176m32.645s
user    0m3.518s
sys     8m56.518s
</pre>
And resulted in,
<pre>
[root@zfsarchive1 ~]# zfs get space jbod17/cal
NAME        PROPERTY              VALUE          SOURCE
jbod17/cal  name                  jbod17/cal     -
jbod17/cal  available             910T           -
jbod17/cal  used                  950G           -
jbod17/cal  usedbysnapshots       477G           -
jbod17/cal  usedbydataset         473G           -
jbod17/cal  usedbyrefreservation  0B             -
jbod17/cal  usedbychildren        0B             -

[root@zfsarchive1 ~]# zfs get compression,compressratio jbod17/cal
NAME        PROPERTY       VALUE           SOURCE
jbod17/cal  compression    on              inherited from jbod17
jbod17/cal  compressratio  1.14x           -

[root@zfsarchive1 ~]# df -i /jbod17/cal
Filesystem            Inodes   IUsed         IFree IUse% Mounted on
jbod17/cal     1953610773572 1985661 1953608787911    1% /jbod17/cal
</pre>
After destroying all of the unmodified snapshots,
<pre>
[root@zfsarchive1 ~]# zfs list -H -o name -t snapshot jbod17/cal  | xargs --no-run-if-empty -n1 zfs destroy

[root@zfsarchive1 ~]# zfs get space jbod17/cal
NAME        PROPERTY              VALUE          SOURCE
jbod17/cal  name                  jbod17/cal     -
jbod17/cal  available             910T           -
jbod17/cal  used                  473G           -
jbod17/cal  usedbysnapshots       0B             -
jbod17/cal  usedbydataset         473G           -
jbod17/cal  usedbyrefreservation  0B             -
jbod17/cal  usedbychildren        0B             -

[root@zfsarchive1 ~]# zfs get compression,compressratio jbod17/cal
NAME        PROPERTY       VALUE           SOURCE
jbod17/cal  compression    on              inherited from jbod17
jbod17/cal  compressratio  1.14x           -
</pre>
Which appears to have been a big NOOP. It appears I fat fingered something and didn't actually hit return on the command to set `compression=off`, which is confirmed from `zpool history`, so trying again, albeit this time with no snapshots in this dataset,
<pre>
[root@zfsarchive1 ~]# zfs set compression=off jbod17/cal

[root@zfsarchive1 ~]# zfs get compression,compressratio jbod17/cal
NAME        PROPERTY       VALUE           SOURCE
jbod17/cal  compression    off             local
jbod17/cal  compressratio  1.14x           -

[root@zfsarchive1 ~]# time zfs rewrite -r /jbod17/cal
</pre>
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-27T21:22:00Z
body: > Which appears to have been a big NOOP. It appears I fat fingered something and didn't actually hit return on the command to set `compression=off`, which is confirmed from `zpool history`, so trying again, albeit this time with no snapshots in this dataset,

It ran faster this time and actually removed compression,
<pre>
[root@zfsarchive1 ~]# time zfs rewrite -r /jbod17/cal
                                                                                         

real    48m48.924s
user    0m2.229s
sys     6m59.283s

[root@zfsarchive1 ~]# zfs get space jbod17/cal
NAME        PROPERTY              VALUE          SOURCE
jbod17/cal  name                  jbod17/cal     -
jbod17/cal  available             910T           -
jbod17/cal  used                  534G           -
jbod17/cal  usedbysnapshots       0B             -
jbod17/cal  usedbydataset         534G           -
jbod17/cal  usedbyrefreservation  0B             -
jbod17/cal  usedbychildren        0B             -

[root@zfsarchive1 ~]# zfs get compression,compressratio jbod17/cal
NAME        PROPERTY       VALUE           SOURCE
jbod17/cal  compression    off             local
jbod17/cal  compressratio  1.00x           -
</pre>
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-27T21:33:44Z
body: Running parallel instances on the other datasets in this test pool appears to have increased the aggregate performance,
<pre>
[root@zfsarchive1 ~]# zpool iostat 5
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
jbod17      21.0T  1.12P  1.21K    156  15.2M  20.1M
jbod17      21.0T  1.12P  49.4K  3.25K  1.11G  1.40G
jbod17      21.0T  1.12P  52.7K  2.89K  1.15G  1.26G
jbod17      21.0T  1.12P  47.6K  3.22K  1.16G  1.43G
jbod17      21.1T  1.12P  39.5K  2.87K  1.01G  1.24G
jbod17      21.1T  1.12P  8.55K    626   109M   203M
jbod17      21.1T  1.12P  8.87K    735   117M   196M
jbod17      21.1T  1.12P  13.0K  1.16K   182M   417M
jbod17      21.1T  1.12P  38.5K  4.42K   717M  1.04G
jbod17      21.1T  1.12P  57.4K  6.41K  1.12G  1.66G
jbod17      21.1T  1.12P  58.1K  6.45K  1.13G  1.67G
jbod17      21.1T  1.12P  55.6K  5.14K  1.11G  1.44G
jbod17      21.1T  1.12P  46.3K  3.75K  1.24G  1.67G
</pre>
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-27T21:35:35Z
body: Do I understand correctly that while it is safe to interrupt and restart a rewrite there is no state preserved for a restart to resume where it last left off?
author_association: NONE
user: login: amotin
created_at: 2025-04-28T01:57:51Z
body: > Do I understand correctly that while it is safe to interrupt and restart a rewrite there is no state preserved for a restart to resume where it last left off?

Right.
author_association: MEMBER
user: login: stuartthebruce
created_at: 2025-04-28T20:18:09Z
body: > > Do I understand correctly that while it is safe to interrupt and restart a rewrite there is no state preserved for a restart to resume where it last left off?
> 
> Right.

Since none of the timstamps returned by **stat()** are updated be **zfs rewrite** are there any internal ZFS timestamps  that indicate the last time a file (or a block of data in a file) was last written?

While I am at it, how about last read, e.g., by **zfs send** or **zpool scrub** ?
author_association: NONE
user: login: Momi-V
created_at: 2025-04-28T20:28:50Z
body: > > > Do I understand correctly that while it is safe to interrupt and restart a rewrite there is no state preserved for a restart to resume where it last left off?
> > 
> > 
> > Right.
> 
> Since none of the timstamps returned by **stat()** are updated be **zfs rewrite** are there any internal ZFS timestamps that indicate the last time a file (or a block of data in a file) was last written?
> 
> While I am at it, how about last read, e.g., by **zfs send** or **zpool scrub** ?

I believe the creation time (or at least birth txg) of a record can be queried with zdb and the right combination of -d -b and -v flags. And since records are atomic and rewritten on modification that should be what you're looking for. Whether that's convenient or practical is another question. Reads are (to my knowledge) not logged at record granularity.

EDIT: per file those metrics should show up with less -dd and -vv spamming, but I'm pretty sure at least some of those are what's reported to stat() anyway.
author_association: NONE
user: login: stuartthebruce
created_at: 2025-04-28T20:46:09Z
body: 
> I believe the creation time (or at least birth txg) of a record can be queried with zdb and the right combination of -d -b and -v flags.

I would like to suggest an interesting RFE for **zfs rewrite** would be to accept time ranges to allow recursive scanning to process only files with records in that range. For example: after changing compression algorithm for new files and confirming from in situ performance to retroactively change old files by re-writing those last written before XXX, or a periodic defrag daemon that rewrites files last written before YYY, or wanting to back out a change and re-write files last written after ZZZ.


author_association: NONE
user: login: amotin
created_at: 2025-04-29T18:31:47Z
body: @stuartthebruce To specify any times we'd at very least need https://github.com/openzfs/zfs/pull/16853 to land first.
author_association: MEMBER
user: login: amotin
created_at: 2025-04-29T18:54:01Z
body: Added some tests (not yet tested ;)) and opaque arg field into the IOCTLs structure to make it even more future-proof.
author_association: MEMBER
user: login: amotin
created_at: 2025-05-01T01:10:18Z
body: > Test cases would be good

@tonyhutter Added and passed.
author_association: MEMBER
user: login: tonyhutter
created_at: 2025-05-01T17:06:02Z
body: You'll want to add some input validation to the `zfs rewrite` test case.  I was able to pass strings instead of numbers to `-o|-l` and was also able to pass offsets/lengths beyond the file size.

author_association: CONTRIBUTOR
user: login: amotin
created_at: 2025-05-01T17:16:27Z
body: > You'll want to add some input validation to the `zfs rewrite` test case. I was able to pass strings instead of numbers to `-o|-l` and was also able to pass offsets/lengths beyond the file size.

Values beyond the file size are not illegal there.  Kernel will rewrite only what is actually there.  Can just add a check for non-numeric value, if you prefer.
author_association: MEMBER
user: login: tonyhutter
created_at: 2025-05-01T17:46:47Z
body: > Values beyond the file size are not illegal there. Kernel will rewrite only what is actually there. 

We should error out if a user is trying to seek/rewrite pass the end of the file.  Maybe they accidentally typed in the wrong offset?   `dd` will print an error if you pass a `skip` value past the end of an input file, for example.

Also, I'm wondering if we should not allow `-r` with `-o|-l`.  It's likely a mistake if someone is doing a recursive rewrite with `-o|-l`, and we should give that feedback to the user.  We may also want to error out if they pass `[-o|-l]` with a directory.
author_association: CONTRIBUTOR
user: login: amotin
created_at: 2025-05-01T17:52:31Z
body: > > Values beyond the file size are not illegal there. Kernel will rewrite only what is actually there.
> 
> We should error out if a user is trying to seek/rewrite pass the end of the file. Maybe they accidentally typed in the wrong offset? `dd` will print an error if you pass a `skip` value past the end of an input file, for example.

I don't think we should, considering the code was planned to work under concurrent load.  We should not fail if the file just got truncated.  We achieved our goal by doing nothing.

> Also, I'm wondering if we should not allow `-r` with `-o|-l`. It's likely a mistake if someone is doing a recursive rewrite with `-o|-l`, and we should give that feedback to the user. We may also want to error out if they pass `[-o|-l]` with a directory.

It may be a weird combination, but again not illegal.  I am actually verifying it in the test, just because I can.
author_association: MEMBER
user: login: stuartthebruce
created_at: 2025-05-01T17:56:58Z
body: > Running parallel instances on the other datasets in this test pool appears to have increased the aggregate performance,

These have now finished after 90 hours without any obvious problems,
<pre>
[root@zfsarchive1 ~]# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
jbod17              16.8T   910T   136K  /jbod17
jbod17/cal           534G   910T   534G  /jbod17/cal
jbod17/dqr          4.06T   910T  3.97T  /jbod17/dqr
jbod17/grb.exttrig  4.99T   910T  4.98T  /jbod17/grb.exttrig
jbod17/idq          2.51T   910T  2.51T  /jbod17/idq
jbod17/pe.o4        4.73T   910T  4.63T  /jbod17/pe.o4

[root@zfsarchive1 ~]# parallel 'zfs set compression=off {} && time zfs rewrite -r /{}' ::: jbod17/dqr jbod17/grb.exttrig jbod17/idq jbod17/pe.o4                                                                    

real    616m48.074s
user    0m18.245s
sys     52m49.744s

real    1188m48.153s
user    0m14.149s
sys     59m4.946s

real    1544m41.673s
user    0m34.218s
sys     77m23.340s

real    5414m12.459s
user    2m19.411s
sys     135m19.536s

[root@zfsarchive1 ~]# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
jbod17              39.6T   887T   136K  /jbod17
jbod17/cal           534G   887T   534G  /jbod17/cal
jbod17/dqr          8.63T   887T  4.69T  /jbod17/dqr
jbod17/grb.exttrig  11.4T   887T  6.40T  /jbod17/grb.exttrig
jbod17/idq          9.13T   887T  6.62T  /jbod17/idq
jbod17/pe.o4        9.95T   887T  5.23T  /jbod17/pe.o4
</pre>
author_association: NONE
user: login: tonyhutter
created_at: 2025-05-06T16:21:03Z
body: >>> Values beyond the file size are not illegal there. Kernel will rewrite only what is actually there.

>> We should error out if a user is trying to seek/rewrite pass the end of the file. Maybe they accidentally typed in the wrong offset? dd will print an error if you pass a skip value past the end of an input file, for example.

> I don't think we should, considering the code was planned to work under concurrent load. We should not fail if the file just got truncated. We achieved our goal by doing nothing.

Ok, I still think this is weird, but am not going to let it hold up the review.  Please document the behavior though:
```diff
diff --git a/man/man8/zfs-rewrite.8 b/man/man8/zfs-rewrite.8
index 5fb360d90..86c75e06e 100644
--- a/man/man8/zfs-rewrite.8
+++ b/man/man8/zfs-rewrite.8
@@ -54,6 +54,13 @@ Print names of all successfully rewritten files.
 .It Fl x
 Don't cross file system mount points when recursing.
 .El
+.Pp
+Note: If a
+.Fl l
+or
+.Fl o
+value would cause a rewrite to regions past the end of the file, then those
+regions are silently ignored, and no error is reported.
 .
 .Sh SEE ALSO
 .Xr zfsprops 7
```
author_association: CONTRIBUTOR
user: login: satmandu
created_at: 2025-05-06T16:41:30Z
body: I was using this built against 2.3.2 using `sudo zfs rewrite -rv <path_on_zpool>` on a 8Tb mirror with ~ 100Gb free and ran out of space, presumably because I used a less compressing compression setting than originally set.

I don't know if documenting that possibility might be helpful to users.
author_association: CONTRIBUTOR
user: login: amotin
created_at: 2025-05-06T17:08:36Z
body: @satmandu I've added few more notes about possible effects.
author_association: MEMBER
user: login: owlshrimp
created_at: 2025-05-27T07:08:42Z
body: > @robn I had some of the same thinking myself. But even more than adding file-oriented sub-command to `zfs` I hate adding new commands. Patching third-party tools to use the IOCTL does not sound realistic for cross-platform feature. I found some solace in the fact that we already have `zfs project` sub-command, that is file-oriented too. I am not particularly locked to the `zfs rewrite` name, haven't used it anywhere yet, but `zfs project` we can't rename already.

re:naming I guess "zfs anneal" could be a possibility if "rewrite" is to be used for something else. That's what I've been calling this kind of functionality myself. It seems to me like a fitting description for the task this tool is meant to accomplish.
author_association: NONE
user: login: vedranmiletic
created_at: 2025-05-27T08:20:59Z
body: > zfs anneal

Neat idea! As an added bonus, it's metaphor from metallurgy, just like scrub and resilver.
author_association: NONE
user: login: stuartthebruce
created_at: 2025-06-21T20:47:21Z
body: FYI, I found another use case for this feature: to heal a remote snapshot backup. For example, I have a large (146TB) snapshot with two (out of 55.7M) files that return EIO CKSUM errors on a remote backup but are OK on the primary production instance (long story). I will use `/bin/dd` to force a re-write on the production instance so that in incremental snapshot update will heal the remote backup, but this would be better done with `zfs rewrite`.
author_association: NONE
user: login: MagicRB
created_at: 2025-07-20T10:11:10Z
body: Could this be the basis of a somewhat naive defrag? Similar to a `zpool scrub`, running `zpool defrag` would start a background job which could walk over all files in some order and issue rewrites for each. It would break up snapshots, but honestly not sure if we'd care, I seem to recall BTRFS breaks up snapshots too.
author_association: NONE
user: login: gamanakis
created_at: 2025-07-20T17:03:35Z
body: I just came across this, it seems we can use this to rewrite files to special vdevs, right?
author_association: CONTRIBUTOR
user: login: MagicRB
created_at: 2025-07-20T17:31:31Z
body: from what i read in this thread, its as if you did a `cp file file2 ; rm file ; mv file2 file` but way safer and faster. So yes
author_association: NONE
user: login: fiveangle
created_at: 2025-07-21T11:49:20Z
body: > > zfs anneal
> 
> Neat idea! As an added bonus, it's metaphor from metallurgy, just like scrub and resilver.

I believe you are all thinking about the metaphor `zfs temper`. Annealing would be to metaphorically normalize an object's structure to be as amorphous as possible, while tempering would be to perform a specialized sequence of discrete operations on an object so that its structure was as optimized and robust as possible.
author_association: NONE
user: login: owlshrimp
created_at: 2025-07-21T19:39:23Z
body: > FYI, I found another use case for this feature: to heal a remote snapshot backup. For example, I have a large (146TB) snapshot with two (out of 55.7M) files that return EIO CKSUM errors on a remote backup but are OK on the primary production instance (long story). I will use `/bin/dd` to force a re-write on the production instance so that in incremental snapshot update will heal the remote backup, but this would be better done with `zfs rewrite`.

This does raise a *significant* semantics question for anyone implementing this feature, I think: How do you handle corrupted files? Do you overwrite the file and mark the whole thing correct with valid checksums, with the corrupted bits set to whatever undefined value you read back (or just zeros)? Or do you leave the checksum errors on that file in place after? I suppose this is highly implementation-dependant at the moment.

This particular use case would suggest clearing the errors and setting the file contents to "something, anything" which seems like an "ok" thing to so, so long as the user understands and accepts that this is going to happen. Maybe warn before executing zfs rewrite on a filesystem with checksum errors? (should it stop if it hits one at runtime?)
author_association: NONE
user: login: owlshrimp
created_at: 2025-07-21T19:47:40Z
body: > > > zfs anneal
> > 
> > 
> > Neat idea! As an added bonus, it's metaphor from metallurgy, just like scrub and resilver.
> 
> I believe you are all thinking about the metaphor `zfs temper`. Annealing would be to metaphorically normalize an object's structure to be as amorphous as possible, while tempering would be to perform a specialized sequence of discrete operations on an object so that its structure was as optimized and robust as possible.

Not exactly. Generally when people think of tempering they think of softening material, usually in the case of steel which has been made glass hard and now needs to be pulled back to something more reasonable to use.

I suggested annealing because annealing removes all sorts of discontinuities in the metal in the process of that normalization, in the same way that ZFS rewrite could remove inconsistencies in compression algorithm, raidz stripe length, etc etc. One of the things that "annealing" covers but "tempering" does not is the softening of copper back to a workable state (after it's been work-hardened from lots of hammering) so that further hammering and shaping can be done. The latter is the analogy I had in mind when I suggested the term.
author_association: NONE
user: login: Momi-V
created_at: 2025-07-21T19:54:22Z
body: > > FYI, I found another use case for this feature: to heal a remote snapshot backup. For example, I have a large (146TB) snapshot with two (out of 55.7M) files that return EIO CKSUM errors on a remote backup but are OK on the primary production instance (long story). I will use `/bin/dd` to force a re-write on the production instance so that in incremental snapshot update will heal the remote backup, but this would be better done with `zfs rewrite`.
> 
> This does raise a _significant_ semantics question for anyone implementing this feature, I think: How do you handle corrupted files? Do you overwrite the file and mark the whole thing correct with valid checksums, with the corrupted bits set to whatever undefined value you read back (or just zeros)? Or do you leave the checksum errors on that file in place after? I suppose this is highly implementation-dependant at the moment.
> 
> This particular use case would suggest clearing the errors and setting the file contents to "something, anything" which seems like an "ok" thing to so, so long as the user understands and accepts that this is going to happen. Maybe warn before executing zfs rewrite on a filesystem with checksum errors? (should it stop if it hits one at runtime?)

I believe you missunderstood. The files you rewrite *aren't the corrupted ones*. The corrupted ones are in the offsite snapshot. So anything that's read and written out again is valid data, with passing checksums. This is just about forcing zfs send to retransmit that data without having to change it from userspace.

Usually this would be handled by doing a replication from scratch or with the standard cp, rm, mv cycle on the live version to force replication to transmit the blocks again and have a new, uncorrupted snapshot in the offsite location.
author_association: NONE
user: login: fiveangle
created_at: 2025-07-23T00:47:56Z
body: > Not exactly.

Yes, not exactly. As all metaphors...

Tempering is the final operation to leave an alloy in its toughest, most optimized, and inexorable form. Annealing leaves metal in its softest, most malleable form... the exact opposite goal for user data.

Anyway, the merge was performed with "rewrite" which is far and away the lowest obfuscation, however boring it may be, so is honestly the best choice for the masses.

> one of the things that "annealing" covers [...] is the softening of [metal] back to a workable state [...] so that further hammering and shaping can be done. [That] is the analogy I had in mind when I suggested the term.

👀  I feel like Mugatu in Zoolander, but nothing a pint at the pub won't solve... 🤣 



author_association: NONE
user: login: owlshrimp
created_at: 2025-07-23T00:55:34Z
body: > Tempering is the final operation to leave an alloy in its toughest, most optimized, and inexorable form. Annealing leaves metal in its softest, most malleable form... the exact opposite goal for user data.

Annealing allows one to do further work to a piece of copper, removing the internal dislocations accumulated from previous work. I'm sure that users would like to continue to work with and shape their arrays over time and this gives them a way to remove accumulated inconsistencies at various points along the way.

> Anyway, the merge was performed with "rewrite" which is far and away the lowest obfuscation, however boring it may be, so is honestly the best choice for the masses.

Really my biggest issue with "rewrite" is that it could be confused with block pointer rewrite, the once promised feature never delivered, which it certainly isn't except in the most roundabout way imaginable. Yes, technically rewriting the data rewrites the block pointers, but it's a very different kind of operation.

author_association: NONE
user: login: clhedrick
created_at: 2025-07-23T01:17:53Z
body: How does this interact with snapshots? If I rewrite everything and have a snapshot, am I now using twice the space?
author_association: NONE
user: login: robn
created_at: 2025-07-23T03:12:22Z
body: > How does this interact with snapshots? If I rewrite everything and have a snapshot, am I now using twice the space?

Yes, if you rewrite a block that's in a snapshot, then the snapshot keeps one, and you get a new one, so it will use more space on the pool. This is mentioned in [`zfs-rewrite(8)`](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rewrite.8.html#NOTES).

It may not be twice the space; it depends on the properties at time of rewrite.
author_association: MEMBER
user: login: robn
created_at: 2025-07-23T03:25:31Z
body: > Really my biggest issue with "rewrite" is that it could be confused with block pointer rewrite, the once promised feature never delivered, which it certainly isn't except in the most roundabout way imaginable. Yes, technically rewriting the data rewrites the block pointers, but it's a very different kind of operation.

I'm not really sure the distinction between "rewrite the data" and "rewrite the block pointer" really means anything here, since the BP describes how to interpret the data, so changing the data layout or transforms necessarily requires the the block pointer to change.

In any case, I doubt there's going to be much confusion. "Block pointer rewrite" is pretty inside-baseball at this point; most people who just use OpenZFS have likely never heard of BPR or if they have, not with any particular idea of what it is or should be. Hell, I've been working on OpenZFS for three years now and I only know it in wish form ("gee, it'd be nice to just upgrade all the block pointers"). Which is what `zfs-rewrite` is, just with a narrower scope.
author_association: MEMBER
user: login: owlshrimp
created_at: 2025-07-23T08:07:04Z
body: How does zfs rewrite interact with an array that has data errors?
author_association: NONE
user: login: maxximino
created_at: 2025-07-23T10:38:13Z
body: > Which is what `zfs-rewrite` is, just with a narrower scope.

AFAIU the biggest difference between "zfs rewrite" and the mythical block-pointer-rewrite is that data rewritten by "zfs rewrite" count as newly-written data, with some unpleasant consequences:
* until you delete old snapshots, the old data is still needed and will still take space. So, it can't be used to implement e.g. arbitrary layout changes in the zpool unless the user agrees to forfeit all snapshots.
* (haven't tried myself, but fairly convinced) if you use zfs send | recv to send backups offsite, you'll be sending everything again, with big waste of bandwidth and space on the receiving side.

At VFS layer, `zfs rewrite` is practically invisible (and awesome). At other layers, not so much. AFAIUm real BP rewrite ideally should be invisible also in other layers. 
author_association: CONTRIBUTOR
user: login: amotin
created_at: 2025-07-23T13:52:32Z
body: > How does zfs rewrite interact with an array that has data errors?

Kernel should return error to user-space, same as read, but user-space there is made to log errors and continue with next file.
author_association: MEMBER
user: login: amotin
created_at: 2025-07-24T23:49:19Z
body: > * (haven't tried myself, but fairly convinced) if you use zfs send | recv to send backups offsite, you'll be sending everything again, with big waste of bandwidth and space on the receiving side.

@maxximino Look here: https://github.com/openzfs/zfs/pull/17565 . :)
author_association: MEMBER
user: login: devifish
created_at: 2025-08-25T12:58:45Z
body: Does this feature solve the problem of raid-z single-disk expansion resulting in less free space than expected?
author_association: NONE
user: login: amotin
created_at: 2025-08-25T13:23:22Z
body: @devifish There are two sides in RAIDZ expansion free space issue.  The first is that old data after expansion really have old parity ratio, taking more space than they could.  Rewrite does help with this.  The second is that free space prediction is still based on the original parity ratio, underestimating free space when reporting.  Rewrite does not help with this, but it is only a reporting issue, and the more you write the lower will be the difference, so you'll be able to use all expected space.
author_association: MEMBER
user: login: owlshrimp
created_at: 2025-08-25T19:03:28Z
body: > The second is that free space prediction is still based on the original parity ratio, underestimating free space when reporting. Rewrite does not help with this, but it is only a reporting issue, and the more you write the lower will be the difference, so you'll be able to use all expected space.

If all the data on the entire raidz (or the entire pool) is rewritten, does the pool ever change to using the new parity ratio in it's calculation of free space?

author_association: NONE
user: login: amotin
created_at: 2025-08-25T19:08:39Z
body: > If all the data on the entire raidz (or the entire pool) is rewritten, does the pool ever change to using the new parity ratio in it's calculation of free space?

It is not currently implemented. But I don't think it is theoretically impossible.
author_association: MEMBER
user: login: CicerBro
created_at: 2025-08-27T09:14:27Z
body: So will this be able to change the recordsize on existing data? I have a few TB of tiny files that I changed the recordsize of. AFAIK send|recv won't do this and I need to use rsync, which isn't really good with millions of small files.
author_association: NONE
user: login: Momi-V
created_at: 2025-08-27T09:19:27Z
body: > So will this be able to change the recordsize on existing data? I have a few TB of tiny files that I changed the recordsize of. AFAIK send|recv won't do this and I need to use rsync, which isn't really good with millions of small files.

Record Size is basically the only thing that isn't changed by rewrite. It rewrites files record by record, and because a single file can only ever have one record size (that can't change as soon as it consists of 2 or more records) the only way to change record size is by creating a new file (like cp --reflink=never, rsync, etc. do) and renaming it back to replace the original.
author_association: NONE
user: login: CicerBro
created_at: 2025-08-27T09:23:11Z
body: > Record Size is basically the only thing that isn't changed by rewrite. It rewrites files record by record, and because a single file can only ever have one record size (that can't change as soon as it consists of 2 or more records) the only way to change record size is by creating a new file (like cp --reflink=never, rsync, etc. do) and renaming it back to replace the original.

Damn. I read `rewrite` and thought "this is exactly what I need" lol. Before ZFS I used rsync to move millions of small files to a different machine and that was a massive pain in the butt. rsync doesn't handle huge amounts of files very well.
author_association: NONE
user: login: the8472
created_at: 2025-08-27T16:58:50Z
body: Since this can rewrite checksums I wonder how airtight the "chain of custody" is, i.e. is there any gap between reading the data with the old checksums, verifying, computing the new checksum and writing it out where a bit could end up being flipped without that being noticed?
author_association: NONE
user: login: owlshrimp
created_at: 2025-08-27T20:11:53Z
body: > Since this can rewrite checksums I wonder how airtight the "chain of custody" is, i.e. is there any gap between reading the data with the old checksums, verifying, computing the new checksum and writing it out where a bit could end up being flipped without that being noticed?

At a guess, no worse than any normal read or write. Whenever reads or writes come from or are presented to userspace they're not checksummed at all. It's sitting in some normal buffer in ram. Underpinning pretty much all storage is the basic assumption that things like your CPU and ram work correctly, which excludes things like random DRAM bitflips. There are ways to mitigate this of course, ECC being the obvious one.

Beyond that you probably start getting into clusters and quorums and distributed consensus, which is kind of out of scope for ZFS which runs on a single host? (Unless you've got some truly bizarre mainframe hardware where every app is run invisibly on multiple CPUs and cross-checked)
author_association: NONE
user: login: the8472
created_at: 2025-08-27T20:39:11Z
body: > no worse than any normal read or write

Well yeah, but for long-term storage once data is written and sitting there it's protected by checksum verification and regular scrubs, it won't be altered.

But taking it out of this state and moving it through a read-write cycle introduces a new sources of corruption, however small it might be. And if a new checksum is calculated then this may be silent.

I think one could do better than a plain read-write. For example O_DIRECT read, checksum, write, and then a second pass of reads from source and destination (through separate buffers or even NUMA nodes or whatever if one is extra paranoid) to ensure that what was written is bit/checksum-identical to the original.
author_association: NONE
user: login: stuartthebruce
created_at: 2025-08-27T20:52:16Z
body: > But taking it out of this state and moving it through a read-write cycle introduces a new sources of corruption, however small it might be. And if a new checksum is calculated then this may be silent.

How about considering it an error if the new checksum does not match the original, and if so keeping the original bits by rolling back the transaction for that specific file? Note, if that seems like a good idea then this extra check should presumably also confirm the checksum was not expectedly different due to a file modification during the re-write transaction?
author_association: NONE
user: login: owlshrimp
created_at: 2025-08-27T21:02:56Z
body: > > no worse than any normal read or write
> 
> Well yeah, but for long-term storage once data is written and sitting there it's protected by checksum verification and regular scrubs, it won't be altered.
> 
> But taking it out of this state and moving it through a read-write cycle introduces a new sources of corruption, however small it might be. And if a new checksum is calculated then this may be silent.
> 
That "state" is "in ram." You are probably incurring the **same risk every time you scrub your array,** since corruption in ram then could lead ZFS to conclude your data is bad and needs to be "fixed." If you're this worried about corruption happening in ram, buy ECC. That should completely solve the issue.

> I think one could do better than a plain read-write. For example O_DIRECT read, checksum, write, and then a second pass of reads from source and destination (through separate buffers or even NUMA nodes or whatever if one is extra paranoid) to ensure that what was written is bit/checksum-identical to the original.

You are running in circles chasing your tail, because you are both distrusting your CPU and RAM, while ALSO trusting them COMPLETELY. You are trusting that they're reading the correct data both times. What if the address is corrupted? You are trusting them to compute the checksum correctly. You are trusting them to report the correct result when you COMPARE the checksums. You are trusting them to write the raw data back out correctly after the checksums have been stripped back off. Remember, CoW filesystems like ZFS *cannot* *by design* write back out to the same location twice.

**The answer to this entire line of reasoning is "don't use ZFS." Go use something like CEPH that is distributed across separate nodes and has a real consensus model.**
author_association: NONE
user: login: Momi-V
created_at: 2025-08-27T21:04:50Z
body: > > no worse than any normal read or write
> 
> Well yeah, but for long-term storage once data is written and sitting there it's protected by checksum verification and regular scrubs, it won't be altered.
> 
> But taking it out of this state and moving it through a read-write cycle introduces a new sources of corruption, however small it might be. And if a new checksum is calculated then this may be silent.
> 
> I think one could do better than a plain read-write. For example O_DIRECT read, checksum, write, and then a second pass of reads from source and destination (through separate buffers or even NUMA nodes or whatever if one is extra paranoid) to ensure that what was written is bit/checksum-identical to the original.

It's a worthwhile discussion and you're free to implement that or file an issue explaining the rationale, but for now this seems like a fairly straightforward implementation mostly based on existing ZFS primitives (basically iterating through a file and marking all records as dirty so they just get transparently rewritten by the lower layers).

Keeping both on disk versions around and active (but unreferenced) to do a second unbuffered read pass on both of them to rule out memory corruption, before finally committing the new one while also handling ongoing I/O transparently (what happens if a file gets partially overwritten/changed from userspace during the rewrite?) isn't nearly as trivial.

This initial version is basically aimed at replacing the userspace shell and python scripts floating around that automate a cp + mv and sometimes can't even do proper locking to ensure a file isn't modified during the copy process. It's relatively clean, simple, doesn't even require ANY on disk format changes, can run without conflicting with other workloads (because the record is just atomically marked as dirty, so any extra incoming write will just be handled by the normal path for multiple overlapping writes).

Further improvements are being made (like the -P physical rewrite flag to skip rewritten records in zfs send) so a feature request for a "strict mode" for systems hosting critical data without ECC RAM could be discussed.

But for now it has the same (or better) guarantees as any alternative rewrite option and should catch everything from disk errors to signal path corruptions on both the read and write path, except for memory errors on non ECC systems during the actual checksum computation.
author_association: NONE
user: login: owlshrimp
created_at: 2025-08-27T21:05:36Z
body: > > But taking it out of this state and moving it through a read-write cycle introduces a new sources of corruption, however small it might be. And if a new checksum is calculated then this may be silent.
> 
> How about considering it an error if the new checksum does not match the original, and if so keeping the original bits by rolling back the transaction for that specific file? Note, if that seems like a good idea then this extra check should presumably also confirm the checksum was not expectedly different due to a file modification during the re-write transaction?

See my other post. There are myriad other ways things can go horribly wrong, and if you distrust your CPU/RAM to such a degree then this problem is completely unsolvable. Stop using a single-node solution like ZFS and go use something with a real distributed consensus model like CEPH.
author_association: NONE
user: login: the8472
created_at: 2025-08-27T21:17:40Z
body: > How about considering it an error if the new checksum does not match the original, and if so keeping the original bits by rolling back the transaction for that specific file?

If the checksum stays the same and this is verified then this is no different then a send-receive. That's fine.

But the feature is described as being able to change the checksum, so in general it seems weaker than send-receive-scrub.

> That "state" is "in ram."

And buses, caches, forwarding buffers, etc. etc.
The data has to pass a lot of hands to go from disk to disk.

> You are probably incurring the same risk every time you scrub your array, since corruption in ram then could lead ZFS to conclude your data is bad and needs to be "fixed."

No? This is covered by copies/parity. Resilver will also have to checksum the reconstruction before even attempting to writing it back. So even a spurious scrub-error should not result in a write of incorrect data.

> You are trusting that they're reading the correct data both times. What if the address is corrupted? You are trusting them to compute the checksum correctly. You are trusting them to report the correct result when you COMPARE the checksums. 

To some extent, I agree, even a verification step is not perfect.

But the failure modes you describe are are less likely to be silent. For example data first has to be corrupted and then an additional verification step has to have a separate failure to flip the checksum-comparison from `false` to `true`, the probabilities of such coincidences go down a lot compared to singular errors that aren't caught because there was a _completely unverified_ step somewhere in the pipeline.

The data sitting around without redundancy, without integrity, enabling silent corruption is what concerns me.
If corruption can be detected by having an unbroken chain of verification, then the residual possibility of errors conspiring to cover up verification failure is more tolerable.

> This initial version is basically aimed at replacing the userspace shell and python scripts floating around that automate a cp + mv and sometimes can't even do proper locking to ensure a file isn't modified during the copy process. 

I see, yes that's an improvement, but in my mind it does fall short of a send-receive then. Possibly even falls short of a paranoid userspace implementation.

author_association: NONE
user: login: Momi-V
created_at: 2025-08-27T21:36:13Z
body: > I see, yes that's an improvement, but in my mind it does fall short of a send-receive then. Possibly even falls short of a paranoid userspace implementation.

send | recv has the same issue: the record arrives on the target system, the checksum is verified and then it is potentially recompressed, changing the checksum and finally it's written out.

Here data is read in, the checksum is verified, then it is potentially recompressed, and finally written out.

ZFS checksums are as far as i know always done on and verified against the physical representaion of the record. In send | recv it's possible to keep the physical representaion the same with -C to tell the sender to send compressed records as long as the receiver accepts them and doesn't overwrite the compression property to use a different level, algorithm, etc.

But that's useless if your goal is to change the compression algorithm, because then the checksum always has to be recomputed at some point. Yes, the process has room for improvement and technically using flock, copying each file individually, running a userspace sha512 to make sure the VFS level contents are the same and then doing an mv is less susceptible to corruption. But it's also not realistic for a production system to have to schedule downtime because you want to flock the postgres data directory.
author_association: NONE
user: login: owlshrimp
created_at: 2025-08-27T21:43:52Z
body: > And buses, caches, forwarding buffers, etc. etc. The data has to pass a lot of hands to go from disk to disk.

> But the failure modes you describe are are less likely to be silent.

Everything you have described doing must **happen** in ram, not just pass through it. Comparison of checksums, calculation of addresses, even the basic variables, pointers, function call addresses, and code of ZFS itself. A random corruption of any of the variables or code of ZFS in ram would instantly and undeniably destroy your entire pool. No matter what verification scheme you propose, it will never be possible to prevent a well-placed bitflip from destroying all of your data in an instant.

If you care about corruption along the way (or indeed, code corruption), use ECC. If you're worried about other parts of your computer lying to you, use a multi-node distributed-consensus system like CEPH that explicitly can handle the total failure/misbehaviour of an entire computer.
author_association: NONE
user: login: the8472
created_at: 2025-08-27T22:06:56Z
body: @HPPinata  Thanks for the explanation. My use is cold data/read-only storage. I'll stick to userspace copies with extra src/dest checksumming then.

If zfs had something like `ioctl_xfs_exchange_range` then userspace impls would be a bit simpler since they'd only have to duplicate the content, not all the attributes.

----

@owlshrimp 

> Everything you have described doing must happen in ram

ECC RAM is great, but it's not the only thing the data flows through, and I don't think it should be the only thing protecting the data, it's still less resilient than SHA256 when it comes to verification.

"all data is end-to-end verified through cryptographic hashes, discrepancies will be detected on read or scrub" is easier to reason about than the handoffs of unverified data between multiple system components with various levels of ECC or parity (if at all... do CPU registers use parity? I don't know).

_Trust, but verify_

>  A random corruption of any of the variables or code of ZFS in ram would instantly and undeniably destroy your entire pool

I doubt that the probabilities of a random data block getting corrupted vs. the entire array getting killed are the same. And when it comes to data integrity probability matters, since we can never have perfect integrity, it's always the nines.

And I'd also expect this kind of failure to be loud, in which case I could at least restore from backups. Silent corruption gets propagated to backups and then the data is permanently damaged.
author_association: NONE
user: login: Momi-V
created_at: 2025-08-27T22:34:49Z
body: Minor note on:
> > That "state" is "in ram."
> 
> And buses, caches, forwarding buffers, etc. etc. The data has to pass a lot of hands to go from disk to disk.

Corruption in buses, caches forwarding buffers etc. is covered for everything I/O side.

The old checksum is verified after the record is read into memory and the new one computed before it's written out to disk.

So any early corruption is caught on the read and any late corruption is caught by the next scrub.

So the true "unprotected" signal path is DRAM -> Memory Controller -> CPU Bus & Cache -> Core and Back. The CPU caches I've seen data sheets for all have some sort of multi bit ECC and the Bus / Data Fabrics generally also have at least functional integrity checks. So for all intents and purposes the CPU internal data path is generally treated as integrity protected. So the only real missing link to full hardware level data integrity guarantees is ECC memory.

Of course if your usecase allows for you to go the extra mile and effort of doing a manual verification of the data before and after a rewrite that's an additional layer of protection (Data Fabrics definitely aren't using SHA512 for their internal consistency guarantees), but it's debatable if it's a Filesystems job to make sure there isn't a random bit flip in the code of a compression library causing silent (logic) translation errors in converting from uncompressed to compressed data buffers.
author_association: NONE
user: login: owlshrimp
created_at: 2025-08-27T22:36:56Z
body: > ECC RAM is great, but it's not the only thing the data flows through, and I don't think it should be the only thing protecting the data, it's still less resilient than SHA256 when it comes to verification.

> "all data is end-to-end verified through cryptographic hashes, discrepancies will be detected on read or scrub" is easier to reason about than the handoffs of unverified data between multiple system components with various levels of ECC or parity (if at all... do CPU registers use parity? I don't know).

You are right, registers do not. Look up "mercurial cores." At that level of paranoia though you don't trust your CPU and the problems get even worse. Your only option is multiple computers with an explicit consensus model. ie CEPH.

> > A random corruption of any of the variables or code of ZFS in ram would instantly and undeniably destroy your entire pool
> 
> I doubt that the probabilities of a random data block getting corrupted vs. the entire array getting killed are the same. And when it comes to data integrity probability matters, since we can never have perfect integrity, it's always the nines.

The probability of the entire array getting trashed is **waaaaaaaaaay** higher, for a given bitflip in ram. The code of ZFS sits in your unprotected ECC-less ram from the time you boot up your computer until you shut it down. There's plenty of time for a particle from space to strike it, vs a buffer that's only a couple of MB at most and is continually being freshly rewritten.

> And I'd also expect this kind of failure to be loud, in which case I could at least restore from backups. Silent corruption gets propagated to backups and then the data is permanently damaged.

A small bitflip in ZFS' code that causes an early return during the writeout could result in bad (incomplete) data getting written and never being checked. A bitflip in the checksum comparison code would cause every checksum comparison to return success. A bitflip in code that computes block pointers could result in the continual writing and checking of (correct!) data to the same spot in the array over and over, and you'd never know until you manually tried to access it and found most of it missing.

You are coming up with ad-hoc solutions to a very small subset of guessed-at problems, without really basing any of this on a set of explicit and consistent assumptions about the state of the system and what you can actually rely on.
author_association: NONE
user: login: gmipf
created_at: 2025-12-04T01:53:13Z
body: I started the rewrite. While it was doing its job, to the same pool01 I wrote a 7GB file from my PC, the copy was very slow and it froze for several seconds too. In that time I noticed a file was skipped. Is that related? Can I avoid this?

OS: TrueNAS Scale 25.10.0.1 - Goldeye
Compression on pool01/backups: ZSTD-15

```
...
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a2141.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a620.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a187.mp4
failed to rewrite /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a2478.mp4: Interrupted system call
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a319.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43532.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43295.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43153.mp4
/mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a778.mp4
...
```
EDIT: Seems it was caused by heavy load, changed the compression to zstd-3 and the rest run without any errors. I think that was too heavy for my low power 4 core cpu.  I have redone the failed files afterwards. 

I strongly recommend to log the output to not miss out any files:
`zfs rewrite -rv /mnt/pool01/backups > rewrite_backups.log 2>&1`

author_association: NONE
user: login: Apachez-
created_at: 2025-12-28T02:54:37Z
body: So for the usecase of rewriting files using updated recordsize setting of a dataset we are stuck with solutions such as this (3rd party scripts)?

https://github.com/markusressel/zfs-inplace-rebalancing

As in there is currently no development of fixing "zfs rewrite" so it can also update recordsize (or for that matter adding a new command such as "zfs rebalance" or such who would do this)?
author_association: NONE
user: login: owlshrimp
created_at: 2025-12-28T05:48:22Z
body: > I started the rewrite. While it was doing its job, to the same pool01 I wrote a 7GB file from my PC, the copy was very slow and it froze for several seconds too. In that time I noticed a file was skipped. Is that related? Can I avoid this?
> 
> OS: TrueNAS Scale 25.10.0.1 - Goldeye Compression on pool01/backups: ZSTD-15
> 
> ```
> ...
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a2141.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a620.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a187.mp4
> failed to rewrite /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a2478.mp4: Interrupted system call
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a319.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43532.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43295.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/219719d5f6f3f1d8d5593066332c2f43153.mp4
> /mnt/pool01/backups/jellyfin/20251104152934_10.11.0.0/data/transcodes/1a6443cf9504c3813c93a185e243cd0a778.mp4
> ...
> ```
> 
> EDIT: Seems it was caused by heavy load, changed the compression to zstd-3 and the rest run without any errors. I think that was too heavy for my low power 4 core cpu. I have redone the failed files afterwards.
> 
> I strongly recommend to log the output to not miss out any files: `zfs rewrite -rv /mnt/pool01/backups > rewrite_backups.log 2>&1`

Might be nice to have it report any failures at the end (what is the returncode of zfs rewrite?), maybe even an option to retry them.
author_association: NONE