zfsonlinux/zfs — pull request #1016 by dechamps (2012-10-05 08:03:18+00:00)
Title: TRIM/UNMAP/DISCARD support for vdevs (2)

This adds TRIM (a.k.a UNMAP, DISCARD, hole punching) support for vdevs. The original patch is from Pawel Jakub Dawidek who wrote it for FreeBSD. I just ported it to ZFS On Linux with added features and fixed bugs. This is a clean version of #924. Note that you need zfsonlinux/spl#168 to build this.

This patch should be 99% stable. I'm still not sure about the last 1% because I'm having ztest failures which are difficult to reproduce and might be caused by ZoL ztest bugs. Unfortunately, I'm running out of time and I won't be able to investigate any further, so I'm leaving this to anyone interested.

In any case, if there are indeed bugs, actual failures should be extremely rare and are very unlikely to cause any data corruption. Nevertheless, I would advise anyone playing with this to err on the safe side with a good backup policy.

@behlendorf: you told me George Wilson is interested in this work. You should point him to this pull request for review. Hopefully he'll be able to tell if there are stability issues with this code or not.

- [FreeBSD thread][freebsd]
- [Illumos thread][illumos]

## Features ##

- Automatic, fully transparent TRIM on data, slog and cache vdevs. TRIM requests are not immediately issued in order not to break pool disaster discovery, and are batched for optimal performance.
- Whole vdev TRIM on zpool create/add/attach (similar to what mke2fs does).
- Whole vdev TRIM on cache vdevs on export/remove/destroy.
- Supports disk vdevs using the standard Linux DISCARD interface (which translates to ATA TRIM, SCSI UNMAP, or any other method supported by the vdev block device driver)
- Supports file vdevs using the Linux file hole punching interfaces.

## Usage instructions ##

Note that the new feature is enabled by default. To disable it, use `zfs_notrim=0` as module parameter. For TRIM to work, you need to have vdevs that support it, e.g. SSDs. If you don't have any, you can still test using a Linux RAM block device (`rd` module), as it supports discards. Note that if you choose to test with `rd`, make sure to use `ashift=12`, else you will trigger [a corruption bug in rd][1].

There is a module parameter (`trim_txg_limit`) that specifies after how many TXGs the freed data is actually discarded on the device (default 32). Indeed, the algorithm doesn't discard data immediately so that pool disaster recovery (i.e. `zpool -F`) don't break. I suggest you test with the default and with a low value (e.g. 1, 0) to make the algorithm more aggressive, which might trigger different bugs.

There is a module parameter (`trim_txg_batch`) that specifies how many TXGs are squashed together in order to batch (merge) TRIM requests (default 32). The optimal value heavily depends on your workload and the performance characteristics of your physical vdevs.

Module parameters `trim_l2arc_limit` and `trim_l2arc_batch` work the same way but are specific to cache devices. They are expressed in seconds instead of TXGs.

There is a module parameter (`zfs_trim_zero`) that, when enabled, makes sure TRIMmed data is zeroed. Note that for disk vdevs, this effectively means that TRIM is disabled and replaced by "write zeroes". This mode allows ztest to test TRIM even on filesystems that don't support file hole punching. Outside ztest, it can also be used to diagnose corruption issues more reliably. Note that, for obvious reasons, performance greatly suffers in this mode.

The patch introduces a new kstat page in `/proc/spl/kstat/zfs/zio_trim`. It allows you to quickly check if TRIM is actually working or not. Keep in mind that if you're using a high `trim_txg_limit` (e.g. 64), then it may take a while before TRIM requests are actually issued.

## TRIM support on file vdevs (a.k.a file hole punching) ##

There are a few prerequisites for *file* (not disk) vdev support using Linux file hole punching capabilities:
- The underlying filesystem (where the file vdevs are stored) must support file hole punching. Currently, only OCFS2, XFS and ext4 support it. tmpfs supports it via the `truncate_range` interface, so it should work (albeit only from kernel space). Ironically, ZFS itself doesn't (yet).
- Of course, `zfs_notrim` still needs to be set to `0`, no matter what the vdev type is.
- In the case of ZFS in userspace (e.g. `ztest`), be sure that your installed Linux headers (i.e. `/usr/include/linux`) are up to date (>= 2.6.38). Also, as mentioned above, if you're using `ztest`, be sure to run it on a filesystem that supports file hole punching (tmpfs doesn't). Note that you can't change parameters using `ztest`, so you'll have to hack `trim_map.c` to manually set `zfs_notrim=0` by default and rebuild.

A good way to be sure that TRIM support on file vdev works is to do something like this with `trim_txg_limit=0`, assuming `/store` is stored on ext4 or another filesystem with file hole punching support:

# dd if=/dev/zero of=/store/pool bs=1 count=1 seek=1024M
# zpool create punchhole /store/pool
# du -h /store/pool
1.2M	/store/pool
# dd if=/dev/urandom of=/punchhole/foo bs=1M count=100
# du -h /store/pool
102M	/mnt/file
# rm /punchhole/foo
# du -h /store/pool
1.7M	/store/pool

Again, this section only applies to *file* vdevs, not disk (block device) vdevs. Most users have no use for TRIM support on file vdevs.

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=47711
[freebsd]: http://lists.freebsd.org/pipermail/freebsd-current/2012-September/036777.html
[illumos]: http://www.listbox.com/member/archive/182191/2012/09/sort/time_rev/page/1/entry/2:96/20120923191210:18E22D6A-05D4-11E2-8CE8-908B9596E3B6/