Switch KM_SLEEP to KM_PUSHPAGE

Prevent snapshot_check to initiate I/O during memory allocation.

Signed-off-by: Massimo Maggi <massimo@mmmm.it>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1023
Make sure free ZIO sizes are correct for gang blocks.

Currently, the PSIZE of the block pointer is used as the I/O size for
all free ZIOs. This is incorrect for gang header blocks, for which
PSIZE is the logical size of the gang block data.

This patch fixes the issue by using SPA_GANGBLOCKSIZE as the I/O size
for "free gang header block" ZIOs.
Export avl_is_empty from zavl.

Currently, zavl users cannot call avl_is_empty since it is not
exported. This patch fixes that.
Add TRIM support.

The code builds a map of regions that were freed. On every write the
code consults the map and eventually removes ranges that were freed
before, but are now overwritten.

Freed blocks are not TRIMed immediately. There is a tunable that defines
how many txg we should wait with TRIMming freed blocks (64 by default).

There is a low priority thread that TRIMs ranges when the time comes.
During TRIM we keep in-flight ranges on a list to detect colliding
writes - we have to delay writes that collide with in-flight TRIMs in
case something will be reordered and write will reached the disk before
the TRIM. We don't have to do the same for in-flight writes, as
colliding writes just remove ranges to TRIM.

NOTE: this patch only adds the core infrastucture for TRIM. It doesn't
implement actual discard requests on vdevs. In other words, the patch
in itself doesn't actually do anything (yet).

NOTE: you need DKIOCTRIM support in SPL for this patch to build.

Signed-off-by: Etienne Dechamps <etienne.dechamps@ovh.net>
Fix frame size for trim_map_free_locked().

Currently, compilation fails because trim_map_free_locked() exceeds the
frame size. This is because we're allocating a large structure (zio_t)
on the stack.

This patch fixes the issue by only allocating the fields we need from
the zio structure.
Add ZIO TRIM statistics.

This patch adds a new kstat page: zio_trim, which indicates the total
number of bytes TRIMmed, and the number of successful, unsupported and
failed TRIM requests. This makes it easy to verify if TRIM is actually
working or not.
Don't test for gang blocks when populating the trim map.

Since 2fd4ce81a6748b46728127212a1ba384c008696f, gang block free zios
have correct size, so there's no need to skip them. Besides, the test
fails with a NULL dereference bug on RAID-Z because io_bp is NULL in
this case.
Don't TRIM after the SPA has been frozen.

Currently, the TRIM module doesn't check for spa_freeze_txg when
issuing TRIM requests. In a normal run this isn't an issue since
vdev_config_sync() doesn't wake the TRIM thread if the SPA is frozen.

However, when closing a frozen pool, trim_thread_destroy() gets called,
which wakes the TRIM thread, which results in a TRIM commit. As a
result, all TXGs until spa_syncing_txg - txg_limit get TRIMmed, which
includes spa_freeze_txg.

As a result, when reopening, ZFS tries to read data from the frozen
TXG, and fails.

This patch fixes the issue by taking spa_freeze_txg into account when
commiting TRIMs.
Add TRIM support for disk vdevs using Linux DISCARD interface.

This adds DISCARD support to vdev_disk_physio(), and uses it to execute
DKIOCTRIM requests. The implementation is inspired by the Linux
blkdev_issue_discard() function.
Add TRIM support for file vdevs.

This adds TRIM support for file vdevs, using the newly-added VOP_SPACE
compatibility interface which translates to Linux
fallocate(FALLOC_FL_PUNCH_HOLE). When a TRIM request is issued to a
file vdev, ZFS will "punch a hole" (deallocate space) inside the
underlying file. Note that few Linux filesystems support hole punching
(currently OCFS2, XFS, and ext4).

Note that for this to work, you need to build against a kernel that
supports file hole punching (i.e. Linux >= 2.6.38). In addition, for
this to work in userspace (e.g. ztest), the installed userspace
kernel headers (i.e. /usr/include/linux) must be up-to-date.

NOTE: you need VOP_SPACE() support in SPL for this patch to build.
Add zfs_trim_zero parameter.

This patch adds a new module tunable: zfs_trim_zero. When this
parameter is set to 1, ZFS makes sure that DKIOCTRIM zeroes the range
being TRIMmed. This makes data corruption issues easier to reproduce,
at the cost of greatly decreased performance.

On disk vdevs, zfs_trim_zero replaces DISCARD with zero page writes,
which means it will work even with vdevs that do not support DISCARD.
Note that DISCARD won't be used even on vdevs that support it.

On file vdevs, if FALLOC_FL_PUNCH_HOLE fails and zfs_trim_zero=1, then
a zero buffer will be written instead. It is not necessary to write
zeroes if hole punching works because FALLOC_FL_PUNCH_HOLE specifies
that the specified range must appear as zeroes after hole punching.
Use zfs_trim_zero=1 in ztest.

For ztest to reliably check for corruption issues in the TRIM code, we
need to make sure TRIM zeroes data, even if the underlying filesystem
doesn't support file hole punching.
Fix assertion failure due to stale inflight writes.

ztest runs with TRIM enabled on a backing filesystem that doesn't
support hole punching revealed the following race condition:

 1. A write is issued to a vdev and added to the TRIM map inflight
    writes tree.
 2. The very first TRIM request for this vdev is issued.
 3. The TRIM request fails with EOPNOTSUPP, as expected. vdev_notrim
    is set.
 4. The write completes. trim_map_write_done() is called.
 5. trim_map_write_done() sees that vdev_notrim is set, and returns
    immediately.
 6. As a result, the write stays in the TRIM map inflight writes tree
    forever.
 7. When the vdev is closed, trim_map_destroy() tries to destroy the
    inflight writes tree, which triggers an assertion failure because
    it isn't empty.

This patch fixes the issue by removing the check on vdev_notrim in
trim_map_write_done().
Fix assertion failure due to inflight frees.

ztest runs with TRIM enabled revealed a race condition which occurs when
a vdev close is pending while there are TRIM requests in flight. When
the TRIM ZIOs complete, the SPA lock is released. The SPA lock is then
immediately taken by the vdev close path, which calls trim_map_destroy()
before trim_map_vdev_commit_done() has a chance to run.
trim_map_destroy() then tries to destroy the inflight free tree, which
fails with an assertion failure because it is non-empty.

This patch fixes the issue by making trim_map_destroy() call
trim_map_vdev_commit_done() before destroying the TRIM map.
Fix TRIM with ashift=12.

With ashift=12, zio_vdev_io_start() expands all ZIOs with size < 4096
using zio_push_transform(), allocating a new ZIO buffer in the process.

The allocation is of course unnecessary for DKIOCTRIM ZIOs, which don't
hold any data. This patch makes sure a new buffer is only allocated on
read/write ZIOs.

This also fixes an assertion failure in __vdev_disk_physio because it
was called with DISCARD and a non-null data buffer, which makes no
sense.
Fix an issue with a NULL zio buf free on raidz.

When a FREE ZIO is issued on raidz, vdev_raidz_map_alloc() skips
allocating data for the raidz columns because it is not needed.

vdev_raidz_map_free(), however, has not been updated accordingly, and
still tries to free the column data. As a result, zio_buf_free() is
used with a NULL buffer, which corrupts the zio_buf_cache.

This patch fixes the issue by making vdev_raidz_map_free() do a NULL
check before freeing the column data.

This fixes the following panic when running zconfig.sh -c:

    SPLError: 17651:0:(spl-kmem.c:1936:spl_kmem_cache_alloc()) ASSERTION(obj) failed
    SPLError: 17651:0:(spl-kmem.c:1936:spl_kmem_cache_alloc()) SPL PANIC
Don't register repair writes in the trim map.

The trim map inflight writes tree assumes non-conflicting writes, i.e.
that there will never be two simultaneous write I/Os to the same range
on the same vdev. This seemed like a sane assumption; however, in
actual testing, it appears that repair I/Os can very well conflict
with "normal" writes.

I'm not quite sure if these conflicting writes are supposed to happen
or not, but in the mean time, let's ignore repair writes for now. This
should be safe considering that, by definition, we never repair blocks
that are freed.
Lock SCL_STATE while waiting for TRIM I/Os to complete.

The current code releases SCL_STATE while it is waiting for TRIM ZIOs
to complete. That's dangerous because the vdev configuration could
change while the I/O is in progress.

This patch makes sure SCL_STATE is held during the whole duration of
the I/O. This is consistent with the way e.g. zil_flush_vdevs() does
things.
Don't use ZIO_FLAG_CONFIG_WRITER for TRIM ZIOs.

There's no reason to use the ZIO_FLAG_CONFIG_WRITER flag for TRIM ZIOS.
Remove it.
TRIM the whole vdev on create/add/attach.

With this patch, ZFS will attempt to TRIM the whole vdev in the
following cases:

 - zpool create, zpool add, zpool attach any vdev
 - zpool import spare and L2ARC devices

We do this to make sure we start from a "clean state". This way, the
physical device knows that all the data it held before it was added to
the pool is now obsolete. This is similar to what mke2fs does.