ZFS replay transaction error 5

When zfs_replay_write() replays TX_WRITE records from ZIL
it calls zpl_write_common() to perform the actual write.
zpl_write_common() returns the number of bytes written
(similar to write() system call) or an (negative) error.
However, the code expects the positive return value to be
a residual counter. Thus when zpl_write_common() successfully
completes it is mistakenly considered to be a partial write and
the error code delivered further. At this point the ZIL processing
is aborted with famous "ZFS replay transaction error 5" error
message given to the message buffer.

The fix is to compare the zpl_write_commmon() return value with
the buffer size and flag error only when they disagree.

Signed-off-by: Cyril Plisko <cyril.plisko@mountall.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #933
Switch KM_SLEEP to KM_PUSHPAGE

This warning indicates the incorrect use of KM_SLEEP in a call
path which must use KM_PUSHPAGE to avoid deadlocking in direct
reclaim.  See commit b8d06fca089fae4680c3a552fc55c512bfb02202
for additional details.

  SPL: Fixing allocation for task txg_sync (6093) which
  used GFP flags 0x297bda7c with PF_NOFS set

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #917
Improve `zpool import` search behavior

The goal of this change is to make 'zpool import' prefer to use
the peristent /dev/mapper or /dev/disk/by-* paths.  These are far
preferable to the devices in /dev/ whos names are not persistent
and are determined by the order in which a device is detected.

This patch improves things by changing the default search path from
just to the top level /dev/ directory to (in order):

  /dev/disk/by-vdev   - Custom rules, use first if they exist
  /dev/disk/zpool     - Custom rules, use first if they exist
  /dev/mapper         - Use multipath devices before components
  /dev/disk/by-uuid   - Single unique entry and persistent
  /dev/disk/by-id     - May be multiple entries and persistent
  /dev/disk/by-path   - Encodes physical location and persistent
  /dev/disk/by-label  - Custom persistent labels
  /dev                - UNSAFE device names will change

The default search path can be overriden by setting the
ZPOOL_IMPORT_PATH environment variable.  This must be a colon
delimited list of paths which are searched for vdevs.  If the
'zpool import -d' option is specified only those listed paths
will be searched.

Finally, when multiple paths to the same device are found.  If one
of the paths is an exact match for the path used last time to import
the pool it will be used.  When there are no exact matches the
prefered path will be determined by the provided search order.

This means you can still import a pool and force specific names by
providing the -d <path> option.  And the prefered names will persist
as long as those paths exist on your system.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #965
Illumos #2088 zdb could use a reasonable manual page

Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Reviewed by: Steve Gonczi <gonczi@comcast.net>
Reviewed by: Richard Elling <richard.elling@richardelling.com>
Approved by: Garrett D'Amore <garrett@damore.org>

References:
      https://www.illumos.org/issues/2088

Ported by: Cyril Plisko <cyril.plisko@mountall.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #682
zfs-0.6.0-rc11
Seg fault 'zpool import -d /dev/disk/by-id -a'

Introduced by commit 44867b6d6effc1628dd00c36821ab044f89fb988.
We should of course check to ensure best isn't NULL before
attempting to dereference it.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #974
Make sure free ZIO sizes are correct for gang blocks.

Currently, the PSIZE of the block pointer is used as the I/O size for
all free ZIOs. This is incorrect for gang header blocks, for which
PSIZE is the logical size of the gang block data.

This patch fixes the issue by using SPA_GANGBLOCKSIZE as the I/O size
for "free gang header block" ZIOs.
Export avl_is_empty from zavl.

Currently, zavl users cannot call avl_is_empty since it is not
exported. This patch fixes that.
Add TRIM support.

The code builds a map of regions that were freed. On every write the
code consults the map and eventually removes ranges that were freed
before, but are now overwritten.

Freed blocks are not TRIMed immediately. There is a tunable that defines
how many txg we should wait with TRIMming freed blocks (64 by default).

There is a low priority thread that TRIMs ranges when the time comes.
During TRIM we keep in-flight ranges on a list to detect colliding
writes - we have to delay writes that collide with in-flight TRIMs in
case something will be reordered and write will reached the disk before
the TRIM. We don't have to do the same for in-flight writes, as
colliding writes just remove ranges to TRIM.

NOTE: this patch only adds the core infrastucture for TRIM. It doesn't
implement actual discard requests on vdevs. In other words, the patch
in itself doesn't actually do anything (yet).

NOTE: you need DKIOCTRIM support in SPL for this patch to build.

Signed-off-by: Etienne Dechamps <etienne.dechamps@ovh.net>
Fix frame size for trim_map_free_locked().

Currently, compilation fails because trim_map_free_locked() exceeds the
frame size. This is because we're allocating a large structure (zio_t)
on the stack.

This patch fixes the issue by only allocating the fields we need from
the zio structure.
Don't test for gang blocks when populating the trim map.

Since 2fd4ce81a6748b46728127212a1ba384c008696f, gang block free zios
have correct size, so there's no need to skip them. Besides, the test
fails with a NULL dereference bug on RAID-Z because io_bp is NULL in
this case.
Don't TRIM after the SPA has been frozen.

Currently, the TRIM module doesn't check for spa_freeze_txg when
issuing TRIM requests. In a normal run this isn't an issue since
vdev_config_sync() doesn't wake the TRIM thread if the SPA is frozen.

However, when closing a frozen pool, trim_thread_destroy() gets called,
which wakes the TRIM thread, which results in a TRIM commit. As a
result, all TXGs until spa_syncing_txg - txg_limit get TRIMmed, which
includes spa_freeze_txg.

As a result, when reopening, ZFS tries to read data from the frozen
TXG, and fails.

This patch fixes the issue by taking spa_freeze_txg into account when
commiting TRIMs.
Add TRIM support for disk vdevs using Linux DISCARD interface.

This adds DISCARD support to vdev_disk_physio(), and uses it to execute
DKIOCTRIM requests. The implementation is inspired by the Linux
blkdev_issue_discard() function.
Add TRIM support for file vdevs.

This adds TRIM support for file vdevs, using the newly-added VOP_SPACE
compatibility interface which translates to Linux
fallocate(FALLOC_FL_PUNCH_HOLE). When a TRIM request is issued to a
file vdev, ZFS will "punch a hole" (deallocate space) inside the
underlying file. Note that few Linux filesystems support hole punching
(currently OCFS2, XFS, and ext4).

Note that for this to work, you need to build against a kernel that
supports file hole punching (i.e. Linux >= 2.6.38). In addition, for
this to work in userspace (e.g. ztest), the installed userspace
kernel headers (i.e. /usr/include/linux) must be up-to-date.

NOTE: you need VOP_SPACE() support in SPL for this patch to build.
Fix assertion failure due to stale inflight writes.

ztest runs with TRIM enabled on a backing filesystem that doesn't
support hole punching revealed the following race condition:

 1. A write is issued to a vdev and added to the TRIM map inflight
    writes tree.
 2. The very first TRIM request for this vdev is issued.
 3. The TRIM request fails with EOPNOTSUPP, as expected. vdev_notrim
    is set.
 4. The write completes. trim_map_write_done() is called.
 5. trim_map_write_done() sees that vdev_notrim is set, and returns
    immediately.
 6. As a result, the write stays in the TRIM map inflight writes tree
    forever.
 7. When the vdev is closed, trim_map_destroy() tries to destroy the
    inflight writes tree, which triggers an assertion failure because
    it isn't empty.

This patch fixes the issue by removing the check on vdev_notrim in
trim_map_write_done().
Fix assertion failure due to inflight frees.

ztest runs with TRIM enabled revealed a race condition which occurs when
a vdev close is pending while there are TRIM requests in flight. When
the TRIM ZIOs complete, the SPA lock is released. The SPA lock is then
immediately taken by the vdev close path, which calls trim_map_destroy()
before trim_map_vdev_commit_done() has a chance to run.
trim_map_destroy() then tries to destroy the inflight free tree, which
fails with an assertion failure because it is non-empty.

This patch fixes the issue by making trim_map_destroy() call
trim_map_vdev_commit_done() before destroying the TRIM map.
Fix TRIM with ashift=12.

With ashift=12, zio_vdev_io_start() expands all ZIOs with size < 4096
using zio_push_transform(), allocating a new ZIO buffer in the process.

The allocation is of course unnecessary for DKIOCTRIM ZIOs, which don't
hold any data. This patch makes sure a new buffer is only allocated on
read/write ZIOs.

This also fixes an assertion failure in __vdev_disk_physio because it
was called with DISCARD and a non-null data buffer, which makes no
sense.
Add ZIO TRIM statistics.

This patch adds a new kstat page: zio_trim, which indicates the total
number of bytes TRIMmed, and the number of successful, unsupported and
failed TRIM requests. This makes it easy to verify if TRIM is actually
working or not.
Add zfs_trim_zero parameter.

This patch adds a new module tunable: zfs_trim_zero. When this
parameter is set to 1, ZFS makes sure that DKIOCTRIM zeroes the range
being TRIMmed. This makes data corruption issues easier to reproduce,
at the cost of greatly decreased performance.

On disk vdevs, zfs_trim_zero replaces DISCARD with zero page writes,
which means it will work even with vdevs that do not support DISCARD.
Note that DISCARD won't be used even on vdevs that support it.

On file vdevs, if FALLOC_FL_PUNCH_HOLE fails and zfs_trim_zero=1, then
a zero buffer will be written instead. It is not necessary to write
zeroes if hole punching works because FALLOC_FL_PUNCH_HOLE specifies
that the specified range must appear as zeroes after hole punching.
Use zfs_trim_zero=1 in ztest.

For ztest to reliably check for corruption issues in the TRIM code, we
need to make sure TRIM zeroes data, even if the underlying filesystem
doesn't support file hole punching.