Revert "Fix __zio_execute() asynchronous dispatch"

This reverts commit 91579709fccd3e55a21970742b66c388fb1403db which
limited the asynchronous dispatch to kernel space.  We want to do
this for two reasons:

1) While we have slightly more headroom in user space excessively
   deep stacks have been observed while running ztest, see #2293.

2) Removing this conditional makes the pipeline behave consistently
   regardless of if it's executing in kernel space or user space.
   This way we're more likely to uncover subtle issues with ztest.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2384
Accept udev and dracut paths specified by ./configure

There are two common locations where udev and dracut components are
commonly installed.  When building packages using the 'make rpm|deb'
targets check those common locations and pass them to rpmbuild.  For
non-standard configurations these values can be provided by the
the following configure options:

  --with-udevdir=DIR      install udev helpers [default=check]
  --with-udevruledir=DIR  install udev rules [[UDEVDIR/rules.d]]
  --with-dracutdir=DIR    install dracut helpers [default=check]

When rebuilding using the source packages the per-distribution
default values specified in the spec file will be used.  This is
the preferred way to build packages for a distribution but the
ability to override the defaults is provided as a convenience.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2310
Closes #1680
Fix zfs.spec.in defaults

Commit 2ee4e7da accidentally introduced two issues which only occur
when rebuilding the ZFS source rpm outside the ZFS build system.

1) The _dracutdir, _udevdir, and _udevruledir macros must be checked
   using the 'undefined' keyword.  This was just overlooked in the
   patch review and does not cause a failure when using 'make pkg'
   because the values are provided by the make target.

2) The default _udevruledir path included a typo.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2310
Tag zfs-0.6.3

META file and release log updated.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Fix man zpool property feature_guid

The property name gets mangled with the explanation due to the property
length.  Fixed by putting the explanation on the next line.

Before:
  unsupported@feature_Info rmation about unsupported features that are
  enabled on the pool. See zpool-features(5) for details.

After:
  unsupported@feature_guid
  Information about unsupported features that are enabled on the pool. See
  zpool-features(5) for details.

Signed-off-by: SenH <sen@senhaerens.be>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2419
Add information about the -o option to zpool replace

Users need to be aware that when replacing devices in an existing
pool they may need to override automatically detected ashift value.
This will all depend on the exact hardware they are using.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2024
Improve differing sector size error

When adding or replacing a vdev with a different sector size the
error message should be more useful.  In addition to describing
the problem provide a hint that the '-o ashift' option can be
used to override the optimal default value.

Since using a non-optimal value may incur a significant performance
penalty we should issue this error.  But there a numerous reasons
why a administrator may wish to do this anyway.

Signed-off-by: Niklas Edmundsson <ZNikke@github>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2421
Comment the lack of real_LZ4_uncompress()

Added several comments regarding the removal of real_LZ4_uncompress()
which exists in the reference implementation but has been removed here
since it's not used.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Illumos #4936 fix potential overflow in lz4

4936 lz4 could theoretically overflow a pointer with a certain input
Reviewed by: Saso Kiselkov <skiselkov.ml@gmail.com>
Reviewed by: Keith Wesolowski <keith.wesolowski@joyent.com>
Approved by: Gordon Ross <gordon.ross@nexenta.com>
Ported by: Tim Chase <tim@chase2k.com>

References:
  https://illumos.org/issues/4936
  https://github.com/illumos/illumos-gate/commit/58d0718

Porting notes:

This fixes the widely-reported "20-year-old vulnerability" in
LZO/LZ4 implementations which inherited said bug from the reference
implementation.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2429
Return default value on numeric properties failing the "head check.

Updates 962d52421236fc9cd61d59b4f18cff3276077da9.

The referenced fix to get_numeric_property() caused numeric property
lookups to consider the type of the parent (head) dataset when checking
validity but there are some cases in the caller expects to see the
property's default value even when the lookup is invalid.

One case in which this is true is change_one() which is part of the
renaming infrastructure.  It may look up "zoned" on a snapshot of a volume
which is not valid but it expects to see the default value of false.

There may be other, yet unidentified cases in which zfs_prop_get_int()
is used on technically invalid properties but which expect the property's
default value to be returned.

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Turbo Fredriksson <turbo@bayour.com>
Closes #2320
Document the optional "device" argument for "zpool split"

Most ZFS implementations seemed to have missed this bit of documentation.
The additional text is based on FreeBSD's man page.

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2416
Fix comment spelling errors.

Signed-off-by: Garrison Jensen <garrison.jensen@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2402
Preserve asize when last mirror child promoted to top-level vdev

If the smaller of 2 different sized child vdev's of a mirrored vdev is
detached, and the pool has the autoexpand property set to off, as the
remaining larger vdev is promoted to a top level vdev it fails to retain
the asize of the original top level mirror vdev and therefore partially
autoexpands.

This partially autoexpanded state leaves the new vdev too large to
re-mirror by adding the smaller vdev back in, and the pool fails to
utilize the space until next imported.

If the autoexpand property is set to on, the child vdev grows
in size after it has been promoted to a top level vdev as expected.

This commit causes the remaining child mirror to retain the asize of its
old parent mirror vdev if the autoexpand property is set to off,
this allows the smaller vdev to be re-added if required the vdev
can then be told to expand if required by the usual using zpool online -e.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Andrew Barnes <barnes333@gmail.com>
Signed-off-by: George Wilson <george.wilson@delphix.com>
Closes #1208
Illumos #3641 compressed block histograms with zdb

This patch is a zdb extension of the '-b' option, producing a histogram
of the physical compressed block sizes per DMU object type on disk. The
'-bbbb' option to zdb will uncover this new feature; here's an example
usage on a new pool and snippet of the output it generates:

    # zpool create tank /dev/vd{b,c,d}
    # dd bs=1k  if=/dev/urandom of=/tank/1kfile  count=1
    # dd bs=3k  if=/dev/urandom of=/tank/3kfile  count=1
    # dd bs=64k if=/dev/urandom of=/tank/64kfile count=1
    # zdb -bbbb tank
    ...
         3  68.0K   68.0K   68.0K   22.7K    1.00    34.26  ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      1 *
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
    ...

The blocks are also broken down by their indirection level. Expanding on
the above example:

    # zfs set recordsize=1k tank
    # dd bs=1k if=/dev/urandom of=/tank/2x1kfile count=2
    # zdb -bbbb tank
    ...
         1    16K      1K      2K      2K   16.00     1.02      L1 ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      1 *
         5  70.0K   70.0K   70.0K   14.0K    1.00    35.71      L0 ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      3 ***
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
         6  86.0K   71.0K   72.0K   12.0K    1.21    36.73  ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      4 ****
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
    ...

There's now a single 1K L1 block which is the indirect block needed for
the '2x1kfile' file just created, as well as two more 1K L0 blocks from
the same file.

This can be used to get a distribution of the block sizes used within
the pool, on a per object type basis.

References:
  https://illumos.org/issues/3641
  https://github.com/illumos/illumos-gate/commit/490d05b

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Boris Protopopov <boris.protopopov@me.com>
Closes #2456
Convert zfs_mg_noalloc_threshold to a module parameter and document

The parameter was added as illumos issue 4081 which was committed to
zfsonlinux in ac72fac3eaa569902cad88053167f7d74e7fe7e4.  This patch
documents the parameter and allows for it to be set as a module parameter.

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2483
Illumos 4168, 4169, 4170: ztest, zdb and zhack fixes

4168 ztest assertion failure in dbuf_undirty
4169 verbatim import causes zdb to segfault
4170 zhack leaves pool in ACTIVE state
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
    https://www.illumos.org/issues/4168
    https://www.illumos.org/issues/4169
    https://www.illumos.org/issues/4170
    https://github.com/illumos/illumos-gate/commit/7fdd916

Porting notes:

Of particular interest when troubleshooting corrupted pools, the
commonly-used "zdb -e" operation may perform verbatim imports and
furthermore, it will soon have direct support for verbatim imports via
a new "-V" option.  The 4169 fix eliminates a common segfault case in
which spa_history_log_version() tries to access an un-opened dsl_pool_t.

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2451
Closes #2283
Closes #2467
zdb: Introduce -V for verbatim import

When given a pool name via -e, zdb would attempt an import. If it
failed, then it would attempt a verbatim import. This behavior is
not always desirable so a -V switch is added to zdb to control the
behavior. When specified, a verbatim import is done. Otherwise,
the behavior is as it was previously, except no verbatim import
is done on failure.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2372
Fix zil_commit() NULL dereference

Update the current code to ensure inodes are never dirtied if they are
part of a read-only file system or snapshot.  If they do somehow get
dirtied an attempt will make made to write them to disk.  In the case
of snapshots, which don't have a ZIL, this will result in a NULL
dereference in zil_commit().

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2405
Move metaslab_group_alloc_update() call

This changes moves the called to metaslab_group_alloc_update() to the
metaslab_sync_reassess() function. The original placement of the call
within metaslab_sync_done() appears to have been a simple mistake,
introduced by ac72fac3eaa569902cad88053167f7d74e7fe7e4.

This aligns us more closely to the upstream illumos code base.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Illumos #4101, #4102, #4103, #4105, #4106

4101 metaslab_debug should allow for fine-grained control
4102 space_maps should store more information about themselves
4103 space map object blocksize should be increased
4105 removing a mirrored log device results in a leaked object
4106 asynchronously load metaslab
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Sebastien Roy <seb@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>

Prior to this patch, space_maps were preferred solely based on the
amount of free space left in each. Unfortunately, this heuristic didn't
contain any information about the make-up of that free space, which
meant we could keep preferring and loading a highly fragmented space map
that wouldn't actually have enough contiguous space to satisfy the
allocation; then unloading that space_map and repeating the process.

This change modifies the space_map's to store additional information
about the contiguous space in the space_map, so that we can use this
information to make a better decision about which space_map to load.
This requires reallocating all space_map objects to increase their
bonus buffer size sizes enough to fit the new metadata.

The above feature can be enabled via a new feature flag introduced by
this change: com.delphix:spacemap_histogram

In addition to the above, this patch allows the space_map block size to
be increase. Currently the block size is set to be 4K in size, which has
certain implications including the following:

    * 4K sector devices will not see any compression benefit
    * large space_maps require more metadata on-disk
    * large space_maps require more time to load (typically random reads)

Now the space_map block size can adjust as needed up to the maximum size
set via the space_map_max_blksz variable.

A bug was fixed which resulted in potentially leaking an object when
removing a mirrored log device. The previous logic for vdev_remove() did
not deal with removing top-level vdevs that are interior vdevs (i.e.
mirror) correctly. The problem would occur when removing a mirrored log
device, and result in the DTL space map object being leaked; because
top-level vdevs don't have DTL space map objects associated with them.

References:
  https://www.illumos.org/issues/4101
  https://www.illumos.org/issues/4102
  https://www.illumos.org/issues/4103
  https://www.illumos.org/issues/4105
  https://www.illumos.org/issues/4106
  https://github.com/illumos/illumos-gate/commit/0713e23

Porting notes:

A handful of kmem_alloc() calls were converted to kmem_zalloc(). Also,
the KM_PUSHPAGE and TQ_PUSHPAGE flags were used as necessary.

Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2488