Illumos 4976-4984 - metaslab improvements

4976 zfs should only avoid writing to a failing non-redundant top-level vdev
4978 ztest fails in get_metaslab_refcount()
4979 extend free space histogram to device and pool
4980 metaslabs should have a fragmentation metric
4981 remove fragmented ops vector from block allocator
4982 space_map object should proactively upgrade when feature is enabled
4983 need to collect metaslab information via mdb
4984 device selection should use fragmentation metric
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <adam.leventhal@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>

References:
  https://www.illumos.org/issues/4976
  https://www.illumos.org/issues/4978
  https://www.illumos.org/issues/4979
  https://www.illumos.org/issues/4980
  https://www.illumos.org/issues/4981
  https://www.illumos.org/issues/4982
  https://www.illumos.org/issues/4983
  https://www.illumos.org/issues/4984
  https://github.com/illumos/illumos-gate/commit/2e4c998

Notes:
    The "zdb -M" option has been re-tasked to display the new metaslab
    fragmentation metric and the new "zdb -I" option is used to control
    the maximum number of in-flight I/Os.

    The new fragmentation metric is derived from the space map histogram
    which has been rolled up to the vdev and pool level and is presented
    to the user via "zpool list".

    Add a number of module parameters related to the new metaslab weighting
    logic.

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2595
Don't upgrade a metaslab when the pool is not writable

Illumos 4982 added code to metaslab_fragmentation() to proactively update
space maps when the spacemap_histogram feature is enabled.  This should
only happen when the pool is writeable.

References:
  https://www.illumos.org/issues/4982
  https://github.com/illumos/illumos-gate/commit/2e4c998

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2595
Drive database update

The Intel DC S3500 and Intel DC S3700 are optimized to handle 4KB
sectors well despite of their 8KB page sizes, so we move them to a new
category for enterprise drives where they will receive ashift=12. They
are joined by the Intel 730 series, which uses the same disk controller,
as well as a San Disk enterprise drive. The drive IDs for these two were
obtained by myself with the drive_id utility. The drive ID for the 240GB
Intel 730 model was extrapolated from the drive ID for the 480GB model.

Lastly, we also add some Western Digital mobile drives.  ryuo in
\#zfsonlinux on freenode obtained "ATA     WDC WD2500BEVT-0" from
running drive_id on his own hardware. The additional drives in that
family were extrapolated from that identifer.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2601
=> #1099: turbo/iscsi - iSCSI sharing for ZoL (shareiscsi).

This is iSCSI sharing for ZoL (shareiscsi).
* Supports the following iSCSI implementations (in this order of discovery):
  + IET		http://iscsitarget.sourceforge.net
  + STGT	http://stgt.berlios.de
  + SCST	http://scst.sourceforge.net
    + Requires that SCST was compiled with /sys support (the default).
* Will refuse to unshares an active target (one with sessions).
* Supports the following options to the 'shareiscsi' property:
  + name/iqn	Full iSCSI Qualified Name (IQN), including identifier.
		This is generated by iscsi_generate_target():
		+ Uses the content of an optional /etc/iscsi_target_id file:
		  Example: iqn.YYYY-MM.tld.domain
		+ If this file doesn't exist, it uses todays year, month and
		  domain name to generate the iqn.
		+ The dataset name is appended at the end of the iqn, with
		  slashes replaced with dots.
		  => <iqn>:<dataset>
  + lun		LUN (0-16384)
		Default: 0 (1 for STGT)
  + type	Share mode (fileio, blockio, nullio, disk, tape)
		STGT: ssc, pt
		Default: blockio (disk for STGT)
  + iomode	IO mode (wb, wt, ro)
		STGT: rdwr, aio, mmap, sg, ssc
		Default: wt (rdwr for STGT)
  + blocksize	Logical block size (512, 1024, 2048, 4096)
		Default: Volume blocksize, 4096 if not usable.
		NOTE: Currently not supported for STGT (doesn't seem to be
		      an option for it in tgtadm).
* If not called with a 'name/iqn' value, then force setting it.
  This so that the IQN doesn't change every month (when it is
  regenerated again). It will be generated by iscsi_generate_target()
  (see above).

NOTE:
+ Moved nfs.c:foreach_nfs_shareopt() to libshare.c:foreach_shareopt()
  so that it can be (re)used in smb.c and iscsi.c.
+ Use the list_{create,insert}() etc for keeping tabs of the linked lists
  instead of using a home-made version.
+ A half second delay had to be added in lib/libzfs/libzfs_mount.c:zfs_unmount()
  after the successful unshare. This to avoid 'dataset busy' when destroying
  recursivly.
+ Fix some gcc-4.8.1 compile fixes. Thanx to munix9.
+ Fix error in manpage.
+ Don't set last char in domainname to NULL willy-nilly. Only if newline.
+ Instead of blindingly generate an IQN, use opts->name if it's set.
+ Don't use c99 comments
=> #1099: turbo/iscsi - iSCSI sharing for ZoL (shareiscsi).

Misc iSCSI fixes/cleanups/attempts to get 'zfs rename' to work.

!! Doesn't - a 'zfs rename' works, but most often than not, the rename
!! IOCTL hangs. Not every time and not always on the first dataset being
!! renamed... Seems a little to random to me.
=> #1476: turbo/smbfs_registry-shares - Rewrite of the Libshare/SMBFS sharing property (sharesmb).

Rewrite of the Libshare/SMBFS
* Add support for options to 'sharesmb'.
  Fixes: #1182
  + Move nfs.c:foreach_nfs_shareopt() to libshare.c:foreach_shareopt()
    so that it can be (re)used in smb.c (and later iscsi.c).
  + Call net(8) with the guest_ok option.
  + Rewrite the sharesmb part of the zfs(8) manpage.
    + Add more examples
    + Inform about the (new) options to sharesmb.
  + Add STDERR_VERBOSE to libzfs_run_process() so that we
    can catch any errors from net(8).
  + Don't call smb_retrieve_shares() after enabling a share.
    No point, it's done when really needed anyway.
  + Support 'y', 'yes' and 'true' for guest_ok.
  * If sharesmb=on, just exit cleanly directly at the top of
    get_smb_shareopts_cb().
  * Remove param 2 and 3 from smb_enable_share_one(). That is
    availible in the first param - the impl_share struct.
  * Extra debugging added in smb_enable_share_one() - print
    all our options just before we call net(8).
* Add support for optional, extra script /sbin/zfs_share_smbfs
  to run after shareing is done.
  + Add an example zfs_share_smbfs script
* The option sharesmb should not be inherited by childs.
  Samba is perfectly able to show child file systems...
* Use dataset name in comment, not the smb share name.
* Use REGISTRY SHARES instead of USER SHARES
  + These are a lot more configurable (more possibility to customize the share).
  + Remove the 'acl' option and replace it with 'writeable' instead.
  + On some machines, there is a "global" share. This isn't really a share,
    but the possibility to modify global options. So first get a line of
    _real_ shares using 'net conf listshares' and _then_ cyckle through
    them retrieving info with 'net conf showshare <share>'.
  + Add an example on how to modify a share after ZFS share and what man
    pages to use for more info.
* Speedups (less share retreivals) and make sure the 'net' command is executable.
  + Since smb_shares is global, we only retreive shares if it's NULL
    in smb_retrieve_shares(). It should limit the number of times we
    fork() and read shares (which takes time).
  + If the net command don't exist (is executable), don't even register
    the smb fstype in libshare_smb_init().
  + access() fix - check if NET_CMD_PATH is _executable_ not just 'exists'.
* Run cstyle.pl to fix any code style problems.
* Output error if 'net' can't talk to samba.
* Improve error message if 'net' isn't executable.
* Move the checks from smb_available() to libshare_smb_init()
  to avoid doing them multiple times.
* Convert homemade linked list to the ZFS versions list_* functions (which uses a link_t).
* Script to list SMB shares added.
* Comment out a call to zfs_unshare_smb(). Don't seem to be needed.
  This because zfs_unshare_smb() is called twice if setting 'sharesmb=off' - once
  'somewhere else' and once here (this one AFTER the share have been unshared). That
  makes this one fail! Don't know if this is correct, but it works to comment out this...
=> #2087: turbo/initramfs - Initramfs scripts for ZoL.

Initramfs scripts for ZoL.
* Put in /usr/share/initramfs-tools/ or /etc/initramfs-tools/.
* With pull request #1476 (not yet merged) comes a verbose warning
  if /usr/bin/net doesn't exist or isn't executable. Just create
  a dummy...
* If user haven't specified pool, but bootfs then use the first part
  of the bootfs path as pool name.
* Add support for booting of a ZFS snapshot.
  Do this by cloning the snapshot into a dataset. If this the resulting
  dataset already exists, destroy it. Then set mountpoint=/ on that
  dataset and mount it on root.
  + If snapshot does not exist, use base dataset (the part before '@')
    as boot filesystem instead.
* Add support for more kernel command line arguments (ideas taken from
  the Fedora/Redhat dracut script):
  + (zfs_force|zfs.force|zfsforce)=(on|yes|1) (ignore case).
  + root=zfs:AUTO => try to find rootfs automatically.
  + root=zfs:<dataset>
  + root=ZFS=<dataset>
* Remove the existing '-f' option to 'zpool import', instead let
  this be controlled on the kernel command line.
* Do not force-set mountpoint=/ - should (??) not be nessesary.
* Support mounting a rootfs with mountpoint=legacy
* Support both RPM based and DEB based system by removing the logic
  from the dracut script and instead use the logic in scripts/zfs-initramfs/scripts/zfs
  for both Debian GNU/Linux (etc) and RedHat/Fedora (etc).
* Only run the local-* script(s) if/when it/they exists.
* Don't auto import pools when loading module - keep better control of the imports.
* If called with only 'rpool' (or only 'zfs-bootfs'), then fake it by setting 'root=zfs:AUTO'
  (to avoid duplication of code etc) so that it will be auto detected later.
  + If pool is specified and we're auto discovering, don't go through ALL pools, only the one we specified.
* Rewrite the auto detection of bootfs in the auto detector.
* Only try to import pool if it haven't already been imported (in the auto detector).
* Add support for only having 'root=pool/dataset' option.
* Remove the use of setting 'ZFS_RPOOL=rpool'. No longer nessesary.
* Check if /scripts/local-* is either file or directory - should be dir, but just to be safe check both.
* Make wait_for_udev wait for 10 seconds.
* Add a configurable ZFS_INITRD_POST_MODPROBE_SLEEP (set in /etc/default/zfs) after the modprobe.
* Merge https://github.com/zfsonlinux/zfs/pull/2196.
  Remove lines that contain only a hyphen (match '^-$' instead of '-').
* Add a 'default' file to go in '/etc/default/zfs'.
* Support mounting additional filesystems (such as /var and /usr etc)
  in the initrd using ZFS_INITRD_ADDITIONAL_DATASETS set in /etc/defaults/zfs.
* Add exceptions to pool imports.
  + Both for init and initrd script.
* Make sure that the zpool.cache file is actually copied onto the initrd!!
* Include /etc/modprobe.d/{zfs,spl}.conf in the initrd if they exist.
* Export all pools found after the modprobe - zfs_autoimport_disable=1'
  don't seem to be working in >= 0.6.3.
  + This snippet will go away as soon as I can figure out why it doesn't,
    it's ugly!!
* Only try /dev/disk/by-* in the initrd if USE_DISK_BY_ID is set.
  + Try this first, then fallback to using the cache file if that exist.
=> #2106: turbo/fixes.init_total_rewrite - Base scripts on the init scripts included with Debian GNU/Linux

Base scripts on the init scripts included with Debian GNU/Linux
then take code from the already existing ones, trying to merge
them into one.
+ NFS server service is just called 'nfs' in Redhat based system, and
  nfs-kernel-server on Debian GNU/Linux based.
+ Merge https://github.com/zfsonlinux/zfs/pull/2148 - Inform OpenRC that ZFS uses mtab.
+ Stop after umountfs
+ When stopping, don't check for LOCKDIR/zfs-mount - it won't be mounted,
  so the file won't exist making the script not run.
+ Add a configurable ZFS_INITRD_POST_MODPROBE_SLEEP used in the initrd to sleep after the modprobe.
+ The import function, do_import(), imports pools by name instead of '-a' [all].
  + Test all '/dev/disk/by-*' dirs for import.
+ Fallback on importing the pool using the cache file (if it exists) if the
  'by-id' didn't work.
+ Add exceptions to pool imports.
=> #2512: turbo/issue-2448 - Default to --with-blkid in RPM packaging
                           - Make linking with and finding libblkid required

Default to --with-blkid in RPM packaging

Support for libblkid has been in the ZoL for years but has been
disabled by default in the packaging.  When libblkid is disabled
the 'zpool create' command will not warn you if you're going to
overwrite an existing filesystem.  For this reason the default
behavior has been changed to require libblkid.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2448

Make linking with and finding libblkid required (so that creating
pools on devices with existing filesystem will get a warning).
Use '--without-blkid' to explicitly disable this.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Closes: #2448
=> #1350: behlendorf/issue-1350 - Adds support for property overrides

Adds support for property overrides (WIP)

Adds support for property overrides (-o property=value), property
excludes (-x property) and dataset limits (-l <volume|filesystem>)
to zfs receive.

Both -o and -x options mirror the functionality already available
in Oracle's ZFS implementation which is also mentioned in the
upstream feature request #2745:

The -l option allows receive to be limited to specific datasets within
the stream effectively allowing partial restores from a multi dataset
stream.

References:
  https://www.illumos.org/issues/2745

Ported-by: Milan-Benes
Issue #1350

NOTES: Man page updates missing and must be added, see patch:
http://blog.multiplay.co.uk/dropzone/freebsd/zfs-recv-properties.patch
=> #2291: behlendorf/issue-2158 - Fixed memory leaks in zevent handling.

Fixed memory leaks in zevent handling

Some nvlist_t could be leaked in error handling paths.
Also make sure cb argument to zfs_zevent_post() cannnot
be NULL.

Signed-off-by: Isaac Huang <he.huang@intel.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2158
=> #2338: hybridcluster/zfs-send-p-fix-71011260 - zfs send -p send properties only for snapshots that are actually sent.

zfs send -p send properties only for snapshots that are actually sent

... as opposed to sending properties of all snapshots of the relevant
filesystem.  The previous behavior results in properties being set on
all snapshots on the receiving side, which is quite slow.
More details are here:
  http://thread.gmane.org/gmane.comp.file-systems.openzfs.devel/346

Behavior of zfs send -R is not changed.

send_iterate_fs: orderly iterate snapshots

The previous commit depends on ordered iteration of the snapshots,
otherwise the code won't be able to correctly detect snapshots
that fall into the requested snapshot range.

This commit can be considered a partial cherry-pick of commit
freebsd/freebsd@4995789cde5 where ordered iteration was introduced
to fix a different problem.
We might want to pick up the complete fix eventually.
=> #2484: ryao/zvol - zvol rework

* Set current->bio_{list,tail} to NULL before submit_bio()
  Linux attempts to avoid recursive processing of bios in
  generic_make_request by storing a list of bios being processed in
  current task_struct. Prior to Linux 2.6.34, this was done with
  current->bio_list and current->bio_tail. When current->bio_tail
  non-NULL, generic_make_request() would add IOs to the existing list and
  return. In Linux 2.6.34, these two were consolidated into
  current->bio_list. The value of that pointer serves the purpose that
  current->bio_tail previously served in generic_make_request().

  This will deadlock when we have a generic_make_request function
  submitting IOs on top of ZFS and we execute a code path that is not
  asynchronous until the operation is handed off to the underlying block
  IO layer via zio_wait(). That causes us to block in zio_wait() block on
  a ZIO that is effectively ignored because it will not be processed until
  we return. Being blocked waiting for something that will not happen
  until we return causes a deadlock.

  An example of this is running mkfs.ext4 on a zvol with the zvol
  refactoring code in place. In that case, zil_flush_vdevs() will hand off
  the flush IOs to the block layer through zio_wait() in a synchronous
  path, which deadlocks. We correct this by saving current->bio_tail or
  current->bio_list depending on the kernel, changing the value is to NULL
  for the duration of submit_bio() and changing it back afterward.

  Signed-off-by: Richard Yao <ryao@gentoo.org>

* zvol processing should use struct bio
  The main impetus for this patch is the Illumos taskq import. The ZoL
  zvol code operated in an interrupt context to take advantage of the IO
  elevator and the SPL taskq code was (ab)used to form the bottom half.
  The SPL taskq code was designed to be callable in an interrupt context
  while the Illumos taskq code was designed to expect the assistance of
  the thread thread scheduler. Since Linux's thread scheduler does not
  provide that assistance, we must trampoline through the system workqueue
  to continue providing the bottom half when using the Illumos taskq code.
  This seems unnecessarily complex when we consider that Illumos itself
  does not perform this emulation and the recent multiqueue work in Linux
  has identified interrupt processing as a major bottleneck on Linux:

  http://kernel.dk/systor13-final18.pdf

  The entry point for zvol operations is zfs_request(). That is called in
  an interrupt context by blk_queue_bio(), which is the kernel's generic
  ->make_request() handler. Registering our own handler allows us to avoid
  this. Comments in the code suggest that this was not done to gain the
  benefit of having Linux's IO elevator do merging. However, the DMU
  already does write merging while arc_read() should implicitly merge read
  IOs because only 1 thread is permitted to fetch the buffer. In addition,
  the main consumers of zvols are VMs and iSCSI shares, which have their
  own elevators to merge IOs. Furthermore, commercial ZFSOnLinux
  distributions report that regular files are more performant than zvols.
  That strongly suggests that the original rationale for using Linux'
  generic ->make_request() handler was invalid.

  Some minor refactoring allows us to register zfs_request() as our
  ->make_request() handler. This eliminates an obstacle to importing the
  Illumos taskq code while giving us several additional benefits:

  1. No per zvol spinlocks.
  2. No redundant IO elevator processing.
  3. Interrupts are disabled only when actually necessary.
  4. No redispatching of IOs under heavy load.
  5. Linux's page out routines will properly block.
  6. Many autotools checks become obsolete.

  It is worth noting that the previous zvol code would redispatch IOs when
  all zio taskq threads were in use. The consequence being that IOs could
  be redispatched and potentially, reordered across barriers. This
  explains an obscure IRC report about corruption on zvol-backed iSCSI
  shares triggered by a kernel panic during heavy load. It was thought
  that the initiator did something wrong, but now it is clear that the
  zvol code itself could reorder, which explains the incident.

  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove rq_for_each_segment() autotools check
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove rq_is_sync() autotools check
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove blk_end_request() autotools check.
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove blk_requeue_request() autotools check
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove blk_fetch_request() autotools check
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove blk_rq_pos() autotools check
  Signed-off-by: Richard Yao <ryao@gentoo.org>

* Remove blk_rq_bytes()/blk_rq_sectors autotools checks
  Signed-off-by: Richard Yao <ryao@gentoo.org>
=> #2598: dweeezil/illumos-4970-to-4974 - illumos 4970 through 4974 - extreme rewind enhancements.

illumos 4970 through 4974 - extreme rewind enhancements

4970 need controls on i/o issued by zpool import -XF
4971 zpool import -T should accept hex values
4972 zpool import -T implies extreme rewind, and thus a scrub
4973 spa_load_retry retries the same txg
4974 spa_load_verify() reads all data twice
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>

References:
    https://www.illumos.org/issues/4970
    https://www.illumos.org/issues/4971
    https://www.illumos.org/issues/4972
    https://www.illumos.org/issues/4973
    https://www.illumos.org/issues/4974

Notes:
    This set of patches adds a set of tunable parameters for the "extreme
    rewind" mode of pool import which allows control over the implicit
    scrub performed during such an import.

Ported by: Tim Chase <tim@chase2k.com>
=> #2595: dweeezil/illumos-4976-to-4984-fragmentation - Illumos 4976-4984 - metaslab improvements.

* Illumos 4976-4984 - metaslab improvements
  4976 zfs should only avoid writing to a failing non-redundant top-level vdev
  4978 ztest fails in get_metaslab_refcount()
  4979 extend free space histogram to device and pool
  4980 metaslabs should have a fragmentation metric
  4981 remove fragmented ops vector from block allocator
  4982 space_map object should proactively upgrade when feature is enabled
  4983 need to collect metaslab information via mdb
  4984 device selection should use fragmentation metric
  Reviewed by: Matthew Ahrens <mahrens@delphix.com>
  Reviewed by: Adam Leventhal <adam.leventhal@delphix.com>
  Reviewed by: Christopher Siden <christopher.siden@delphix.com>
  Approved by: Garrett D'Amore <garrett@damore.org>

  References:
      https://www.illumos.org/issues/4976
      https://www.illumos.org/issues/4978
      https://www.illumos.org/issues/4979
      https://www.illumos.org/issues/4980
      https://www.illumos.org/issues/4981
      https://www.illumos.org/issues/4982
      https://www.illumos.org/issues/4983
      https://www.illumos.org/issues/4984

  Notes:
      The "zdb -M" option has been re-tasked to display the new metaslab
      fragmentation metric and the new "zdb -I" option is used to control
      the maximum number of in-flight I/Os.

      The new fragmentation metric is derived from the space map histogram
      which has been rolled up to the vdev and pool level and is presented
      to the user via "zpool list".

      Add a number of module parameters related to the new metaslab weighting
      logic.

  Ported by: Tim Chase <tim@chase2k.com>

* Don't upgrade a metaslab when the pool is not writable
  Illumos 4982 added code to metaslab_fragmentation() to proactively update
  space maps when the spacemap_histogram feature is enabled.  This should
  only happen when the pool is writeable.

  References:
      https://www.illumos.org/issues/4982
=> #2612: bprotopopov/illumos-bbaf8e - illumos-gate commit bbfa8ea8bb4168c969ba27d632dfe0aeec3fc0da
=> #2480: bprotopopov/master - Illumos 4631 - zvol_get_stats triggering too many reads.

illumos-gate commit bbfa8ea8bb4168c969ba27d632dfe0aeec3fc0da
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   Thu Jul 17 09:27:13 2014 -0800

    4631 zvol_get_stats triggering too many reads
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Approved by: Dan McDonald <danmcd@omniti.com>
=> #2479: bprotopopov/zvol-scale - zvol_create_minors(): check snapdev before traversing snapshots
=> #2388: ryao/recv - ZFS_IOC_RECV should only call zvol_create_minors() on zvols.

ZFS_IOC_RECV should only call zvol_create_minors() on zvols

perf profiling revealed significant time spent in these functions on a
system without zvols during a recv. I know of one company that is
explicitly disabling this in their own builds because they found that it
wastes CPU time. This is the intermediate approach.

Signed-off-by: Richard Yao <ryao@gentoo.org>
My commits...