Illumos 4897 - Space accounting mismatch in L2ARC/zpool

4897 Space accounting mismatch in L2ARC/zpool

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Boris Protopopov <bprotopopov@hotmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>

From the illumos issue tracker:

	L2ARC vdev space usage statistics are calculated as the delta
	between the maximum and minimum vdev offset ever written to
	by the L2ARC fill thread, but do not inform the user of how
	much space in between these two offsets is actually taken up by
	cached buffers. This fix changes that so that vdev space usage
	stats on L2ARC devices accurately track the volume of buffers
	stored on them, allowing users to see the exact L2ARC usage in
	"zpool iostat -v".

References:
  https://www.illumos.org/issues/4897
  https://github.com/illumos/illumos-gate/commit/3038a2b

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2555
Illumos 4881 - zfs send performance regression with embedded data

4881 zfs send performance degradation when embedded block pointers
     are encountered

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>

References:
  https://www.illumos.org/issues/4881
  https://github.com/illumos/illumos-gate/commit/06315b7

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2547
Illumos 4914 - zfs on-disk bookmark structure should be named *_phys_t

4914 zfs on-disk bookmark structure should be named *_phys_t

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Reviewed by: Saso Kiselkov <skiselkov.ml@gmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>

References:
  https://www.illumos.org/issues/4914
  https://github.com/illumos/illumos-gate/commit/7802d7b

Porting notes:

There were a number of zfsonlinux-specific uses of zbookmark_t which
needed to be updated.  This should reduce the likelihood of further
problems like issue #2094 from occurring.

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2558
Add bash completions by Aneurin Price.

These can be manually installed as needed by end users.  They
have been added to the repository so they can be kept up to date
with the latest code.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1588
Use KM_PUSHPAGE in dsl_dataset_rollback_check()

The dsl_dataset_rollback_check() function is executed in the
txg_sync context.  To prevent a potential deadlock due to direct
memory reclaim it must use KM_PUSHPAGE.  This was introduced by
the recent 'zfs bookmark' features, commit da53684.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Eric Dillmann <eric@jave.fr>
Closes #2569
Avoid dynamic allocation of 'search zio'

As part of commit e8b96c6 the search zio used by the
vdev_queue_io_to_issue() function was moved to the heap
to minimize stack usage.  Functionally this is fine, but
to maximize performance it's best to minimize the number
of dynamic allocations.

To avoid this allocation temporary space for the search
zio has been reserved in the vdev_queue structure.  All
access must be serialized through the vq_lock.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Closes #2572
Avoid 128K kmem allocations in mzap_upgrade()

As originally implemented the mzap_upgrade() function will
perform up to SPA_MAXBLOCKSIZE allocations using kmem_alloc().
These large allocations can potentially block indefinitely
if contiguous memory is not available.  Since this allocation
is done under the zap->zap_rwlock it can appear as if there is
a deadlock in zap_lockdir().  This is shown below.

The optimal fix for this would be to rework mzap_upgrade()
such that no large allocations are required.  This could be
done but it would result in us diverging further from the other
implementations.  Therefore I've opted against doing this
unless it becomes absolutely necessary.

Instead mzap_upgrade() has been updated to use zio_buf_alloc()
which can reliably provide buffers of up to SPA_MAXBLOCKSIZE.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Close #2580
Add zfs_iput_async() interface

Handle all iputs in zfs_purgedir() and zfs_inode_destroy()
asynchronously to prevent deadlocks.  When the iputs are allowed
to run synchronously in the destroy call path deadlocks between
xattr directory inodes and their parent file inodes are possible.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Closes #457
Revert "Revert "Revert "Fix unlink/xattr deadlock"""

This reverts commit 7973e46 which brings the basic flow of the
code back in line with the other ZFS implementations.  This
was possible due to the following related changes.

e89260a Directory xattr znodes hold a reference on their parent
6f9548c Fix deadlock in zfs_zget()
0a50679 Add zfs_iput_async() interface
4dd1893 Avoid 128K kmem allocations in mzap_upgrade()

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Closes #457
Closes #2058
Closes #2128
Closes #2240
Replace __va_list with va_list

Most of the code base already uses va_list, which is specified by
iso-c. gcc/glibc provides 'typedef __gnuc_va_list va_list'. and
when not using gcc/glibc we can't expect to find __gnuc_va_list.

Signed-off-by: Alec Salazar <alec.j.salazar@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2588
Avoid PAGESIZE redefinition

Add #ifndef PAGESIZE to avoid redefinition warning on platforms
where this value is already provided.

Signed-off-by: Alec Salazar <alec.j.salazar@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2588
Correct autodetection of bootfs property

Remove lines that contain only a hyphen (match '^-$' instead of '-').

I had a root fs with a hyphen in the name (fedora/ROOT/Fedora20-Dev),
it was not detected because sed eliminated that line of output from
'zpool list -Ho bootfs'.

Signed-off-by: Evan Susarret <evansus@gmail.com>
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2196
Include sys/taskq.h in linux/vfs_compat.h

We should have included sys/taskq.h directly because we use the taskq
code here, but we instead had files that included sys/taskq.h also
include sys/kmem.h, which happened to include sys/taskq.h. sys/kmem.h no
longer does this, so we must define the include as we should
have done in the first place.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2411
Create an 'overlay' property

Add a new 'overlay' property (default 'off') that controls whether the
filesystem should be mounted even if the mountpoint is busy or if it
should fail with a 'mountpoint not empty'.

Doing overlay mounts is the default mount behavior on Linux, but not
in ZFS. It have been decided that following the ZFS behavior should
be the default, but this overlay allows for site administrator to
override this decision on a per-dataset basis.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes: #2503
Illumos 4976-4984 - metaslab improvements

4976 zfs should only avoid writing to a failing non-redundant top-level vdev
4978 ztest fails in get_metaslab_refcount()
4979 extend free space histogram to device and pool
4980 metaslabs should have a fragmentation metric
4981 remove fragmented ops vector from block allocator
4982 space_map object should proactively upgrade when feature is enabled
4983 need to collect metaslab information via mdb
4984 device selection should use fragmentation metric
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <adam.leventhal@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>

References:
  https://www.illumos.org/issues/4976
  https://www.illumos.org/issues/4978
  https://www.illumos.org/issues/4979
  https://www.illumos.org/issues/4980
  https://www.illumos.org/issues/4981
  https://www.illumos.org/issues/4982
  https://www.illumos.org/issues/4983
  https://www.illumos.org/issues/4984
  https://github.com/illumos/illumos-gate/commit/2e4c998

Notes:
    The "zdb -M" option has been re-tasked to display the new metaslab
    fragmentation metric and the new "zdb -I" option is used to control
    the maximum number of in-flight I/Os.

    The new fragmentation metric is derived from the space map histogram
    which has been rolled up to the vdev and pool level and is presented
    to the user via "zpool list".

    Add a number of module parameters related to the new metaslab weighting
    logic.

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2595
Don't upgrade a metaslab when the pool is not writable

Illumos 4982 added code to metaslab_fragmentation() to proactively update
space maps when the spacemap_histogram feature is enabled.  This should
only happen when the pool is writeable.

References:
  https://www.illumos.org/issues/4982
  https://github.com/illumos/illumos-gate/commit/2e4c998

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2595
Drive database update

The Intel DC S3500 and Intel DC S3700 are optimized to handle 4KB
sectors well despite of their 8KB page sizes, so we move them to a new
category for enterprise drives where they will receive ashift=12. They
are joined by the Intel 730 series, which uses the same disk controller,
as well as a San Disk enterprise drive. The drive IDs for these two were
obtained by myself with the drive_id utility. The drive ID for the 240GB
Intel 730 model was extrapolated from the drive ID for the 480GB model.

Lastly, we also add some Western Digital mobile drives.  ryuo in
\#zfsonlinux on freenode obtained "ATA     WDC WD2500BEVT-0" from
running drive_id on his own hardware. The additional drives in that
family were extrapolated from that identifer.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2601
Illumos 4631 - zvol_get_stats triggering too many reads

4631 zvol_get_stats triggering too many reads

Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>

References:
  https://www.illumos.org/issues/4631
  https://github.com/illumos/illumos-gate/commit/bbfa8ea

Ported-by: Boris Protopopov <bprotopopov@hotmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2612
Closes #2480
Fixed memory leaks in zevent handling

Some nvlist_t could be leaked in error handling paths.
Also make sure cb argument to zfs_zevent_post() cannnot
be NULL.

Signed-off-by: Isaac Huang <he.huang@intel.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2158
Initramfs scripts for ZoL.
* Put in /usr/share/initramfs-tools/ or /etc/initramfs-tools/.
* With pull request #1476 (not yet merged) comes a verbose warning
  if /usr/bin/net doesn't exist or isn't executable. Just create
  a dummy...
* If user haven't specified pool, but bootfs then use the first part
  of the bootfs path as pool name.
* Add support for booting of a ZFS snapshot.
  Do this by cloning the snapshot into a dataset. If this the resulting
  dataset already exists, destroy it. Then set mountpoint=/ on that
  dataset and mount it on root.
  + If snapshot does not exist, use base dataset (the part before '@')
    as boot filesystem instead.
* Add support for more kernel command line arguments (ideas taken from
  the Fedora/Redhat dracut script):
  + (zfs_force|zfs.force|zfsforce)=(on|yes|1) (ignore case).
  + root=zfs:AUTO => try to find rootfs automatically.
  + root=zfs:<dataset>
  + root=ZFS=<dataset>
* Remove the existing '-f' option to 'zpool import', instead let
  this be controlled on the kernel command line.
* Do not force-set mountpoint=/ - should (??) not be nessesary.
* Support mounting a rootfs with mountpoint=legacy
* Support both RPM based and DEB based system by removing the logic
  from the dracut script and instead use the logic in scripts/zfs-initramfs/scripts/zfs
  for both Debian GNU/Linux (etc) and RedHat/Fedora (etc).
* Only run the local-* script(s) if/when it/they exists.
* Don't auto import pools when loading module - keep better control of the imports.
* If called with only 'rpool' (or only 'zfs-bootfs'), then fake it by setting 'root=zfs:AUTO'
  (to avoid duplication of code etc) so that it will be auto detected later.
  + If pool is specified and we're auto discovering, don't go through ALL pools, only the one we specified.
* Rewrite the auto detection of bootfs in the auto detector.
* Only try to import pool if it haven't already been imported (in the auto detector).
* Add support for only having 'root=pool/dataset' option.
* Remove the use of setting 'ZFS_RPOOL=rpool'. No longer nessesary.
* Check if /scripts/local-* is either file or directory - should be dir, but just to be safe check both.
* Make wait_for_udev wait for 10 seconds.
* Add a configurable ZFS_INITRD_POST_MODPROBE_SLEEP (set in /etc/default/zfs) after the modprobe.
* Merge https://github.com/zfsonlinux/zfs/pull/2196.
  Remove lines that contain only a hyphen (match '^-$' instead of '-').
* Add a 'default' file to go in '/etc/default/zfs'.
* Support mounting additional filesystems (such as /var and /usr etc)
  in the initrd using ZFS_INITRD_ADDITIONAL_DATASETS set in /etc/defaults/zfs.
* Add exceptions to pool imports.
  + Both for init and initrd script.
* Make sure that the zpool.cache file is actually copied onto the initrd!!
* Include /etc/modprobe.d/{zfs,spl}.conf in the initrd if they exist.
* Export all pools found after the modprobe - zfs_autoimport_disable=1'
  don't seem to be working in >= 0.6.3.
  + This snippet will go away as soon as I can figure out why it doesn't,
    it's ugly!!
* Only try /dev/disk/by-* in the initrd if USE_DISK_BY_ID is set.
  + Try this first, then fallback to using the cache file if that exist.
  + Add /dev as a last ditch attempt.