Illumos #3035 LZ4 compression support in ZFS and GRUB

3035 LZ4 compression support in ZFS and GRUB

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Christopher Siden <csiden@delphix.com>

References:
  illumos/illumos-gate@a6f561b4aee75d0d028e7b36b151c8ed8a86bc76
  https://www.illumos.org/issues/3035
  http://wiki.illumos.org/display/illumos/LZ4+Compression+In+ZFS

This patch has been slightly modified from the upstream Illumos
version to be compatible with Linux.  Due to the very limited
stack space in the kernel a lz4 workspace kmem cache is used.
Since we are using gcc we are also able to take advantage of the
gcc optimized __builtin_ctz functions.

Support for GRUB has been dropped from this patch.  That code
is available but those changes will need to made to the upstream
GRUB package.

Lastly, several hunks of dead code were dropped for clarity.  They
include the functions real_LZ4_uncompress(), LZ4_compressBound()
and the Visual Studio specific hunks wrapped in _MSC_VER.

Ported-by: Eric Dillmann <eric@jave.fr>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1217
Remove NPTL_GUARD_WITHIN_STACK

Commit 4b2f65b253952c5103311cc8bb4b8cdc6836fd7e increased the user
space stack by 4x to resolve certain stack overflows.  As such it
no longer makes sense to worry about a single extra page which
might or might not be part of the process stack.  There is now
ample headroom for normal usage.

By eliminating this configure check we are also resolving the
following segfault which intentionally occurs at configure time
and may be logged in dmesg.

  conftest[22156]: segfault at 7fbf18a47e48 ip 00000000004007fe
  sp 00007fbf18a4be50 error 6 in conftest[400000+1000]

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Retire zpool_id infrastructure

In the interest of maintaining only one udev helper to give vdevs
user friendly names, the zpool_id and zpool_layout infrastructure
is being retired.  They are superseded by vdev_id which incorporates
all the previous functionality.

Documentation for the new vdev_id(8) helper and its configuration
file, vdev_id.conf(5), can be found in their respective man pages.
Several useful example files are installed under /etc/zfs/.

  /etc/zfs/vdev_id.conf.alias.example
  /etc/zfs/vdev_id.conf.multipath.example
  /etc/zfs/vdev_id.conf.sas_direct.example
  /etc/zfs/vdev_id.conf.sas_switch.example

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #981
Illumos #3447 improve the comment in txg.c

3447 improve the comment in txg.c

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: Richard Elling <richard.elling@dey-sys.com>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
  illumos/illumos-gate@adbbcfface63b3a71922d5a25d34a2018c0435de
  https://www.illumos.org/issues/3447

Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Fix rounding discrepancy in sa_find_sizes()

A rounding discrepancy exists between how sa_build_layouts() and
sa_find_sizes() calculate when the spill block needs to be kicked in.
This results in a narrow size range where sa_build_layouts() believes
there must be a spill block allocated but due to the discrepancy there
isn't.  A panic then occurs when the hdl->sa_spill NULL pointer is
dereferenced.

The following reproducer for this bug was isolated:

    truncate -s 128m /tmp/tank
    zpool create tank /tmp/tank
    zfs create -o xattr=sa tank/fish
    ln -s `perl -e 'print "z" x 41'` /tank/fish/z
    setfattr -hn trusted.foo -v`perl -e 'print "z"x45'` /tank/fish/z

This test results in roughly the following system attribute (SA)
layout:

  176 bytes - "standard" SA's
   41 bytes - name of symbolic link target
  100 bytes - XDR encoded nvlist for xattr
  ---
  317 bytes - total

Because 317 is less than DN_MAX_BONUSLEN (320), sa_find_sizes()
decides no spill block is needed. But sa_build_layouts() rounds 41 up
to 48 when computing the space requirements so it tries to switch to
the spill block.

Note that we were only able to reproduce this bug using a combination
of symbolic links and the Linux-specific xattr=sa dataset property.
So while this issue is not technically Linux-specific, it may be
difficult or impossible to hit the narrow size range needed to
reproduce it on other platforms.

To fix the discrepancy, round the running total in sa_find_sizes() up
to an 8-byte boundary before accounting for each SA, since this is how
they will be stored in the bonus and (possibly) spill buffers.

To make the intent of the code more clear, explicitly assert key
assumptions about expected alignment of data and whether spill-over
will occur.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1240
Fix mismatch between SA header size and layout

When a system attribute layout is created an inconsistency may occur
between the system attribute header (sa_hdr_phys_t) size and the
variable-sized attribute count stored in the layout.  The inconsistency
results in the following failed assertion when SA_HDR_SIZE_MATCH_LAYOUT
returns false:

SPLError: 11315:0:(sa.c:1541:sa_find_idx_tab())
ASSERTION((IS_SA_BONUSTYPE(bonustype) && SA_HDR_SIZE_MATCH_LAYOUT(hdr,
tb)) || !IS_SA_BONUSTYPE(bonustype) || (IS_SA_BONUSTYPE(bonustype) &&
hdr->sa_layout_info == 0)) failed

The bug originates in this snippet from sa_find_sizes().

    if (is_var_sz && var_size > 1) {
            if (P2ROUNDUP(hdrsize + sizeof (uint16_t),
                *total < full_space) {
                    hdrsize += sizeof (uint16_t);

This assumes that the current variable-sized attribute will be stored in
the current buffer and accounts for the space needed to store its size
in the sa_hdr_phys_t. However if the next attribute spills over we need
to store a blkptr_t at the end of the bonus buffer to point to the spill
block. If the current attribute is in the way of the blkptr_t then it
too will be relocated into the spill block. But since we've already
accounted for it in the header size we get the inconsistency described
above.

To avoid this, record the index of the last variable-sized attribute
that prompted a hdrsize increase, and reverse the increase if we later
determine that that attribute will be relocated to the spill block.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1250
Honor 80 character limit in 'zpool status'

This is a minor nit, but the second line of the 'action' message
when you need to upgrade your pool to support feature flags exceeds
the standard 80 character limit.  Fix it by moving the word
'feature' on to the third line.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add zfs_disable_dup_eviction module option

Commit 1eb5bfa introduced a new zfs_disable_dup_eviction tunable.
It should have been made available as a module option in the
original patch but was overlooked.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add zfs_arc_memory_throttle_disable module option

The way in which virtual box ab(uses) memory can throw off the
free memory calculation in arc_memory_throttle().  The result is
the txg_sync thread will effectively spin waiting for memory to
be released even though there's lots of memory on the system.

To handle this case I'm adding a zfs_arc_memory_throttle_disable
module option largely for virtual box users.  Setting this option
disables free memory checks which allows the txg_sync thread to
make progress.

By default this option is disabled to preserve the current
behavior.  However, because Linux supports direct memory reclaim
it's doubtful throttling due to perceived memory pressure is ever
a good idea.  We should enable this option by default once we've
done enough real world testing to convince ourselve there aren't
any unexpected side effects.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #938
ZFS 0.6.0-rc14
Add zpool-features(5) man page

The zpool-features(5) man page was accidentally omitted from the
build target when feature flags was merged.  As a result it doesn't
get installed as part of 'make install' so none of the packages
include this man page.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1262
Fix 1M references in zpool-features.5

The zpool-features(5) man page should reference the Linux zfs(8)
and zpool(8) man pages.  The 1M convention isn't used on Linux.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1184
Cast 'zfs bad bloc' to ULL for x86

Explicitly case this value to an unsigned long long for 32-bit
systems to inform the compiler that a long type should not be
used.  Otherwise we get the following compiler error:

  dmu_send.c:376: error: integer constant is too large for
  ‘long’ type

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Fix function relocations in libzpool

binutils 2.23.1 fails in situations that generate function relocations
on PowerPC and possibly other architectures. This causes linking of
libzpool to fail because it depends on libnvpair. We add a dependency on
libnvpair to lib/libzpool/Makefile.am to correct that.

Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1267
Remove unused machelf.h header

The machelf.h header is never included by anything in the zfs
build process.  It is all effectively dead code which can be
safely removed.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1265
Switch KM_SLEEP to KM_PUSHPAGE

Two more locations where KM_SLEEP was used in a call which must
use KM_PUSHPAGE were found while using the zpool upgrade command.
See commit b8d06fc for additional details.

Also make a small correction to the comment block above
dsl_dir_open_spa().

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1268
Update the zfs.8 "ZFS Volumes as Swap" section

As of 0.6.0-rc11 using ZFS volumes as Linux swap devices is
supported.  Swapping to files in ZFS filesystems is not.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1189
Add explicit MAXNAMELEN check

It turns out that the Linux VFS doesn't strictly handle all cases
where a component path name exceeds MAXNAMELEN.  It does however
appear to correctly handle MAXPATHLEN for us.

The right way to handle this appears to be to add an explicit
check to the zpl_lookup() function.  Several in-tree filesystems
handle this case the same way.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1279
Make spa.c assertions catch unsupported pre-feature flag pool versions

A couple of assertions in spa.c were designed to prevent the use of
invalid pool versions. They were written under the assumption
that all valid pools are less than SPA_VERSION. Since feature flags
jumped from 28 to 5000, any numbers in the range 28 to 5000
non-inclusive will fail to trigger them.  We switch to the new
SPA_VERSION_IS_SUPPORTED macro to correct this.

Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1282
Fix broken RPATH in spec file

Rather then setting _prefix=/ and having to override all the
default install locations.  It's cleaner, and more understandable,
to leave prefix=/usr and only override _sbindir and _libdir.  This
fixes three issues:

* The commands no longer get built with an incorrect rpath for
  the libraries.   This is good because fixing this sort of
  thing is required by the Fedora packaging guidelines.

    http://fedoraproject.org/wiki/Packaging:Guidelines#Beware_of_Rpath

* The various AUTHORS, COPYRIGHT, etc files are now correctly
  installed under /usr/share/doc instead of /share/doc.

* _libexecdir is now handled properly for each distribution.
  Fedora/RHEL=/usr/libexec, OpenSUSE/SLES=/usr/lib, Debian=/usr/lib/rpm

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1058