Illumos #3641 compressed block histograms with zdb

This patch is a zdb extension of the '-b' option, producing a histogram
of the physical compressed block sizes per DMU object type on disk. The
'-bbbb' option to zdb will uncover this new feature; here's an example
usage on a new pool and snippet of the output it generates:

    # zpool create tank /dev/vd{b,c,d}
    # dd bs=1k  if=/dev/urandom of=/tank/1kfile  count=1
    # dd bs=3k  if=/dev/urandom of=/tank/3kfile  count=1
    # dd bs=64k if=/dev/urandom of=/tank/64kfile count=1
    # zdb -bbbb tank
    ...
         3  68.0K   68.0K   68.0K   22.7K    1.00    34.26  ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      1 *
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
    ...

The blocks are also broken down by their indirection level. Expanding on
the above example:

    # zfs set recordsize=1k tank
    # dd bs=1k if=/dev/urandom of=/tank/2x1kfile count=2
    # zdb -bbbb tank
    ...
         1    16K      1K      2K      2K   16.00     1.02      L1 ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      1 *
         5  70.0K   70.0K   70.0K   14.0K    1.00    35.71      L0 ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      3 ***
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
         6  86.0K   71.0K   72.0K   12.0K    1.21    36.73  ZFS plain file
    psize (in 512-byte sectors): number of blocks
                              2:      4 ****
                              3:      0
                              4:      0
                              5:      0
                              6:      1 *
                              7:      0
    ...
                            127:      0
                            128:      1 *
    ...

There's now a single 1K L1 block which is the indirect block needed for
the '2x1kfile' file just created, as well as two more 1K L0 blocks from
the same file.

This can be used to get a distribution of the block sizes used within
the pool, on a per object type basis.

References:
  https://illumos.org/issues/3641
  https://github.com/illumos/illumos-gate/commit/490d05b

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Boris Protopopov <boris.protopopov@me.com>
Closes #2456
Convert zfs_mg_noalloc_threshold to a module parameter and document

The parameter was added as illumos issue 4081 which was committed to
zfsonlinux in ac72fac3eaa569902cad88053167f7d74e7fe7e4.  This patch
documents the parameter and allows for it to be set as a module parameter.

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2483
Illumos 4168, 4169, 4170: ztest, zdb and zhack fixes

4168 ztest assertion failure in dbuf_undirty
4169 verbatim import causes zdb to segfault
4170 zhack leaves pool in ACTIVE state
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
    https://www.illumos.org/issues/4168
    https://www.illumos.org/issues/4169
    https://www.illumos.org/issues/4170
    https://github.com/illumos/illumos-gate/commit/7fdd916

Porting notes:

Of particular interest when troubleshooting corrupted pools, the
commonly-used "zdb -e" operation may perform verbatim imports and
furthermore, it will soon have direct support for verbatim imports via
a new "-V" option.  The 4169 fix eliminates a common segfault case in
which spa_history_log_version() tries to access an un-opened dsl_pool_t.

Ported by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2451
Closes #2283
Closes #2467
zdb: Introduce -V for verbatim import

When given a pool name via -e, zdb would attempt an import. If it
failed, then it would attempt a verbatim import. This behavior is
not always desirable so a -V switch is added to zdb to control the
behavior. When specified, a verbatim import is done. Otherwise,
the behavior is as it was previously, except no verbatim import
is done on failure.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2372
Fix zil_commit() NULL dereference

Update the current code to ensure inodes are never dirtied if they are
part of a read-only file system or snapshot.  If they do somehow get
dirtied an attempt will make made to write them to disk.  In the case
of snapshots, which don't have a ZIL, this will result in a NULL
dereference in zil_commit().

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #2405
Include sys/taskq.h in linux/vfs_compat.h

We should have included sys/taskq.h directly because we use the taskq
code here, but we instead had files that included sys/taskq.h also
include sys/kmem.h, which happened to include sys/taskq.h. sys/kmem.h no
longer does this, so we must define the include as we should
have done in the first place.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Use Linux kzalloc()/kfree() for superblocks

We store the bdi in the superblock. This is accessed using virt_to_page
in bit_waitqueue(), which does not work on kernel virtual memory. We
switch to kzalloc()/kfree() to fix this.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Mark DMU transaction processing with PF_FSTRANS

It was thought that we could ensure forward progress by avoiding memory
allocations during transaction processing and avoiding KM_SLEEP in the
few instances in which we could not.

Recent tests involving L2ARC on systems without swap have shown that
this is not enough. VMWare's bencharmk on a system with 8GB of RAM and
200GB of L2ARC will reliably deadlock when atime is enabled, but will
run fine when atime is not enabled. This was initially thought to be a
mix of an extreme case of ARC contention and atime updates somehow
blocking on transaction group commit, but further examination showed
that only 137MB of RAM were in use by L2ARC's headers and additional
testing showed no deadlocks occurred with 24GB of RAM when atime was
enabled.

Subsequent review of the code revealed that the only change was that
direct reclaim no longer started transactions to perform atime updates.
The logical conclusion is that direct reclaim can occur in locations
that hold locks that block an open transaction through some unforeseen
dependency chain. This patch attempts to address that by modifying the
KM_SLEEP avoidance mechanism to be used in all transaction processing,
rather than just operations on zvols. We also take the opportunity to do
some cleanup by replacing PF_NOIO with PF_FSTRANS, which is the proper
Linux flag for this.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Add add_prop_list_default helper

Adding to a property list only if there is no existing value is used
twice. Once by zpool create -R and again by zpool import -R. Now that
zpool create -t and zpool import -t also need it, lets refactor it into
a helper function to make the code more readable.

Signed-off-by: Richard Yao <ryao@gentoo.org>
zpool import -t should not update cachefile

zpool import's -t parameter is intended for use with -R when operating
on pools that belong to other systems. Like -R, pools imported in this
way should not update the cachefile unless explicitly requested. The
initial implementation allowed the cachefile to be updated when -R was
not used. This went uncaught during testing because -R had implicitly
disabled use of the cachefile.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Implement -t option to zpool create for temporary pool names

Creating virtual machines that have their rootfs on ZFS on hosts that
have their rootfs on ZFS causes SPA namespace collisions when the
standard name rpool is used. The solution is either to give each guest
pool a name unique to the host, which is not always desireable, or boot
a VM environment containing an ISO image to install it, which is
cumbersome.

26b42f3f9d03f85cc7966dc2fe4dfe9216601b0e introduced `zpool import -t
...` to simplify situations where a host must access a guest's pool when
there is a SPA namespace conflict. We build upon that to introduce
`zpool import -t tname ...`. That allows us to create a pool whose
in-core name is tname, but whose on-disk name is the normal name
specified.

This simplifies the creation of machine images that use a rootfs on ZFS.
That benefits not only real world deployments, but also ZFSOnLinux
development by decreasing the time needed to perform rootfs on ZFS
experiments.

Signed-off-by: Richard Yao <ryao@gentoo.org>
ZFS_IOC_RECV should only call zvol_create_minors() on zvols

Signed-off-by: Richard Yao <ryao@gentoo.org>
zfs send -p send properties only for snapshots that are actually sent

... as opposed to sending properties of all snapshots of the relevant
filesystem.  The previous behavior results in properties being set on
all snapshots on the receiving side, which is quite slow.
More details are here:
  http://thread.gmane.org/gmane.comp.file-systems.openzfs.devel/346

Behavior of zfs send -R is not changed.
send_iterate_fs: orderly iterate snapshots

The previous commit depends on ordered iteration of the snapshots,
otherwise the code won't be able to correctly detect snapshots
that fall into the requested snapshot range.

This commit can be considered a partial cherry-pick of commit
freebsd/freebsd@4995789cde5 where ordered iteration was introduced
to fix a different problem.
We might want to pick up the complete fix eventually.
zp->z_sync_cnt should not use atomic operations on Linux

On most UNIX systems, the VFS will notify the filesystem about all
open() and close() operations on files and ZFS will take advantage of
this to track the number of synchronous opens in zp->z_sync_cnt. This
affects the zil_commit behavior. However, Linux only notifies us about
the last close on a file, which means that concurrent synchronous opens
will leave zp->z_sync_cnt set to a non-zero value on last close. This
means that all subsequent opens receive synchronous behavior even if
they had not asked for it.

We address this by modifying zfs_open()/zfs_close() to use
zp->z_sync_cnt on Linux as a flag variable that is set on synchronous
open and cleared on synchronous close.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Remove kmutex_t & kcondvar out of arc_buf_hdr_t

Under Linux these data types together consume around 100 bytes of space.
Having them in the arc_buf_hdr_t makes them nearly 50% larger than they
would be otherwise. When lock debugging is enabled it gets even larger.

This patch moves them to a single global array to be shared similar to
how the znode hold locks in zfs_sb work. We're trading some potential
for artificial lock contention for drastically smaller kernel memory
objects.
Mark SoftNAS SP2 Release

Signed-off-by: Richard Yao <ryao@gentoo.org>