illumos #3834

3834 incremental replication of 'holey' file systems is slow
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

References:
  https://www.illumos.org/issues/3834

Ported by: Tim Chase <tim@chase2k.com>
illumos #4047

4047 panic from dbuf_free_range() from dmu_free_object() while doing zfs receive
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@nexenta.com>

References:
  https://www.illumos.org/issues/4047

Ported by: Tim Chase <tim@chase2k.com>
Fix bug introduced by the fix for illumos #4047.

In the email list referenced below, Matthew Ahrens said:

    The problem occurs when an object is reused for a new file which has a
    non-power-of-2 blocksize.

    The problem occurs due to the following sequence of events:

    First we process the OBJECT record in the send stream.
        Call dmu_object_reclaim() with the new, non-power-of-2 blocksize
            Call dmu_free_long_range() to free the entire file
                New code (27504) in dmu_free_long_range_impl() does not pass
    length=-1 to dnode_free_range()
                    dnode_free_range() does not realize we are truncating, and thus
    does not decrease dn_maxblkid
                    Note, the fix for 26642 removes the dn_maxblkid-decreasing code
    from dnode_free_range() entirely.
            Call dnode_reallocate() to change the dnode configuration
                If debug build fail ASSERT(dn_maxblkid == 0)

    Assuming we didn't fail the assertion, next we will process the FREE record
    which frees the remainder of the file (e.g from 2.5K to the end)
        Eventually call dmu_tx_hold_free(off=2.5K, len=<big>)
            New code (24744) calls dmu_tx_count_write() on entire specified range
            Too much space has been scheduled to write, so fail with EFBIG

    The most expedient fix is to add code to zero out dn_maxblkid when doing
    dmu_object_reclaim().

    We can also make the code in dmu_tx_hold_free() more resilient by having it
    dmu_tx_count_write() only the first block of non-power-of-2 blocksize files
    (which only have one block).

    The problem can be reproduced by the following script:

    #!/bin/bash -x

    zpool create test c1t1d0

    # clean up from previous run
    zfs destroy -r test/fs
    zfs destroy -r test/recvd

    zfs create -o recordsize=8k test/fs

    dd if=/dev/zero of=/test/fs/big bs=1024k count=100

    # need to create enough files that ZFS will go back and reuse
    # object numbers
    for (( i=0; i<4000; i=i+1 )); do
            echo >/test/fs/empty-$i
    done

    zfs snapshot test/fs@big

    find /test/fs | xargs rm
    sync

    # replace "big" file with a file with uneven blocksize (1.5k)
    for (( i=0; i<100; i=i+1 )); do
            dd if=/dev/zero of=/test/fs/small-$i bs=1200 count=1
    done

    zfs snapshot test/fs@small

    zfs send test/fs@big | zfs recv test/recvd

    # this receive will fail
    zfs send -i @big test/fs@small | zfs recv test/recvd

References:
  http://www.listbox.com/member/archive/182191/2013/09/sort/time_rev/page/5/entry/3:140/20130909182626:D79EC5B8-199E-11E3-8BF5-CB08091A731B/