zfsonlinux/zfs — pull request #2386 by b333z (2014-06-11 01:21:04+00:00)
Title: Problem with vdev_asize when last mirror child promoted to top level vde...

If the smaller of 2 different sized child vdev's of a mirrored vdev is
detached, and the pool has the autoexpand property set to off, as the
remaining larger vdev is promoted to a top level vdev it fails to retain
the asize of the original top level mirror vdev and therefore partially
autoexpands.

This partially autoexpanded state leaves the new vdev too large to
re-mirror by adding the smaller vdev back in, and the pool fails to
utilize the space until next imported.

If the autoexpand property is set to on, the child vdev grows
in size after it has been promoted to a top level vdev as expected.

This commit causes the remaining child mirror to retain the asize of its
old parent mirror vdev if the autoexpand property is set to off,
this allows the smaller vdev to be re-added if required the vdev
can then be told to expand if required by the usual using zpool online -e.

This was found investigating solution to issue #1208 that @DeHackEd identified.

Has survived a few hours of ztest, but I'm unfamiliar with much...
Still unsure if anything needs to happen in regards to ```vdev_min_asize``` and ```vdev_max_asize``` ?

Found some great George info about expanding vdevs here: http://blog.delphix.com/gwilson/2013/03/08/lun-expansion/

Using this to test ( https://gist.github.com/b333z/c3e6da6001fccf9d39d1/raw/54bd2306b30e60a3c1e608705c7fcc017dd44720/test-1208 )
```
dd if=/dev/zero of=/tmp/disk0.img bs=1024k count=100
dd if=/dev/zero of=/tmp/disk1.img bs=1024k count=200

zpool create -o autoexpand=off -f testpool mirror /tmp/disk0.img /tmp/disk1.img
zpool list -v testpool
zpool iostat -v testpool

zpool detach testpool /tmp/disk0.img
zpool get size,autoexpand,expandsize testpool
zpool list -v testpool
zpool iostat -v testpool
zdb -C testpool

zpool attach testpool /tmp/disk1.img /tmp/disk0.img

zpool export testpool

zpool import -d /tmp testpool
zpool get size,autoexpand,expandsize testpool
zpool list -v testpool
zpool iostat -v testpool

rm /tmp/disk0.img
rm /tmp/disk1.img
```

In results below notice how capacity/size info from zpool differ from the asize in zdb after the smaller mirror child disk0.img is detached, and how the zpool matches up with zdb showing the extra capacity once re-imported.

Running the above against unpatched Linux - ZOL - 3.6.9 - master (21b446a79) - https://gist.github.com/b333z/c3e6da6001fccf9d39d1/raw/9c2ef889bf3f9b757da7d3e6873aac8955cce97d/test-1208.linux.txt
( links to results on same test for illumos and freebsd below )
```
# dd if=/dev/zero of=/tmp/disk0.img bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0831649 s, 1.3 GB/s

# dd if=/dev/zero of=/tmp/disk1.img bs=1024k count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.161863 s, 1.3 GB/s

# zpool create -o autoexpand=off -f testpool mirror /tmp/disk0.img /tmp/disk1.img

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 95.5M 136K 95.4M 0% 1.00x ONLINE -
mirror 95.5M 136K 95.4M -
/tmp/disk0.img - - - -
/tmp/disk1.img - - - -

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
------------------ ----- ----- ----- ----- ----- -----
testpool 136K 95.4M 2 36 2.58K 33.1K
mirror 136K 95.4M 2 36 2.58K 33.1K
/tmp/disk0.img - - 3 19 6.46K 271K
/tmp/disk1.img - - 0 20 3.88K 271K
------------------ ----- ----- ----- ----- ----- -----

# zpool detach testpool /tmp/disk0.img

# zpool get size,autoexpand,expandsize testpool
NAME PROPERTY VALUE SOURCE
testpool size 95.5M -
testpool autoexpand off default
testpool expandsize 0 -

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 95.5M 172K 95.3M 0% 1.00x ONLINE -
/tmp/disk1.img 95.5M 172K 95.3M -

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
---------------- ----- ----- ----- ----- ----- -----
testpool 172K 95.3M 0 30 3.84K 356K
/tmp/disk1.img 172K 95.3M 0 30 3.84K 356K
---------------- ----- ----- ----- ----- ----- -----

# zdb -C testpool

MOS Configuration:
version: 5000
name: 'testpool'
state: 0
txg: 6
pool_guid: 15451571592932472266
errata: 0
hostid: 811808865
hostname: 'b1'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 15451571592932472266
create_txg: 4
children[0]:
type: 'file'
id: 0
guid: 11218347111503734635
path: '/tmp/disk1.img'
metaslab_array: 34
metaslab_shift: 19
ashift: 9
asize: 204996608
is_log: 0
create_txg: 4
features_for_read:

# zpool attach testpool /tmp/disk1.img /tmp/disk0.img
cannot attach /tmp/disk0.img to /tmp/disk1.img: device is too small

# zpool export testpool

# zpool import -d /tmp testpool

# zpool get size,autoexpand,expandsize testpool
NAME PROPERTY VALUE SOURCE
testpool size 196M -
testpool autoexpand off default
testpool expandsize 0 -

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 196M 224K 195M 0% 1.00x ONLINE -
/tmp/disk1.img 196M 224K 195M -

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
---------------- ----- ----- ----- ----- ----- -----
testpool 224K 195M 33 74 337K 434K
/tmp/disk1.img 224K 195M 33 74 337K 434K
---------------- ----- ----- ----- ----- ----- -----

# rm /tmp/disk0.img

# rm /tmp/disk1.img
```
Same test against other platform unpatched can be seen here:

FreeBSD 10.0-RC5 - https://gist.github.com/b333z/c3e6da6001fccf9d39d1/raw/e2784ce9022d3bc0d5283d6b204ec1c3ad0b28c0/test-1208.freebsd.txt

Omnios - 151010j - https://gist.github.com/b333z/c3e6da6001fccf9d39d1/raw/9913f36d1559cabf09c8752c4335f90f650b4b3d/test-1208.omnios.txt

Here is the same test against linux after applying the patch:
```
# dd if=/dev/zero of=/tmp/disk0.img bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.220252 s, 476 MB/s

# dd if=/dev/zero of=/tmp/disk1.img bs=1024k count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2.62054 s, 80.0 MB/s

# zpool create -o autoexpand=off -f testpool mirror /tmp/disk0.img /tmp/disk1.img

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 95.5M 94K 95.4M 0% 1.00x ONLINE -
mirror 95.5M 94K 95.4M -
/tmp/disk0.img - - - -
/tmp/disk1.img - - - -

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
------------------ ----- ----- ----- ----- ----- -----
testpool 94K 95.4M 6 74 6.09K 61.6K
mirror 94K 95.4M 6 74 6.09K 61.6K
/tmp/disk0.img - - 2 42 13.1K 750K
/tmp/disk1.img - - 6 45 15.5K 750K
------------------ ----- ----- ----- ----- ----- -----

# zpool detach testpool /tmp/disk0.img

# zpool get size,autoexpand,expandsize testpool
NAME PROPERTY VALUE SOURCE
testpool size 95.5M -
testpool autoexpand off default
testpool expandsize 100M -

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 95.5M 132K 95.4M 0% 1.00x ONLINE -
/tmp/disk1.img 95.5M 132K 95.4M 100M

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
---------------- ----- ----- ----- ----- ----- -----
testpool 132K 95.4M 5 61 14.7K 958K
/tmp/disk1.img 132K 95.4M 5 61 14.7K 958K
---------------- ----- ----- ----- ----- ----- -----

# zdb -C testpool

MOS Configuration:
version: 5000
name: 'testpool'
state: 0
txg: 5
pool_guid: 3842628794123118842
errata: 0
hostid: 811808865
hostname: 'b15'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 3842628794123118842
create_txg: 4
children[0]:
type: 'file'
id: 0
guid: 14782448015729639620
path: '/tmp/disk1.img'
metaslab_array: 34
metaslab_shift: 19
ashift: 9
asize: 100139008
is_log: 0
create_txg: 4
features_for_read:

# zpool attach testpool /tmp/disk1.img /tmp/disk0.img

# zpool export testpool

# zpool import -d /tmp testpool

# zpool get size,autoexpand,expandsize testpool
NAME PROPERTY VALUE SOURCE
testpool size 95.5M -
testpool autoexpand off default
testpool expandsize 0 -

# zpool list -v testpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
testpool 95.5M 232K 95.3M 0% 1.00x ONLINE -
mirror 95.5M 232K 95.3M -
/tmp/disk1.img - - - -
/tmp/disk0.img - - - -

# zpool iostat -v testpool
capacity operations bandwidth
pool alloc free read write read write
------------------ ----- ----- ----- ----- ----- -----
testpool 232K 95.3M 105 576 114K 668K
mirror 232K 95.3M 105 576 114K 668K
/tmp/disk1.img - - 132 460 1.79M 2.31M
/tmp/disk0.img - - 128 456 1014K 2.31M
------------------ ----- ----- ----- ----- ----- -----

# rm /tmp/disk0.img

# rm /tmp/disk1.img
```