Fix rlock to properly return UINTMAX to trigger blocksize growth in ^2. This
fixes the "accessing past object" soft panic, due to blocksize growing from
zfs_extend().

Fix up more VNOPs to return ENOTSUP.