Debugging for zio_wait() hangs

From the information gathered about issue #2523 it's clear that
somehow the spin lock is being damaged.  This could occur if the
spin lock is reinitialized, the memory is accidentally overwritten,
or freed back to the cache to soon.  To determine exactly what is
happening this patch adds a couple new sanity tests.

* A zio->io_magic field is added before the io_lock.  This field
  is designed to act as a red zone allow us to detect if the zio
  has been written.

* The zio->io_magic field is also used to detect if somehow the
  constructor or destructor is running multiple for the object.
  This would effectively cause the spin lock to be reinitialized.

* The destructor has been updated to poison the entire structure.
  This should cause us to quickly detect any use-after-free bugs.

Once the root cause of this issue can be determined this patch
should be reverted.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2523
Add zio_wait() executor/waiter locking

There exists a plausible cache concurrency issue with zio_wait().
This might occur because the zio->io_waiter to not assigned under
a lock in zio_wait(), is not checked under a lock in zio_done(),
and the zio may be dispatched to another thread for handling.
That said, none of the actual crash dumps I've looked at show
that this has ever occurred.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2523