Found the source of my corruption.  The && between the queue depth check needed to be an ||.  Also, to prevent queues from growing, always write out enough samples to bring it back down to the queue depth specified.

Also some cleanup.