one thing to consider... I've noticed with voxel packets that if the pipe is very full (like at the limit of what the channel can handle) then often small UDP packets will get sent out of order before larger packets. So you might end up getting these small packets arriving ahead of the full size packets. you may need a sequence number scheme to address this.

With voxels we don't care if packets come out of order, because they're all self contained and time independent. But I suspect your audio packets must arrive in order for your mixing and playback code to behave well.