audio: Remove unnecessary reallocation of the channel matrix.