Handle different number of channels. Order samples according to their stream time.