Before revision:  Thus, the model can capture the token relations by self-attention on the aggregated hidden representations from the whole speechsignal rather than autoregressive modeling on tokens . Without explicitly autoregressive language modeling, this model predicts all tokens in the sequence in parallel so that the inference is efficient. Moreover, we propose a cross-modal transfer learning method to use a text-modal language model to improve the performanceof speech-modal LASO by aligning token semantics.
After revision:  Thus, the model can capture the token relations by self-attention on the aggregated hidden representations from the speech. The PDS uses positional encodings corresponding to tokens to convert the acoustic representations into token-level representations. The decoder further captures token-level relationships with the self-attention mechanism. At last, the probability distribution on the vocabulary is computed for each token position. Therefore, speech recognition is re-formulated as a position-wise classification problem. Further, we propose a cross-modal transfer learning method to use a text-modal language model to improve the performanceof speech-modal LASO by aligning token semantics.