Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers
Authors: Adam Stooke
Rohit Prabhavalkar
Khe Chai Sim
Pedro J Moreno Mengibar
Venue: NeurIPS 2024 spotlight (2024)
Keywords: ASR
Speech
Transducers
Transformers
Alignment
Primary area: speech_and_audio
TL;DR: We find that transformer encoders can perform audio-to-text alignment internally during a forward pass, a new phenomenon that allows greatly simplified ASR models.

Abstract: Modern systems for automatic speech recognition, including the RNN-Transducer and Attention-based Encoder-Decoder (AED), are designed so that the encoder is not required to alter the time-position of information from the audio sequence into the embedding; alignment to the final text output is processed during decoding. We discover that the transformer-based encoder adopted in recent years is actually capable of performing the alignment internally during the forward pass, prior to decoding. This new phenomenon enables a simpler and more efficient model, the ''Aligner-Encoder''. To train it, we discard the dynamic programming of RNN-T in favor of the frame-wise cross-entropy loss of AED, while the decoder employs the lighter text-only recurrence of RNN-T without learned cross-attention---it simply scans embedding frames in order from the beginning, producing one token each until predicting the end-of-message. We conduct experiments demonstrating performance remarkably close to the state of the art, including a special inference configuration enabling long-form recognition. In a representative comparison, we measure the total inference time for our model to be 2x faster than RNN-T and 16x faster than AED.  Lastly, we find that the audio-text alignment is clearly visible in the self-attention weights of a certain layer, which could be said to perform ''self-transduction''.

Aligner-Encoders:
Self-Attention Transformers Can Be Self-Transducers
Adam Stooke Google, USA
Rohit Prabhavalkar Google, USA
Khe Chai Sim Google, USA
Pedro Moreno Mengibar"
astooke@google.com com
Abstract
Modern systems for automatic speech recognition, including the RNN-Transducer and Attention-based Encoder-Decoder (AED), are designed so that the encoder is not required alter the time-position of information from the audio sequence into the embedding; alignment to the final text output is processed during decoding. We discover that the transformer-basee encoder adopted recent years is actually capable of performing the alignment internally during the forward pass, prior to decoding. This new phenomenon enables a simpler and more efficient model, the Alligner-Encoder" To train it, we discard the dynamic programming of RNN-T in favor of the frame-wise cross-entropy loss of AED, while the decoder employs the lighter text-only recurrence of RNN-T without learned cross-attention simply seans embedding frames in order from the beginning, producing one token each until predicting the and-of-message. We conduct experiments demonstrating performance remarkably close to the state of the art, including special inference configuration enabling long-form recognition. In representative comparison, measure the total inference time for our model to be 2x faster than RNN-T and 16x faster than AED. Lastly, find that the audio-text alignment is clearly visible in the self-attention weights of a certain layer, which could be said to perform "self tran--rraadut.on...
Introduction
The task of sequence transduction requires mapping from input sequence an output sequence. It appears in several widespread applications including machine translation and automatic speech recognition (ASR). In ASR, which is the focus of this paper, the two sequences lie in completely different modalities, and the audio input representation is typically much longer than the output text. Thus, in addition identifying and converting speech sounds, the model must find an "alignment" that moves information from wherever it appears in the input sequence to wherever it belongs in the output. Among other issues, the many possible variations in the pace of pronunciation make transduction a challenging problem.
More than decade ago. and almost decade apart, two powerful algorithms relying on dynamic programming were developed to perform the sequence transduction task with recurrent neural net- works (RNNs). The motivation behind these algorithms was that RNNs require training every output frame in the sequence with one output per input frame, they needed to be trained with alignments already prepared. The new algorithms allowed training without prepared alignments by computing the probability of the label marginalized over all possible alignments as the optimization objective. The differences in sequence length are accommodated by learning to output "blank" symbols in between true labels and post-processing them out at inference time. The first algorithm. Connectionist Temporal Classification (CTC) I. models the output frames as conditionally independent, and this
Work performed while Google, USA.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

limitation was overcome in the follow-on work RNN-Transducer (RNN-T) auto-regressive decoding achieve better performance.
2 which introduced
A third algorithm appeared shortly after, called attention-based encoder-decoder (AED). It uses a learned mechanism account for alignments 8 the encoder is followed by recurrent decoder that cross-attends at every step between its RNN state and the entire encoder embedding sequence to produce its next state. The cross-entropy loss is applied straightforwardly between the plain label sequence and the leading output frames, with no blank symbol and no dynamic programming required. The resulting model produces its output reggessgrss until an end-of-sentence (<E0S>) token halts decoding. AED models, together with CTC and RNN-T, have propelled end-to-end deep neural networks to become the best performing ASR systems across academia and industry (e.g., see surveys 8 D.
Despite their great successes, each of these algorithms has its downsides. Implementing the probability summations for CTC and especially RNN-T require non-trivial effort. naive formulation of the principle is intractable to compute, requiring a dynamic programming approach, which in turn has been the subject of optimization efforts by expert practitioners [9H12. Beyond that, RNN-T training requires computing potentially unwieldy tensor quantities to do with cross-pairing every frame of input with every token of output. Both models suffer inefficiencies in decoding. RNN-T processes all frames recurrently while producing typically much shorter output sequence. AED requires computing over the entire encoder embedding sequence at every decoding step, which leads to slow operation from the high compute load within the auto-regressive loop.
In an effort to resolve these difficulties, we ask the question: with the advent of transformer-based the ASR encoder itself learn to perform the alignment? The answer found is: yes, it can. Our main contribution is to show that it is now possible to train neural network speech recognizers with light-weight decoders in the style of RNN-T, but with the simple frame-wise cross-entropy loss of AED. The resulting networks provide accuracy remarkably close to state-of- the-art models while being more efficient training and decoding than any previous model. No explicit dynamic programming is employed: instead, the encoder learns to perform the alignment internally during the forward pass. The encoded embedding frames are decoded consecutively from the beginning. one at a time, in conjunction with small recurrent network that only reads in the previous label, producing exactly one label per frame until emitting <EOS>.
encoders [3
Having originated in the era of long short-term memory (LSTM) 21 22 encoders, the preceeding algorithms all allow the encoder to process information in-place in the time dimension, resulting in embeddings with relevant information in roughly the same position in the sequence as it appeared in the input. For several years now, however, ASR systems have benefited from the adoption of more powerful transformer encoders, but without changing their role. Our Aligner-Encoded in that, simultaneous encoding, they perform the additional task of moving the relevant information to the beginning of the embedding sequence into a tubel-aligned position. As it is performed solely through the self-attention mechanism within the forward pass of the encoder, one might call this a The previous style of information flow is contrasted with ours in Figure [ A main benefit of our model utilizing an Aligner- Encoder that it dramatically simpler to conceptualize, implement, and use.
are different
This paper is organized as follows. First, we describe Aligner-Enncodr models and relate them to RNN-T and AED. Next, we relate other works which have aimed at improving RNN-T modeling effectiveness or efficiency. Then we report experiments demonstrating the accuracy of Aligner models while also finding limitations for generalizing to long test utterances. In response, we introduce techniques which can be employed inference-time to provide good long-form performance without perform
any additional training. In further experiments, we analyze how and when the alignment and make the surprising discovery that it can happen primarily within a single self- attention layer. Lastly, since ASR alignments are monotonic, we demonstrate that Aligners can at least readily handle reverse alignments, making our model promising for future work in non- non-monotonic applications such as machine- translationn or speech- -rench-translation.
2

---

Audio-Aligned Embedding (to Decoder)
Text-Aligned Embedding (to Decoder)
Traditional Encoder (CTC/ CACC/RNNT//// AED)
Aligner- Encoeer
Audio Frames (Input)
Audio Frames (Input)
Figure 1: Information flow through an Aligner-Encoder Encoder versus traditional audio encoders.
2 Model
2.1 Aligner-Encoder Model
The Aligner-Encoder model combines the best elements of RNN-T and AED systems into more compact form. By requiring the encoder itself to text-align its embedding, we avoid using I) dynamic programming to sum probabilities in the loss and 2) full-sequence cross-attention in the decoder (in all models we refer to everything inside the auto-regressive portion as the "decoder"). It may be simplest first lay out our model, which formulated using the same components as RNN-T, and afterwards draw contrasting points with the heritage.
We begin with an input sequence (y1, 122- ..., yo of length U < T. An encoder network, Sencs processes the input sequence into an acoustic embedding sequence, can depend on every frame of input, and typically T" < 'T with subsampling. A prediction network, fpred. processes text labels with forward recurrence only to produce text embedding sequence, (g),9) 92, ..., 9. The acoustic and text embeddings are fed into a j.joint network, fjoint, to produce the final prediction vector of dimension V according the vocabulary, for each frame, with softmax probability normalization. The difference in our model is that we enforce the alignment at the encoder outpur by restricting the model use the acoustic and text embedding frames in one-to-one fashion (Equation (3)). The entire model is written by its recurrence relations as:
(I1, I2, ..., IT) of length T, and output sequence y
(h1, .h....., hr.,), where each frame of the embedding sequence
h fenc (x)
(1)
9i fpea(gi-10 1, 11-1), U
P(y x, V = faoim (h:.g:), i U
(3)
The encoder and the which includes both the prediction and joint networks are parameterized and learned together an end-to-end manner, with total parameters We maximize the log probabilities of the correct labels, resulting in the familiar cross-entropy loss:
Clligner - log P(y: ye: 0)
Σ
The loss only applies to encoder frames within the length of the label, T" S U; all remaining frames (T' > U) ignored. Hence the encoder must also learn be aligner.
We seed the prediction network with a ofart-of-sentence token, <SOS>, at the first step. Yo. and empty state ho 0. As is often the case with AED and other uses of cross-entropy loss, we find label smoothing to be a beneficial regularizer and always use it (2) (weight 0.1). During inference the label length is unknown, the model must predict it, in AED. We train the model to always predict the ofd-te--eetence token, <EOS>, as the final token of every training example") so decoding proceeds tokenbby-toker until <EOS> is predicted.
We include the <BOS> in the label count U, without loss of generality.

---

It is also possible to formulate non autoregreesivve (non-AR) Aligner, which applies decoder, find- to the embedding frames independently, as CTC:
P(yl(x)= find(h)
(5)
and it can be trained under the same loss. During inference, all tokens are predicted independently, and any tokens after the earliest <EOS> in the sequence are discarded. We include experiment with this formulation; however, we found its performance to be significantly worse than CTC in all but the shortest-utterance datasets.
2.2 Advantages over RNN-T & AED
Given that previous models were developed to overcome contemporary neural encoders' inability to align, the new model naturally brings simplifications. RNN-T trains by explicitly marginalizing over all possible alignments in the loss. This can be visualized in a label-frame decoding lattice (see Figure 1 in (2), where each node represents an (encoder-frame, text-frame) pair, and every pairing is valid state to be considered, resulting in a U x T rectangular grid (abbreviating T" as T). Beyond requiring sophisticated dynamic programming implementation calculate tractably (involving forward and backward probabilities scanned across the lattice), the marginalization still leads to computing all logits of the lattice, potentially memory-intensive step (it is computing Equation (3) over all pairs of indices as fjoin (,, P3 )). The Aligner loss can be viewed as prescribing only the main diagonal of the lattice be valid, reducing the realized logits the bare minimum U V.
U x T
Savings come during inference, as well. The decoder in AED operates with OUU x T) complexity, since cross-attends to the entire encoder embedding sequence at every step, for U steps. Furthermore the constant factor is often relatively large in practice since a more powerful decoder network is needed for computing the attention (the recurrence relation is 9; fEE (h, 9:-1: 1/-1). compare at our Equation (2) which lacks h or even h2). The RNN-T decoder has complexity O(T + U), since it must emit a blank at every frame to advance the end of the lattice in addition to the steps emitting a label. In contrast, by preparing the alignment within the encoder, our model reduces decoder complexity to OU)) the most efficient possible for auto-regressive token prediction. In practice our constant factor will also be small like RNN-T.
One final savings relative to RNN-T comes when using beam search during inference, which can significantly improve overall accuracy. The emissions blank tokens in RNN-T means that the same text hypothesis
be represented by different paths through the decoding lattice. Thus proper search requires a routine to combine the probabilities from equivalent lattice paths at every step. Known as "path merging", it can actually be an expensive operation, leading others to sometimes approximate the search without it, despite the potential loss of beam diversity (e.g., see discussion in (26)). Since our model, like AED, does not emit any blank tokens, path merging does not apply.
3 Related Works
Before end-to-end ASR, the previous generation of "hybrid" ASR systems 27 relied on a separate base system to provide frame-level alignments which could then be used to train frame-level neural network acoustic models. Most modern ASR systems use an end-to-end framework; the most prominent modeling techniques are CTC D. RNN-T 22, and AED [5).
A number of works have aimed at modifying the base RNN-T model in order to gain efficiency or efficacy. In Monotonic RNN-T 28 the loss and decoder are modified step forward every frame, preventing multiple token emissions per label and speeding up the decoder complexity to O(T). More recently. [12] proposed a method to reduce memory usage of the RNN-T loss, especially for cases with large vocabulary size such as Chinese character-based by using a lincarized joint network first to identify a relevant sub-region of the lattice likely to contain the true alignment. They only compute the full joint network on that sub-region and still train successfully. Another issue with RNN-T systems arises when trying to treat the prediction network as a language model, which can be improved through text-only training, because its language modeling function is polluted by having to predict blanks. To ameliorate this, multiple works [29-3]] have introduced a separate network component for predicting blanks, and re-formulated the overall prediction model accordingly, resulting in much greater receptiveness to language training- One aim of our design was to avoid the use of blank tokens entirely, although in the present work we do not pursue additional text training.

---

Monotonic alignments have also been developed for attention iitention-based systems [32. requiring newly devised loss functions to enforce monotonicity differentiably and encourage discreteness. Integrate- and-fire systems are another approach proposed to limit the context window needed for cross-attention to the encoder embedding. by introducing a soft monotonic alignment mechanism R5). It steps forwards through the encoder frames one at a time, accumulating weighted information from the embedding until an activation threshold is reached, upon which a token is emitted. CTC and ditentiio-bbsed systems have previously been combined [36 37 resulting in improved learning and decoding, and it is not uncommon to train AED models with CTC auxiliary loss. We did not pursue the analogous combination of non-AR and AR decoders for Aligners, although it would be simple to implement. Another interesting, concurrent line of work is the Bayes- CTC [38] and Trans-Trans [39] models, which add modulating factor to the respective losses in order to encourage earlier token emission in decoding; however, only our simpler model achieves full text alignment with no blank embedding frames between tokens.
common way to improve the efficiency of ASR systems is to downsample the time dimension within the encoder, known as frame reduction (i.e., T" < T,; [40). In addition to reducing encoder complexity, which scales as O(T²) with self-attention, frame reduction yields decoder efficiencies. It was necessary in the original Listen, Attend Spell (LAS) model [5] to enable learnability for the cross-attention by presenting a manageable number of frames to the decoder. In RNN-T it reduces the number of decoder steps, which scale with T, and can in some circumstances be carried out to an extreme, packing multiple labels per frame [4T] In the few cases we tried, we found Aligners to have similar robustness downsampling as RNN-T, but we did not study this in- Aligners are like CTC in that they cannot downsample the encoder to fewer frames than the length of the text sequence, although perhaps they could be trained to decode multiple tokens per frame.
4 Experiments
We conducted a range of experiments to explore the performance of Aligners and compare them against previous models on an equal footing. After describing the common configurations and datasets used, we present three different sets of experiments. The first explores the performance of basic Aligner models, the second studies techniques to employ at test-time to attain long-form recognition, and the final set examines the alignment process occurring within the encoders.
4.1 Datasets
We experiment on three U.S. English datasets with very different characteristics. The first is LibriSpeech-960 hour (LS) 42). The second is Voice Search (VS) dataset comprised of real traffic, totalling over 100M utterances and nearly 500k hours of speech, with an average duration of 3.4 seconds; utterances are anonymized before processing. The majority of the utterances pseudo-labeled by a teacher model 43,. and a small portion are human transcribed. Only 5% of the queries greater than 7.6 seconds long. The test sets for VS include main set which covers many common types of utterances, and several rare-word sets generated using TTS from specific domains-maps. maps, queries, and news. Lastly, we include long-form dataset drawn from random YouTube videos (YT), The training set is comprised of utterances with pscudo-labels generated by previous ASR system [43), cleaned to ensure no speaker overlap occurs, and ranging from 5-15 seconds each. The total training set size is roughly 670K hours. The test set includes 30 hours of long-form audio, at an average of 8 minutes per example, spanning range of topic sources including lifestyle, entertainment, sports. gaming, and society. In all datasets, the audio input is represented using log mel features with 32ms window size 10ms stride.
4.2 Neural Networks Specifications
We used the same neural encoder architecture for every algorithm, with settings adjusted for each dataset as listed in Table [] Our networks begin with learnable 2-D convolutional subsampling layers, with kernel size 3 and stride 2, 128 features in the first layer and 32 thereafter. We employ state-of-the-art Conformer encoders (20). which include in each layer: a feed-forward unit, multi- headed self-attention layer, a 1-D convolution layer for localized processing, followed by an outgoing feed-forward unit and finally residual connection. The number of layers and model dimension vary by datasct, as shown in the table. We did not find RNN-T nor Aligner experiments to be highly sensitive
5

---

to decoder dimensions they can operate with suprisingly small prediction networks networks-although results sometimes did improve slightly with larger prediction networks when also using a wider beam search.
All our models use a word-piece tokenizer 44 as is common in state-of-the-art systems. Word pieces may be as small as single letter, or could be an en- tire word, such "wednesday". Unlike phonemes, word pieces of a given vocabulary exhibit a wide range in duration, not to mention complexity, of audio content associated with them. Although this likely makes the alignment problem more difficult, word piece vocabularies are very effective because they cover the text distribution efficiently. For la- bel smoothing, we estimate the word-piece prior on-the-fly using batch-wide label counts.
Table 1: Settings used with each dataset: Lib-
riSpeech, Voice Search, and YouTube.
Setting
VS
YT
Log-No -Mel Features
128
128
2-D Conv Layers
Enc Dimension Enc # Layers
512
768
1024
17
24
24
Enc # Params
100M
300M
600M
LSTM Size Vocab Size
1x640
1x256
1x1024
1024
4096
4096
To enhance the beam search, we apply label smoothing debiasing 455 which essentially removes
low-probability tokens from the posterior and then re-normalizes it. Similar [5). we find that relatively large debiasing parameter of 2 (so the threshold becomes 2/V) is helpful, and we use with modest beam size of 6 in our experiments for small but consistent improvement in accuracy.
4.3 Base Model Results
Table 2 shows results in Voice Search, includ- ing the rare-word test sets, and Aligners perform comparably to RNN-T in all cases. We also com- pare CTC and non-AR Aligners, which perform nearly as well in Voice Search. We found that increasing the vocabulary size 32K improved the performance of the non-AR Aligner, yet it still suffers from many deletions on the NEWS rare-word set, which contains longer utterances. Our non-AR model performed relatively poorly on LibriSpeech and YouTube. For sequences beyond very short length, an auto-regressive decoder, however small, essential for producing a high quality final output from the encoder embedding.
Table 2: WER (%) on the Voice Search test sets. VS: Main Test, and rore-words RM: Maps, RN: News, RQ: Search Queries
VS
RM
RN
RQ
RNN-T
3.6
12.6
14.6
20.5
ALIGNER
3.7
12.5
13.1
20.4
CTC NON-AR ALIGNER
4.3
14.3
17.8
23.8
4.5
15.3
27.3
23.5
In LibriSpeech, we report results comparing CTC, RNN-T, ABD, and Aligners in Table 3 All models used the same 17-layer conformer encoder architecture and dimensions, except for differences in relative position attention, described in the following paragraph. Performance our Aligner models was remarkably close to RNN-T, matched or beat AED, and clearly beat CTC. For each model, we report the best score from small number of runs and checkpoints. Full training settings are provided in the appendix, along with a brief commentary on the relative performance between RNN-T and AED.
The distribution of training utterance lengths in LibriSpeech has sharp drop-off after 17 sec- onds, and we observed both AED and Aligners losing performance when generalizing to longer test utterances, which run as long as 36 seconds. (See Tables 7 in the appendix for a breakdown by test utterance duration.) To improve perfor- mance of these models, we randomly concate- nated a small subset of utterances in each train- ing batch (e.g., 15%), creating some examples up to 36 seconds. Thus, with adequate training. Aligners were able to colf-transduce sequences of up to 900 frames. Learned relative attention encod- ing [46] trained slowly, so AED, CTC, and our model were trained with the faster Rotary Position Embedding [7 (RNN-T achieved slightly better test scores with learned relative encoding, reported in the table). the following section, we discuss ways to use our model to perform long-form recognition to arbitrary lengths beyond what is seen in training.
Table 3: WER (%) on LibriSpeech.
TEST-CLEAN
TEST-OTHER
CTC
2.6
2.8
6.4
RNN-T
2.1
2.1
4.6
AED
2.2
2.4
5.5
ALIGNER
2.2
2.3
5.1

---

4.4 Long-Form Recognition
is not always practical to train with examples as long as the desired usage, and for any given Digner-Encoder architecture there is likely some maximum alignable length. So it may be necessary to perform recognition on utterances longer than a trained model's capability. Our baseline method for comparing long-form recognition is "blind segmenting": 1) cut the audio into fixed-length, non- overlapping segments, 2) transcribe them separately, and 3) concatenate the resulting text segments. Perhaps the main shortcoming of blind segmenting is that recognition at the boundaries is prone to error, where the audio might cut in the middle of word. One applicable solution is to use overlapping segments with an appropriate routine stitch the hypothesis together in post-processing [48] 49). Here, we instead describe how to improve continuity using the model itself, using only inference configurations which require no further training.
In our approach, cutting and re-concatenating the sequence happens within the model-called "chunk- ing" to distinguish from the baseline. We preserve some continuity by cutting the audio only after the 2-D convolutional feature layers. The chunks are processed independently in parallel through the conformer, after which they re-concatenated before decoding. The decoder processes the full-length embedding sequence into the full transcription hypothesis, using awareness of the chunk boundaries. Within each chunk, the decoder ignores frames after the first <EOS> emission, and resumes decoding at the beginning of the next chunk. Ideally, the decoder's prediction network will carry its state forward across the chunk boundaries to evenly incorporate all tokens.
When we experimented with chunk sizes close to the training length, however, it led to increased deletions near the chunk ends. It was actually better to reset the prediction network state. This is understandable as the decoder was only ever trained to begin each utterance with a blank state, and it may be helping middle approach: at the chunk boundary we reset the prediction network state but then "prime" by re-processing the last several tokens through it. We found state-priming to reduce the number of errors at the boundary without raising later deletions.
count frames until <EOS>, The best performance, however, came from
For testing, we turn to the YouTube domain, where the training utterances ranged from 5-15 seconds long but test utterances spanned several minutes. Table compares results from RNN-T and the Aligner-Encodee Encoder. They achieved equal WER (7.6%) with a 15-second blind segmenter, due equivalent errors the boundaries. In the unsegmented arrangement, the RNN-T, with local attention of width 256, generalized well to the full utterance length; its WER improved to 6.8%% Inside the Aligner, we applied chunking period of 14 seconds (176 embedding frames). We found it best to reset the prediction network at the chunk boundaries and prime it with 10 tokens of history, which achieved 7.3% WER (7.5% without state-priming). While falling short of the RNN-T in this case, the Aligner did improve performance by smoothing the chunk boundaries, even without having been trained do so. This result demonstrates our model's 's capability for long- form recognition, and could possibly be improved, such as by introducing additional training for this mode.
Table 4: WER (%) on YouTube long-form test set.
15s SEGMENTED
UNSEGMENTED
RNN-T ALIGNER
7.6
6.8
7.6
7.3
4.5
Alignments
4.5.1
Self-Attention Probabilities
To study how the encoder predicts the alignment, we examine the self-attention probabilities it computes during the forward pass. Figure shows self-attention probabilities (every row sums to one) from the same head at different layers within one of our 17-layer encoders trained on LibriSpeech. striking pattern emerges. One might expect the alignment to happen gradually in a network replete with residual connections, and indeed the self-attention in layer 4 is highly diagonal along the sequence; its output positions (vertical) draw from very near their own position from the input (horizontal)) By layer 13, however, this pattern changes completely, and information concentrated in the front of the sequence appears to distributed broadly. Suddenly, in the next two layers, the
'It is not automatically the case that RNN-T models exhibit good length generalization; it is by some combination of deliberate preparation and chance that cases do.
7

---

audio-to-text alignment is clearly visible. The label contains 12 word pieces, and precisely that many output positions receive information from points distributed monotonically along the imputs these layers. The remaining output positions are possibly populated with filler values-these positions in the final output embedding receive no training. The alignment is complete even before the final layer, where unrelated distribution is seen.
In this network, all the attention heads showed the same alignment operation hap- pening in the 14th and 15th layers, for every input example we observed. Early to middle layers performed different mix- tures of diagonal (local) and non-dian non-diagonal (global) attentions for different heads, which were less interpretable. Overall, the self-attention layers appear to be primar- ily performing audio encoding roughly in- place for many layers, and then they very explicitly perform the full alignment within as little as two layers. Interestingly, posi- tions near the beginning and end of the sequence are perhaps used as margins for bookkeeping, where the training utterances often contain silence anyway.
"Ilustration italian millet" (12 word-pieces)
Layer 4
Layer 13
The alignment itself is useful in many ap- plications, and can be estimated from to- ken emission positions when decoding with RNN-T. Our model has obscured that infor- mation, but the observed behavior affords the possibility of extracting alignments dur- ing inference. The bottom subplot of Fig- ure 3 shows the average of all attention heads in Layer 15. Seen side by side, they closely track the RNN-T alignment, which likely close to ground truth.
Laver 14
Layer 15
Layer 16
Layer
4.5.2 Embedding Alignment
To further study the alignment process, we also examined the intermediate em- beddings themselves by the following pro- cedure. Given the converged Aligner- Encoder model, we trained an RNN-T model of the same size, which is randomly initialized except that the parameter values from the beginning layers of the Aligner are used in the corresponding layers of the new encoder. The Aligner weights were model
Chttpat Position
Input Position
Figure 2: Self-attention probabilities from single head different layers in a 17-layer Aligne-Ennoode per- forming audio-to-text alignment.
kept frozen, so that the portion trained by RNN-T receives input the in- tenmediate Aligner embedding. Finally, we could measure alignments through the RNN-T decoder as usual-we computed the full forward-backwe this procedure using different numbers of Aligner-Encode layers as the foundation, we traced the alignment progression of the internal representation through the forward pass of the original model.
probabilities of the decoding lattice. By repeating
The result is shown in Figure 3] where the suddenness of the process is apparent. Within single layer the representation shifts from its original layout to become fully front-aligned with the output. This is same layer that exhibited the most pronounced alignment self-attention probabilities previously. In the figure, each subplot is independently normalized, and the decoding lattice probabilities (computed from the forward and backward passes) are re-computed using high temperature to exaggerate the tails of the distributions, which are tightly peaked. The bottom subplot shows the self-attention

---

probabilities averaged over all heads in Layer 15, normalized by row but not temperature-adjinn adjusted. The hot spots track closely with the original RNN-T alignment- the lattice calculation fills in probability mass where the blank token is to be emitted, whereas the Aligner self-attention ignores those segments. The close correspondence of the pauses especially visible.
4.5.3 Failure Mode Analysis
Using the same diagnostic technique, we can examine the failure mode of poor length generalization. Figure shows such case, for utterance that is 1.5x longer than any training ex- ample. The model is only able to align a portion of utterance, explaining why sometimes many deletions occur from dropping the latter part of the text. Interestingly, the RNN-T model trained atop the 15 frozen Aligner- Encoder layers (as in the previous sub- section) still assigns some probabil- ity the remainder of the label se- quence, in a somewhat aligned fash- ion but pushed the tail of the em- bedding sequence. This part is not de- coded successfully. As we increased the number of Aligner layers used, we observed that the RNN-T hypotheses began to lose tail words several layers prior to the apparent alignment layer. Along with the self-attention visual- ization of Figure 2] this supports the conclusion that the Aligner works for several layers to prepare for the align- ment operation; the over-length sequences already begin to be disrupted earlier. Separately, note that for sequences which are near the Length-capacio of the all-Aligner model, when deletions begin to occur, observed them to often happen somewhere in the middle.
Aoorlae odrantogs Lhee not aitacc posses Usit oxspandt.w univanaly irve. the aluacio milion and Ry."
Decoding Lattice: RNN-T Only
Decoding Lattice: A-Ainneer-Lave RNN-T
Decoding Lattice: 15-Aligner-Laye RNN-T
Self-Attention- Aligner Laver
50
100
150
200
Figure 3: Decoding lattice probabilities (U vs 7) from RNN- T-on-Aligner and self-attention weights exhibiting successful alignment, within a specific layer.
4.5.4 Reverse Alignment
Decoding Lattice: 14 14-Aligner-Lave RNN-T
We conducted final experiment to demonstrate the likely applicability of Aligner-Encoders to tasks other than ASR. Specifically, we desired to show that model has greater flexibility in aligning information than only the monotonic and increasing alignment relationship of standard ASR. To do of
Decoding Lattice: 15-Ali Aliinlgg---ave Layer RNN-T
Self- Aalf-Attention Aligner Layer 15
so, constructed the extreme decreasing alignment by completely reversing the audio input, so frames at the end of the audio input corre- spond to the beginning of the text out- put, and vice versa. Experimenting in LibriSpeech, the resulting model achieved similar performance as the regular model on utterances within the training length. Further, the self- attention weights exhibited the same behavior as in the forward model, except that in the aligning layer the weights trace from the upper-left to lower-right, showing the sequence reversal in action
250
500
750
Figure 4: Decoding lattice probabilities (U vs T) from RNN- T-on-Aligner and self-attention weights exhibiting failure mode for utterance 1.5x longer than trained.

---

(compare against the bottom panel of Figure B) We include alignment plots for this model in the appendix, Figure []] The equal ability completely reverse the order of the information in the encoder embedding strongly suggests that our model could perform non-monotonic tasks, as well, which require more varied information re-ordering. Two prominent non-monotonic tasks where Aligner-Encoders have potential to simplify modeling include machine translation (MT) (e.g. (50)) and automatic speech translation (AST) (e.g. [51454]).
4.6 Computational Efficiency
Considering training speed, memory usage, and inference speed, we observed favorable comparisons for our model owing to its reduced complexity. While the exact numbers will depend on many hardware and implementation details. we present a representative case using our -0rrmearre encoder LibriSpeech models in Table Notably, the RNN-T I spent roughly 290ms per training step in the decoder and loss (forward + backward propagation), whereas our model required only 29ms, a 10x gain. Our RNN-T loss implementation is highly optimized, so the majority of the gains came from eliminating the sean associated with the T dimension of the joint network output. The peak memory usage for our model decreased by 18% (by 1.4G per accelerator f The 4-layer AED decoder trained with a step time essentially as fast as ours, thanks to good parallclizatio of transformers, and did tend to converge in significantly fewer steps (see A.2
The inference speed of our model, however, was by far the fastest, especially relative to AED. We profiled all models using 11.5 sec-
Table 5: Example measured compute times for our LibriSpeech models (lower is better).
onds of audio, a batch size of 8, and beam search of size 6 (effective decoder batch size 48). The decode step for the Aligner and RNN- (L.STM + Joint network) measured a mere 190ps. As discussed earlier, the total decode time as the step time multi- plied by the text length (U) for our model, versus multiplying by the sum of the audio and text lengths (T + U) for the RNN-T, re- sulting in substantial savings. We approximate with T = 300, U = 100. Turning to AED, each step through the transformer decoder was almost two orders of magnitude slower, at 8.5ms (and U steps are executed) o The forward pass itself required 5ms per step, and re-ordering the decoder state as part of the beam search occupied the other 3.5ms. The re-ordering time was negligible in model due to the minuscule LSTM state. Including the encoder, the total inference time of our model is estimated at 2x faster than RNN-T and 16x faster than AED in this scenario.
(MILLISECONDS)
AED
RNN-T
ALIGNER
TRAINING STEP: (ENcoDER=560MS)
DECODER+LOSS
31
290
29
TOTAL
591
850
589
can estimate
INFERENCE: (ENCODE= 33Ms;
T=300,
-=
DECODE STEP
8.5
0.19
0.19
DECODE TOTAL
850
76
19
832
108
51
5 Conclusion
We have shown that transformer-basee encoders are able to perform audio-to-ted sequence alignment internally during a forward pass, using only self-attention (i.e., prior to decoding, unlike previous models). This finding enables the use of much simpler ASR model, the Aligner-Encoder, which has simple training loss and achieves lower decoding complexity than past models. A key design point of our model is to keep the benefit of auto-regressive modeling while removing as much computation as possible from the auto-regressive loop. Desirable extensions to model for ASR that would require further study include: use in streaming recognition, ability for multilingual recognition, and incorporation of traditional pretrained encoders, to name few. Fusion with an external language model 24 could be simplified relative to RNN-T owing to the absence of blank tokens, and for the same reason our model may be more receptive training other losses such from text-only data. Looking beyond ASR, application to machine (text) translation would require some modification to permit output sequences longer than the input, although speech translation would not. Altogether, Higner-Encodeer and their newly identified ability to learn self-transduction present promising opportunities for future research and application.
'Total training time was roughly day and half.
5 'e disabled path merging for RNN-T.
OOur implementation included both self- eeffaateetion and cross-attention in all layers, and decode cache.
10

---

Acknowledgments and Disclosure of Funding
The authors would like to acknowledge the numerous members the Google Speech Team who contributed either directly indirectly through work shared code base, data preparation and maintenance, experiment and evaluation infrastructure, and furthermore by providing guidance and answering questions pertaining to these elements and other past research experience. We also thank the several anonymous reviewers who provided valuable feedback resulting in significant improvements in the quality of this paper.
References
[1] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[2] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv: 2012.
[3] Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end con- tinuous speech recognition using attention-based recurrent mn: First results. arXiv preprint 1412 1602, 2014.
[4] Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. Advances in neural information processing systems, 28, 2015.
[5] William Chan, Navdeep Jaitly, Quoe V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv: 1, 2015.
[6] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philémon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4945-4949, 2016.
[7] Jinyu Li. Recent advances in end-to-end automatic speech recognition. APSIPA Transactions on Signal and Information Processing, 1 11()), 2022.
[8] Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, and Shinji Watanabe. End-to- end speech recognition: A survey, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:325-33: 35 2024. doi: 10.109/TAALLP22033333222
[9] Tom Bagby, Kanishka Rao, and Khe Chai Sim. Efficient implementation of recurrent neural network transducer in tensorflow. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 506-512, 2018. doi: 10. 20199SLL.201886639669
[10] Khe Chai Sim, Arun Narayanan, Tom Bagby, Tara N. Sainath, and Michiel Bacchiani. Improving the efficiency of forward-backwan backward algorithm using batched computation in tensorflow. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 258-264.
2017. doi: 10. 10.1109// 1010//RU.220 177.8200...2699444 8268944.
[11] Jay Mahadeokar, Yuan Shangguan, Due Le, Gil Keren, Hang Su, Thong Le, Ching-Feng Yeh, Christian Fuegen, and Michael L. Seltzer. Alignment restricted streaming recurrent neural network transducer. 2021 IEEE Spoken Language Technology Workshop (SLT), pages 52-59, 2021.
[12] Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. Pruned rnn-t for fast, memory-efficient asr training, 2022.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.

---

[14] Linhao Dong, Shuang Xu, and Bo Xu. Opeech-transforee A no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5884-5888, 2018.
[15] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, and Wangyou Zhang. A comparative study on transformer vs rnn in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 446, 2019.
[16] Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, and Zhengqi Wen. Self-attention transducers for end-to-end speech recognition. In Interspeech, pages 2019-20200 2023, 2019.
[17] Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, Yonggiang Wang, Due Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, and Michael L Seltzer. Transformer-transduuerr End-to- end speech recognition with self-attention. arXiv preprint arXiv: 1910: 2019.
[18] Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874-6878, IEEE, 2020.
[19] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and mn-t loss. In ICASSP 2020 2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833, 2020. doi: 10.1 10.110//CASSP400776 STW.SAS544777..22.99955955 2020.905386..
[20] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmenn transformer for speech recognition. arXiv preprint arXiv:2005. 08100, 2020.
[21] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[22] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural computation, 12(10):2451-247 2000.
[23] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826 2016.
[24] Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence to sequence models. arXiv preprint arXiv: wet:::11200000 2.06695, 2016.
[25] Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. State-of-the-art speech recognition with sequence-to- sequence models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4778, 2018. doi: 10. 10.1199/CAASP22211..4222
[26] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 193-199, IEEE, 2017.
[27] Hervé Bourlard and Nelson Morgan. Hybrid connectionist models for continuous speech recognition. In Automatic Speech and Speaker Recognition: Advanced Topics, pages 259-283.
Springer, 1996.
[28] Anshuman Tripathi, Han Lu, Hasim Sak, and Hagen Soltau. Monotonic recurrent neural network transducer and decoding strategies. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 944.948, 2019, doi: 10.1 AS.1109/SSUU4699..... AAAAS5444991200 2019.
9003822.
12

---

[29] Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley. Hybrid autoregressive transducer (hat). In ICASSP 2020-2022 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6139-6143, IEFE, 2020.
[30] Xie Chen, Zhong Meng. Sarangarajan Parthasarathy, and Jinyu Li. Factorized neural transducer for efficient language model adaptation. In ICASSP 2022 -202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1132-8136, 2022.
[31] Zhong Meng, Tongzhou Chen, Rohit Prabhavalkar, Yu Zhang, Gary Wang, Kartik Audhkhasi, Jesse Emond, Trevor Strohman, Bhuvana Ramabhadran, W Ronny Huang, et al. Modular hybrid autoregressive transducer. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 197-204. IEEE, 2023.
[32] Colin Raffel, Minh- Minh-Thang Luong, Peter J. Liu, Ron Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments, 2017.
[33] Linhao Dong and Bo Xu. Cif: Continuous integrate-and-fire for end-to-end speech recognition, 2020.
[34] Keqi Deng and Philip C Woodland. Label- neural transducer for end-to-end ast. arXiv preprint arXiv:2307.0000 2023.
[35] Tian-Hao Zhang. Dinghao Zhou, Guiping Zhong, Jiaming Zhou, and Baoxiang Li. Cif-t: A novel cif-based transducer architecture for automatic speech recognition. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10531-10535, 2024.
[36] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid etc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1 11(8):2240-1255 2017.
[37] Yun Tang, Anna Sun, Hirofumi Inaguma, Xinyue Chen, Ning Dong, Xutai Ma, Paden Tomasello, and Juan Pino. Hybrid transducer and attention based encoder encoder-decoder modeling for speech too text tasks. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki, editors, Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12441-12455, July 2023.
[38] Jinchuan Tian, Brian Yan, Jianwei Yu, Chao Weng, Dong Yu, and Shinji Watanabe. Bayes risk ctc: Controllable etc alignment in sequence-to-sen to-seqeenc tasks. arXiv preprint ar arXiv:2210.07499, 2022.
[39] Jinchuan Tian, Jianwei Yu, Hangting Chen, Brian Yan, Chao Weng, Dong Yu, and Shinji Watanabe. Bayes risk transducer: Transducer with controllable alignment prediction. arXiv preprint 23X.v:2308..1111 2023.
[40] Weiran Wang, Rohit Prabhavalkar, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Zhong Meng, CJ Zheng, Yanzhang He, Tara Sainath, and Pedro Moreno Mengibar. Massive end-to-end models for short search queries, 2023.
[41] Rohit Prabhavalkar, Zhong Meng, Weiran Wang, Adam Stooke, Xingyu Cai, Yanzhang He, Arun Narayanan, Dongseong Hwang, Tara N Sainath, and Pedro Moreno. Extreme encoder output frame rate reduction: Improving computational latencies of large end-to-end models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11816- 1820. IEEE, 2024.
[42] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEFE, 2015.
[43] Dongseong Hwang, Khe Chai Sim, Zhouyuan Huo, and Trevor Strohman. Pseudo label is better than human label. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 1421-1425. ISCA, 2022. doi: 122443/NNTER33PFFC222222211 1034. URL httpstt::: / /ooi org/ 10. 21437/ htte//peeeeereeeeeeeeeeeeeeeeece 2022 11034
13

---

[44] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXivv 1609.00 08 144, 2016.
[45] Bowen Liang, Pidong Wang, and Yuan Cao. The implicit length bias of label smoothing on beam search decoding. arXiv preprint arXhv:2205.0063. 2022.
[46] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.0280 02860, 2019.
[47] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiigg 568:127063, 2024.
[48] Chung-Cheng Chiu, Arun Narayanan, Wei Han, Rohit Prabhavalkar, Yu Zhang, Navdeep Jaitly, Ruoming Pang, Tara N. Sainath, Patrick Nguyen, Liangliang Cao, and Yonghui Wu. Rnn-t models fail to generalize to out-of-domain audio: Causes and solutions. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 873-880, 2021. doi: 10. 1091009944490000202220 9383518.
[49] Tae Gyoon Kang, Ho-Gyeeon Kim, Min-Joong Lee, Jihyun Lee, and Hoshik Lee. Partially overlapped inference for long-form speech recognition. In ICASSP 2021-2021 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pages 9993.
IEEE, 2021.
[50] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015,
[51] Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In NIPS Workshop on End-to- End Learning for Speech and Audio Processing, 2016.
[52] Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. Cross attention augmented transducer networks for simultaneous translation. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 39-55, November 2021.
[53] Shun-Po Chuang, Yung-Sung Chuang, Chih-Chiang Chang, and Hung-yi Lee. Investigating the reordering capability in CTC-based non-autoregressive end-to-end speech translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1068-1077, Online, August 2021,
[54] Jian Xue, Peidong Wang, Jinyu Li, Matt Post, and Yashesh Gaur. Large-Scale Streaming End-to- Speech Translation with Neural Transducers. In Proc. Interspeech 2022, pages 2022.
[55] Kwangyoun Kim, Felix Wu, Yifan Peng. Jing Pan, Prashant Sridhar, Kyu J Han, and Shinji Watanabe. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 84-91. IEEE, 2023.
14

---

A Appendix supplemental material
A.1 LibriSpeech Expanded Settings and Results
Table 6: LibriSpeech common training settings.
SETTING
VALUE
LEARNING RATE
5.0
LEARNING RATE DECAY
SQUARE-ROOT LINEAR (10,000)
LEARNING RATE WARMUP (STEPS)
OPTIMIZER
ADAM
OPTIMIZER BETA-I, BETA-2
0.9, 0.98
OPTIMIZER EMA DECAY
0.9999
CLIP GRAD NORM
5.0
L-2 REGULARIZER WEIGHT
1E-6
BATCH SIZE
2,048
TRAINING STEPS
150,000
VARIATIONAL. NOISE SCALE
0.075
VARIATIONAL. NOISE VARIABLES
TEXT EMBEDDING, LSTM
SPECTRUM AUGMENTATION
TIME & (RNN-T: 32)
CONFORMER CONY KERNEL
AILF-ATTENTIO HEADS
8
TEXT EMBEDDING SIZE
128 (AED: 512)
LABEL SMOOTHING WEIGHT
0.1 (RNN-T: N/A)
Table 7: WER (%) on LibriSpeech Test-Clean set by utterance duration, including models trained with concatenated training examples covering up to the maximum test length of 36s. The number of test utterances in each category is 2466, 89, and 65, in order of length.
TEST-CLEAN
< 17s
17-21s
21s
ALL
CTC
2.8
2.7
3.5
2.8
RNN-T
2.1
1.9
2.8
2.1
AED ALIGNER
2.3
2.1
15.3
3.3
2.4
7.0
28.0
4.8
AED-CONCAT
2.4
2.1
2.8
2.4
ALIGNER-CONCEE
2.2
2.1
2.9
2.3
Table 8: WER (%) on LibriSpeech Test-Other set by utterance duration, including models trained with concatenated training examples covering up to the maximum test length of 36s. The number of test utterances in each category is 2834, 70, and 35, in order of length.
TEST-OTHER
< 17s
17-21s
21s
ALL
CTC
6.6
6.0
4.3
6.4
RNN-T
4.7
4.4
3.1
4.6
AED ALIGNER
5.4
5.4
24.2
6.3
5.2
8.4
29.2
6.5
AED-CONCAT
5.6
5.6
3.0
5.5
ALIGNER-CONEE
5.2
5.3
3.3
5.1
A.2 LibriSpeech AED versus RNN-T Performance
While historically AED has out-performed RNN-T on LibriSpeech, this appears to have changed with the introduction of the Conformer (20 To this point, there are several relevant factors worth noting for our case. Both models receive positional encoding following the feature convolution layers. Our RNN-T is trained and operated in non-streaming mode (e.g., with non-causal attention). We trained
15

---

AED with label smoothing (weight of 0.1) to prevent overfitting. Our AED models did often converge much faster, sometimes as few as 25k training steps; we conducted small hyperparameter search over learning rate and schedule to ensure against overfitring. Lastly, our AFD decoder is 4-layer transformer with 18M parameters (total model parameters 128M), which already significantly larger than the 3.5M-pan M-parameterr LSTM decoder for the RNN-T. A 148M-paramett conformer AED was reported in [55] to achieve 2.16% and 74% on Test-Clean and Test-Other, respectively, which is better than our smaller AED baseline but still less good than our RNN-T, which we keep as the relevant SOTA for comparison.
A.3 Reverse Alignment Experiment Figure
"anctherr wiicato comcanatioe inmoaniy
attucks of miiddww
Layer
Layer 14
Leyer 15
Layer
Layer
0utouf Rostion
InpuC Foston
Figure Self-attention weights an Aligner-Encoder Encoder (from single head) trained on reversed audio; the reverse alignment is clearly visible in layers 15 and 16. (LibriSpeech, 17-layer encoder).
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper is carefully scoped to describe a new, simplified model for ASR which is enabled by newly-observed learnable capability of self-attention. Numerous experiments showcase the new model and compare against the pre-existing state of the art systems.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The paper discusses a primary limitation we discovered relative to some previous models, which is inability generalize longer sequence lengths (some previous models share this limitation). Significant experiments were conducted to address this limitation, with good success but more could be done. Possibly other limitations exist which we did not explore thoroughly, such as: is more data required to train our model versus the previous models- none of our standard experimental settings showed an effect, but they also would not be considered low-resource datasets. We do not hide that our model is very slightly lower recognition performance than the best RNN-T models with the same settings in most of our experiments-we make claims to achieve new state-of-the-at tfr in terms of WER. We have also not claimed to be able to operate our model streaming fashion, which may be seen by some limitation.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
17

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide complete description of the model, and make an effort to convey the relevant settings for training and evaluation (see appendices). Furthermore, we strove to change settings little as possible from those employed with previous models, showing that it should likely be relatively straightforward to adopt the new model. Lastly, one of the main benefits of the new model is its simplicity-sines it uses standard deep learning components and removes complications relative previous models, the description should suffice for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
18

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We conducted experiments on two proprictary datasets, which cannot be released, and to balance this we also conducted experiments open-source data which is already fully available. We are also unable to release code, however as discussed under reproducibility, we share full details of the implementation settings, and by the simpler nature of our model, it requires no coding tricks relative to previous models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
bablicc/uides//eee wowuiddss/ooeeuumissiiiiiiiii www.aimiisssoooooocccccoc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guwdes/ SwodeeSubmissiooool.....o
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: The body of the paper describes the main points of training and model settings, and the appendix includes more detailed settings.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to computation constraints, we elected to explore training multiple different datasets (including their own hyperparameter searches) better use resources to build confidence in the method, rather than running more repeats of the same setting for any given dataset to produce true confidence intervals. As we do claim SOTA, our results do not hinge too closely on the exact precision of model performance, and we do not report an excessive number of significant digits.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss the compute resources, memory, and time of execution in the computational efficiency section. Our experiments are more efficient but require a similar amount of compute as established ASR systems.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
20

---

9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Any data potentially containing PII has been anonymized prior to use. Our method introduces no new potential societal impacts distinct from existing ASR systems.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work introduces no new societal impacts distinct from existing ASR systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
21

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode. www.coo//aaoo//c datastt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
22

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "fAzxVe810A", "reviewer": "Reviewer_JgvT", "summary": "A new simplified encoder-decoder model is presented without the attention. The decoder is generating the labels auto-regressively as usual until end-of-sentence (EOS). In contrast to attention-based encoder-decoder (AED) models, the attention is replaced by simply taking the same frame in the encoder - i.e. in decoder step u, it will take frame u also from the encoder. Thus the output sequence can never be longer than the input sequence. The idea is that the encoder with self-attention can already realign the information as necessary to output it label by label. The remaining encoder output frames after EOS are ignored, but of course all intermediate encoder frames are used due to self-attention.\n\nIt's an interesting test to see whether the self-attention is enough to already learn this. And the answer is yes, it can learn this.\n\nExperiments are performed on three speech recognition tasks:\n\n* Librispeech with 960h train data\n* Voice Search with 500kh train data\n* YouTube videos with 670kh train data\n\nIn all cases, a Conformer encoder is used. Word pieces are used as output labels.\n\n\nThe self-attention weights are analyzed and it is observed that the realignment happens in layer 14. This is also verified in another way, by freezing the first N layers of the encoder, randomly resetting the other encoder parameters, and adding a RNNT on top of it and training that, and then generating the RNNT soft alignment. With N>14, the alignment looks like the identity mapping.", "questions": "How is the convergence rate in comparison to CTC, RNNT, AED? How is the alignment behavior early in training?\n\nTable 3, what is dev, is that dev-clean, dev-other, or both combined?\n\nTable 3, it seems a bit weird to me that the RNN-T is so much better than the AED model. I would actually expect the opposite. For example:\n\n* A Comparison of Sequence-to-Sequence Models for Speech Recognition,\nhttps://www.isca-archive.org/interspeech_2017/prabhavalkar17_interspeech.html\n* On the Comparison of Popular End-to-End Models for Large Scale Speech\nRecognition, https://www.microsoft.com/en-us/research/uploads/prod/2020/11/template-5fa34dc776e7f.pdf\nBoth show that AED is better than RNN-T. And this is what I have seen on many other occasions as well. Was this expected to you? Why? Or if not, how do you explain it?\n\nWhat happens when f_pred is just a feed-forward network without the recurrence (dependence on g_{i-1}), i.e. you would get a model with only the last label as context? For RNN-T, it has been shown that this performs equally well as using the whole history. It would be interesting to see how it behaves for this model. (For an AED model, this is not really possible because the cross attention mechanism needs it.)", "limitations": "-", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Interesting idea and model.\n\nThe work shows that this simple idea seems to work, even though its performance stays behind the other existing models (CTC/RNNT/AED).\n\nInteresting analysis on the attention weights and retraining the encoder partly and looking at the RNNT soft alignment.", "weaknesses": "No source code to reproduce the results?\n\nNo references are given to Voice Search and YouTube data, so it's impossible to reproduce and verify the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work closely.  \n\nThe convergence rates are similar among CTC, RNN-T, and our model (good checkpoints are between 100k-150k training steps on LibriSpeech).  Interestingly, AED models sometimes did converge faster (as fast as 25k training steps for the best checkpoint).  We did not investigate this further other than to run AED again with a lower learning rate in case that was too high, but we weren't able to find a better result.  So in this regard our model is more similar to RNN-T.\n\nIn Table 3 Dev is Dev-Clean only, we will edit the column label.\n\nThis is an interesting question about AED versus RNN-T.  A possible explanation is the use of the conformer encoder.  \"A Comparison of Sequence-to-Sequence Models for Speech Recognition\" uses attention architectures that predate transformers.  \"On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition\" finds an improvement using transformer-AED over RNN-AED, but it only reports RNN-T in its Table 1.  Our results more closely match those of the Transformer-AED and Conformer-Transducer given in the Conformer paper https://arxiv.org/abs/2005.08100 (Table 2) which also experiments on LibriSpeech.  In searching around for this question, we found a paper we could cite which gives a score for Conformer-AED on LibriSpeech is: https://arxiv.org/abs/2210.00077, which reports test-clean 2.16% and test-other 4.74%.  This is better than our AED results (with conformer), although they use more learnable parameters: 148M versus ours at 128M. It is still less good than our RNN-T (conformer), so it would not change the SOTA against which we are comparing.  Still, it seems worth citing and including in the comparison, and investigating their other settings.\n\nThis is a very interesting question about running our model with limited recurrence in the decoder.  It is true that RNN-T models can often perform well using a history of only the 2 most recent tokens.  Time permitting we will attempt to launch this experiment, thank you for the suggestion.\n\nUnfortunately our implementation cannot be disentangled from a code base which cannot be shared.  We hope that the fact that our model is a simplification over previous models, and uses more standard deep learning components, will make it relatively easier to reproduce the results, and we have attempted to include extensive hyperparameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbI6LVzIso", "reviewer": "Reviewer_RgJ5", "summary": "This paper proposes a new speech recognition (or, more generally, sequence-to-sequence) architecture. This architecture performs an alignment process between input and output features via self-attention mechanisms in an encoder. The decoder network is a simplified version of the combination of the RNN-T or AED decoder, and its relationship is detailed in Section 2.2. The primary advantage of this method is reducing the computational cost by avoiding the dynamic programming to adjust the input and output length in RNN-T or cross-attention to consider all possible scores across the input frame and output token. The experiments show the comparable performance of the proposed method to RNN-T and AED, but it significantly reduces the computational costs.", "questions": "- Section 2.1: Do you need $U \\leq T$? Can we apply this method for $T \\leq U$? This would happen in the general sequence-to-sequence problem like MT or TTS.\n- Section 4.2: Did you try it with the other encoder layer than the Conformer (e.g., vanilla transformer)? I'm curious because the alignment properties of this method might depend on the convolution operation in the Conformer.\n- Section 4.5.2: I'm not sure how each attention head behaves. Can you discuss a bit more about how this behavior is different across the head?\n- Section 4.5: Do you have some results on MT or AST?\n\nSuggestions\n- I recommend the authors emphasize the practical benefit of the computational complexity of this method in the abstract.\n- Sections 4.5.3 and 4.5.5: These sections provide the main experimental results in the appendix, which are not recommended. This way, it breaks the page limit rule and is unfair compared with the other papers, which put all main results in the main body. These sections should be rewritten to avoid using the appendix results. Note that some supplemental use of the appendix is no problem (e.g., Table 5 is a good example. This is too detailed and may not be so crucial for understanding the main idea of this paper, but it is essential for reproducibility. So, it is adequate to be located in an appendix section.).", "limitations": "- Since this is not based on a hard alignment approach (RNN-T and CTC are based on hard alignment), I'm curious about the hallucination issues often observed in AED or decoder-only architectures. For example, OpenAI's Whisper is based on AED (soft alignment). It has a serious hallucination issue (despite its outstanding performance), and I think this is a potential limitation of the soft alignment-based approaches in general. I want the authors to discuss this aspect. This method probably has an advantage over AED due to its shallow decoder architecture, but I'm not very sure.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Novel speech recognition architecture or, more generally, novel sequence-to-sequence architecture. \n- Comparable performance to other SOTA speech recognition architecture (AED and RNN-T) while reducing the computational complexity.\n- Interesting analysis of the alignment behaviors and detailed ablation studies", "weaknesses": "- Weak reproducibility due to the use of non-public data for main ASR experiments and the lack of source code release. Note that I did not penalize this point in my initial judgment. But if this part is improved, I'll raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful consideration of our work.\n\nImportant question about the requirement for U <= T, especially for machine-translation.  One possible solution to extend to U>T would be to pad the input with some (fixed) number, P, of learnable input frames, which would provide the model the ability to write T+P output tokens.  (This is similar to the \"registers\" for vision transformers: https://arxiv.org/abs/2309.16588.)  Another possibility, for U >> T, would be to train the model to decode two (or more) tokens per embedding frame.   We did not need these changes for any of our ASR experiments, but would like to include them as suggestions in a revised discussion section.\n\nWe only tried Conformer because it tends to perform better and was already in use for many pre-existing baselines, so we could expect the encoder quality to degrade without the convolutions (even if the alignment is still possible to perform solely with the transformer).\n\nIn Figure 2, we plotted attention weights for a single head.  The patterns do look different for different heads in the early layers (this also happens in RNN-T), but we found them all to express the alignment as seen in Layers 14 & 15.  So in Figure 3 we were able to average across all heads when showing the alignment in Layer 15, which is probably more accurate than relying on a single head.\n\nUnfortunately we do not have expertise or infrastructure for MT or AST---we welcome any correspondence from future readers over future work!\n\nRegarding sections 4.5.3 and 4.5.4 figures, we placed images for these ablations into the appendix so they could be printed as large as possible.  On re-reading, we did not adequately describe the figure from 4.5.4 for the text to serve as a standalone result--will revise!\n\nOn hallucinations, after looking at many, many examples, we did not observe this issue in any of our models, and did not see abnormally high insertion errors.  In other work, we have observed hallucinations when using a larger language model in AED-style, so it seems this is a characteristic of LLMs. It's possible that we have some small number of tokens hallucinated during silence only, which RNN-T models may also do.\n\nIt is unfortunate that we are unable to release some of the datasets and our actual code--our hope is that the result on LibriSpeech (public) is sufficient for reproduction, paired with the fact that our method is only a simplification over previous models.  We are happy to correspond with anyone re-implementing in a public code repository.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D5SxmUF5WM", "reviewer": "Reviewer_Btkf", "summary": "The paper introduces Aligner, which is to take the best parts from RNN-Transducer and AED (Attention Encoder-Decoder) models. The idea comes from the intuition that the transformer encoder with self-attention can already learns to align the input and the output -- which is explicitly modeled by previous approaches like applying techniques to ensure the monotonic alignment or doing dynamic programming to find the alignment. Aligner just simply train with cross-entropy loss, and the results surprisingly show that the encoder can internally learn to align the input and the output during the forward pass. Experimental results showed that Aligner can perform comparably with existing baselines, and show the computational efficiencies.", "questions": "No questions for now.", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation about the computational efficiencies of existing models and the idea of developing Aligner is very clear. \n- The idea is derived from the observation about internal alignment of Transformers, which give meaningful insights to the readers.\n- It successfully tackles the limitations of existing ASR models with simple and intuitive ways, and the results show that it works well.\n- The paper includes sufficient amount of evaluation results and showing the limitations (in 4.5.3) as well, providing insights to the readers about these models much.\n- I feel the writing is also very clear.", "weaknesses": "- The idea is yet confirmed in the model fine-tuning setups; as recent Whisper model [1] shows that large-scale pretraining can leads to performant ASR models, I hope this Aligner work applied to the large-scale pretraining setups and show the effectiveness as well. Note that I feel this is not a critical weakness of this paper.\n- Honestly, I am not following the most recent state-of-the-art models for ASR systems, and not sure if there are some missing baselines that the authors should also compare with. I am willing to listen to other reviewers' opinions about this.\n\n[1] Robust Speech Recognition via Large-Scale Weak Supervision, Radford et al, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for considering our work closely, your summary is accurate to what we intended.\n\nIt is a worthwhile question whether pre-training can be combined with our model.  As it stands, our model seems to use several of the same layers in the conformer for 1) encoding and 2) alignment, whereas existing pre-training methods will only train encoding.  It would be interesting for example to take an existing good encoder (e.g. from RNN-T) and see if it can be fine-tuned for a small number of steps to learn the alignment, rather than needing to start from scratch.  We may attempt this experiment, thank you for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UUbpAjKifu", "reviewer": "Reviewer_VLfL", "summary": "This paper proposes a new ASR model that connects a self-aligned encoder and the light text-only recurrence of RNN-T. The proposed model can be trained with label-wise cross entropy loss, which is computationally efficient than RNN-T training. The authors show the limitation of the model to inference on long-form audios and give a special inference configuration to mitigate the issue. Experiments on Librispeech and larger-scale ASR datasets demonstrated the close performance compared to other ASR models. The authors also show the audio-text alignment in the self-attention weights of a certain layer, which could be said to perform “self-transduction”.", "questions": "Several additional questions:\n\n1. If I remember correctly, the AED model can achieve no worse WER than RNN-T on Librispeech in literature, I am wondering did the authors look into other reasons why AED in Table 3 is much worse than RNN-T (except the long-form problem)?\n2. Any results comparing rotary positional embedding and relative positional embedding for speech encoder on long-form audio? The authors mentioned that RNN-T used RPE, is it the reason why RNN-T is good at long-form audio?", "limitations": "The main limitation is the use case of the proposed method, aka, non-streaming applications. Even though, the method can be helpful for many different purposes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposed a new ASR model for better training and inference efficiency. The idea can be similar to performing down-sampling on encoder side but is more aggressive (to token-level). The strengths of the paper are:\n1. The proposed model consumes way less GPU memory than RNN-T and achieves much lower latency than AED.\n2. The analysis of the text-audio alignment behavior in self-aligned encoder.\n3. A modification on Aligner without training to enable long-form audio decoding.", "weaknesses": "The major concerns are the presentation and the limitation of the proposed model. \n1. The explanation of the aligner modification for long-form audio is somewhat hard to follow. It may be better to have figures, or equations involved in the paragraph.\n2. As the claim of the paper is that Aligner encoder achieves better efficiency than RNN-T and AED, it would be better to summarize the experimental numbers in Section 4.6 into a table for reader's convenience. People might be interested in those strong numbers.\n3. The proposed model may have limited use cases e.g. offline ASR because the encoder cannot be implemented for streaming with the current design.\n4. A related work that replaces RNN-T loss with the cross-entropy loss and reduces the GPU memory usage. (https://arxiv.org/pdf/2307.14132).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the close review of our work.\n\nThank you for the suggestion to revise the section on the long-form inference modification.  In earlier drafts of the paper, this has been the most difficult part to write clearly.  Space permitting, a figure or perhaps a pseudo-algorithm would be helpful, we will continue to revise to try to improve this section.\n\nGood idea to highlight the computational efficiency / latency results in a table.\n\nIt is true we make no claims toward streaming capability.  It is possible that our model is capable of performing streaming recognition like RNN-T models trained for this purpose (although perhaps not with as low of latency).  Or it might require adaptations similar to what has been done to make Whisper streaming-capable, for example: https://aclanthology.org/2023.ijcnlp-demo.3.pdf.  Since our model decodes non-streaming with much lower latency than AED (owing to our small decoder), it is likely that such adaptations could produce a much lower latency in our model.  It would definitely be worthwhile to investigate streaming capabilities of models based on our Aligner Encoder.  Since this potential limitation is not only for our model, but also applies to AED, and it could require significant further study and reporting, we hope that it does not diminish too much the significance/relevance of our submission, where we have included many other detailed analyses and ablations.\n\nYes, we have cited https://arxiv.org/pdf/2307.14132 in our manuscript: [34].\n\nAnother reviewer also asked about why AED could be worse than RNN-T on LibriSpeech.  We believe this reversal happened with the introduction of the Conformer, as that paper shows RNN-T achieving better performance than transformer-AED.  In researching this question we also discovered a reference that reports a conformer-AED on LibriSpeech (https://arxiv.org/abs/2210.00077), which is still not quite as good as RNN-T (so the SOTA we are comparing against remains the same).  It is better than our conformer-AED result, but it also used more learnable parameters.  Still, we should include it as a point of comparison and double-check all the settings.  We welcome any other suggested references or explanations. \n\nGood question about RoPE versus relative position encoding.  In the RNN-T baselines we ran, the training run with relative positional embedding produced very slightly better results than with RoPE (e.g. 2.1 versus 2.2, 4.6 versus 4.7), so we reported the relative positional embedding results.  However relative positional embedding is significantly slower to train than RoPE, so for the remaining LibriSpeech models (including ours) we used RoPE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the Aligner, a simple yet efficient end-to-end (E2E) model that merges the benefits of RNN-Transducer (RNN-T) and Attention-based encoder-decoder (AED) for automatic speech recognition (ASR). The authors found that the transformer-based encoder is sufficient to handle alignment between speech and text. Leveraging this finding, they utilized the proposed aligner encoder to reposition relevant information to the start of the embedding sequence, enabling them to employ frame-wise cross-entropy loss from AED during training instead of relying on the intricate dynamic programming required by RNN-T, thereby simplifying the training process significantly. During inference, the Aligner demonstrates reduced decoder complexity compared to AED. The new model achieves recognition accuracy similar to that of popular AED or RNN-T models.  

This paper presents a novel approach that is poised to make a significant impact on the speech recognition community. AED and RNN models currently dominate E2E models in ASR, and although numerous new architectures have been proposed recently, none has achieved the success of the Aligner introduced in this paper. It is impressive how this new model seamlessly integrates the strengths of AED and RNN-T models. The efficacy of the proposed method has been demonstrated across three distinct tasks: Librispeech from the public domain, an in-house voice search, and YouTube test sets, thereby reinforcing the credibility of the results. 

The reviewers unanimously agreed this is a high-quality paper and recommended it for acceptance. In the rebuttal, the authors addressed most of the reviewers' questions. Several reviewers questioned why RNN-T outperforms AED in this study, given that AED is generally seen as more powerful while RNN-T is better for streaming. The authors explained that both models have an identical speech encoder setup and cited the Conformer-Transducer paper to support their argument. However, Table 1 in "A Comparison of Sequence-to-Sequence Models for Speech Recognition" shows RNN-AED outperforming RNN-T. I believe the performance difference between AED and RNN-T architectures should be less dependent on the speech encoder, whether it's a Transformer or LSTM. 

In brief, this is an excellent paper featuring a new E2E architecture that merges the strengths of AED and RNN-T. It has the potential to significantly influence the speech community.
Decision: Accept (spotlight)