Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers

Authors: adam  stooke
rohit prabhavalkar google
pedro  moreno mengibar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:09:44

Abstract: Modern systems for automatic speech recognition, including the RNN-Transducer and Attention-based Encoder-Decoder (AED), are designed so that the encoder is not required to alter the time-position of information from the audio sequence into the embedding; alignment to the final text output is processed during decoding. We discover that the transformer-based encoder adopted in recent years is actually capable of performing the alignment internally during the forward pass, prior to decoding. This new phenomenon enables a simpler and more efficient model, the ''Aligner-Encoder''. To train it, we discard the dynamic programming of RNN-T in favor of the frame-wise cross-entropy loss of AED, while the decoder employs the lighter text-only recurrence of RNN-T without learned cross-attention---it simply scans embedding frames in order from the beginning, producing one token each until predicting the end-of-message. We conduct experiments demonstrating performance remarkably close to the state of the art, including a special inference configuration enabling long-form recognition. In a representative comparison, we measure the total inference time for our model to be 2x faster than RNN-T and 16x faster than AED.  Lastly, we find that the audio-text alignment is clearly visible in the self-attention weights of a certain layer, which could be said to perform ''self-transduction''.

Main paper: 
 introduction: 
 The task of sequence transduction requires mapping from an input sequence to an output sequence. It appears in several widespread applications including machine translation and automatic speech recognition (ASR). In ASR, which is the focus of this paper, the two sequences lie in completely different modalities, and the audio input representation is typically much longer than the output text. Thus, in addition to identifying and converting speech sounds, the model must find an "alignment" that moves information from wherever it appears in the input sequence to wherever it belongs in the output. Among other issues, the many possible variations in the pace of pronunciation make transduction a challenging problem.More than a decade ago, and almost a decade apart, two powerful algorithms relying on dynamic programming were developed to perform the sequence transduction task with recurrent neural networks (RNNs). The motivation behind these algorithms was that RNNs require training every output frame in the sequence-with one output per input frame, they needed to be trained with alignments already prepared. The new algorithms allowed training without prepared alignments by computing the probability of the label marginalized over all possible alignments as the optimization objective. The differences in sequence length are accommodated by learning to output "blank" symbols in between true labels and post-processing them out at inference time. The first algorithm, Connectionist Temporal Classification (CTC) [1] , models the output frames as conditionally independent, and this limitation was overcome in the follow-on work RNN-Transducer (RNN-T) [2] , which introduced auto-regressive decoding to achieve better performance.A third algorithm appeared shortly after, called attention-based encoder-decoder (AED). It uses a learned mechanism to account for alignments [3] [4] [5] [6] ; the encoder is followed by a recurrent decoder that cross-attends at every step between its RNN state and the entire encoder embedding sequence to produce its next state. The cross-entropy loss is applied straightforwardly between the plain label sequence and the leading output frames, with no blank symbol and no dynamic programming required. The resulting model produces its output auto-regressively until an end-of-sentence (<EOS>) token halts decoding. AED models, together with CTC and RNN-T, have propelled end-to-end deep neural networks to become the best performing ASR systems across academia and industry (e.g., see surveys [7, 8] ).Despite their great successes, each of these algorithms has its downsides. Implementing the probability summations for CTC and especially RNN-T require non-trivial effort. A naive formulation of the principle is intractable to compute, requiring a dynamic programming approach, which in turn has been the subject of optimization efforts by expert practitioners [9] [10] [11] [12] . Beyond that, RNN-T training requires computing potentially unwieldy tensor quantities to do with cross-pairing every frame of input with every token of output. Both models suffer inefficiencies in decoding. RNN-T processes all frames recurrently while producing a typically much shorter output sequence. AED requires computing over the entire encoder embedding sequence at every decoding step, which leads to slow operation from the high compute load within the auto-regressive loop.In an effort to resolve these difficulties, we ask the question: with the advent of transformer-based encoders [13] [14] [15] [16] [17] [18] [19] [20] , can the ASR encoder itself learn to perform the alignment? The answer we found is: yes, it can. Our main contribution is to show that it is now possible to train neural network speech recognizers with light-weight decoders in the style of RNN-T, but with the simple frame-wise cross-entropy loss of AED. The resulting networks provide accuracy remarkably close to state-ofthe-art models while being more efficient at training and decoding than any previous model. No explicit dynamic programming is employed; instead, the encoder learns to perform the alignment internally during the forward pass. The encoded embedding frames are decoded consecutively from the beginning, one at a time, in conjunction with a small recurrent network that only reads in the previous label, producing exactly one label per frame until emitting <EOS>.Having originated in the era of long short-term memory (LSTM) [21, 22] encoders, the preceeding algorithms all allow the encoder to process information in-place in the time dimension, resulting in embeddings with relevant information in roughly the same position in the sequence as it appeared in the input. For several years now, however, ASR systems have benefited from the adoption of more powerful transformer encoders, but without changing their role. Our Aligner-Encoders are different in that, simultaneous to encoding, they perform the additional task of moving the relevant information to the beginning of the embedding sequence into a label-aligned position. As it is performed solely through the self-attention mechanism within the forward pass of the encoder, one might call this a "self-transducer". The previous style of information flow is contrasted with ours in Figure 1 . A main benefit of our model utilizing an Aligner-Encoder is that it is dramatically simpler to conceptualize, implement, and use. This paper is organized as follows. First, we describe Aligner-Encoder models and relate them to RNN-T and AED. Next, we relate other works which have aimed at improving RNN-T modeling effectiveness or efficiency. Then we report experiments demonstrating the accuracy of Aligner models while also finding limitations for generalizing to long test utterances. In response, we introduce techniques which can be employed at inference-time to provide good long-form performance without any additional training. In further experiments, we analyze how and when Aligner-Encoders perform the alignment and make the surprising discovery that it can happen primarily within a single selfattention layer. Lastly, since ASR alignments are monotonic, we demonstrate that Aligners can at least readily handle reverse alignments, making our model promising for future work in non-monotonic applications such as machine-translation or speech-translation. Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The paper is carefully scoped to describe a new, simplified model for ASR which is enabled by a newly-observed learnable capability of self-attention. Numerous experiments showcase the new model and compare it against the pre-existing state of the art systems.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
aligner-encoder model: 
 The Aligner-Encoder model combines the best elements of RNN-T and AED systems into a more compact form. By requiring the encoder itself to text-align its embedding, we avoid using 1) dynamic programming to sum probabilities in the loss and 2) full-sequence cross-attention in the decoder (in all models we refer to everything inside the auto-regressive portion as the "decoder"). It may be simplest to first lay out our model, which is formulated using the same components as RNN-T, and afterwards draw contrasting points with the heritage.We begin with an input sequence x = (x 1 , x 2 , ..., x T ) of length T , and output sequence y = (y 1 , y 2 , ..., y U ) of length U ≤ T . An encoder network, f enc , processes the input sequence into an acoustic embedding sequence, h = (h 1 , h 2 , ..., h T ), where each frame of the embedding sequence can depend on every frame of input, and typically T ≤ T with subsampling. A prediction network, f pred , processes text labels with forward recurrence only to produce a text embedding sequence, g = (g 1 , g 2 , ..., g U ). The acoustic and text embeddings are fed into a joint network, f joint , to produce the final prediction vector of dimension V according to the vocabulary, for each frame, with softmax probability normalization. The difference in our model is that we enforce the alignment at the encoder output by restricting the model to use the acoustic and text embedding frames in a one-to-one fashion (Equation (3)). The entire model is written by its recurrence relations as:EQUATIONEQUATIONThe encoder and the decoder-which includes both the prediction and joint networks-are parameterized and learned together in an end-to-end manner, with total parameters θ. We maximize the log probabilities of the correct labels, resulting in the familiar cross-entropy loss:EQUATIONThe loss only applies to encoder frames within the length of the label, T ≤ U ; all remaining frames (T > U ) are ignored. Hence the encoder must also learn to be an aligner.We seed the prediction network with a start-of-sentence token, <SOS>, at the first step, y 0 , and empty state h 0 = 0. As is often the case with AED and other uses of cross-entropy loss, we find label smoothing to be a beneficial regularizer and always use it [23] [24] [25] (weight 0.1). During inference the label length is unknown, so the model must predict it, as in AED. We train the model to always predict the end-of-sentence token, <EOS>, as the final token of every training example2 , so decoding proceeds token-by-token until <EOS> is predicted.It is also possible to formulate a non-autoregressive (non-AR) Aligner, which applies a decoder, f ind , to the embedding frames independently, as in CTC:EQUATIONand it can be trained under the same loss. During inference, all tokens are predicted independently, and any tokens after the earliest <EOS> in the sequence are discarded. We include an experiment with this formulation; however, we found its performance to be significantly worse than CTC in all but the shortest-utterance datasets.Given that previous models were developed to overcome contemporary neural encoders' inability to align, the new model naturally brings simplifications. RNN-T trains by explicitly marginalizing over all possible alignments in the loss. This can be visualized in a label-frame decoding lattice (see Figure 1 in [2] ), where each node represents an {encoder-frame, text-frame} pair, and every pairing is a valid state to be considered, resulting in a U × T rectangular grid (abbreviating T as T ). Beyond requiring a sophisticated dynamic programming implementation to calculate tractably (involving forward and backward probabilities scanned across the lattice), the marginalization still leads to computing all U × T × V logits of the lattice, a potentially memory-intensive step (it is computing Equation ( 3) over all pairs of indices as f joint (h i , g j )). The Aligner loss can be viewed as prescribing only the main diagonal of the lattice to be valid, reducing the realized logits to the bare minimum U × V .Savings come during inference, as well. The decoder in AED operates with O(U × T ) complexity, since it cross-attends to the entire encoder embedding sequence at every step, for U steps. Furthermore the constant factor is often relatively large in practice since a more powerful decoder network is needed for computing the attention (the recurrence relation isg i = f dec,AED (h, g i-1 , y i-1), compare at our Equation (2) which lacks h or even h i ). The RNN-T decoder has complexity O(T + U ), since it must emit a blank at every frame to advance to the end of the lattice in a addition to the steps emitting a label. In contrast, by preparing the alignment within the encoder, our model reduces decoder complexity to O(U ), the most efficient possible for auto-regressive token prediction. In practice our constant factor will also be small like RNN-T.One final savings relative to RNN-T comes when using beam search during inference, which can significantly improve overall accuracy. The emissions of blank tokens in RNN-T means that the same text hypothesis can be represented by different paths through the decoding lattice. Thus a proper search requires a routine to combine the probabilities from equivalent lattice paths at every step. Known as "path merging", it can actually be an expensive operation, leading others to sometimes approximate the search without it, despite the potential loss of beam diversity (e.g., see discussion in [26] ). Since our model, like AED, does not emit any blank tokens, path merging does not apply.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discusses a primary limitation we discovered relative to some previous models, which is inability to generalize to longer sequence lengths (some previous models share this limitation). Significant experiments were conducted to address this limitation, with good success but more could be done. Possibly other limitations exist which we did not explore thoroughly, such as: is more data required to train our model versus the previous models-none of our standard experimental settings showed an effect, but they also would not be considered low-resource datasets. We do not hide that our model is very slightly lower recognition performance than the best RNN-T models with the same settings in most of our experiments-we make no claims to achieve a new state-of-the-art in terms of WER. We have also not claimed to be able to operate our model in streaming fashion, which may be seen by some as a limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related works: 
 Before end-to-end ASR, the previous generation of "hybrid" ASR systems [27] relied on a separate base system to provide frame-level alignments which could then be used to train frame-level neural network acoustic models. Most modern ASR systems use an end-to-end framework; the most prominent modeling techniques are CTC [1] , RNN-T [2] , and AED [4, 5] .A number of works have aimed at modifying the base RNN-T model in order to gain efficiency or efficacy. In Monotonic RNN-T [28] , the loss and decoder are modified to step forward every frame, preventing multiple token emissions per label and speeding up the decoder complexity to O(T ). More recently, [12] proposed a method to reduce memory usage of the RNN-T loss, especially for cases with large vocabulary size such as Chinese character-based, by using a linearized joint network first to identify a relevant sub-region of the lattice likely to contain the true alignment. They only compute the full joint network on that sub-region and still train successfully. Another issue with RNN-T systems arises when trying to treat the prediction network as a language model, which can be improved through text-only training, because its language modeling function is polluted by having to predict blanks. To ameliorate this, multiple works [29] [30] [31] have introduced a separate network component for predicting blanks, and re-formulated the overall prediction model accordingly, resulting in much greater receptiveness to language training. One aim of our design was to avoid the use of blank tokens entirely, although in the present work we do not pursue additional text training.Monotonic alignments have also been developed for attention-based systems [32] , requiring newly devised loss functions to enforce monotonicity differentiably and encourage discreteness. Integrateand-fire systems are another approach proposed to limit the context window needed for cross-attention to the encoder embedding, by introducing a soft monotonic alignment mechanism [33] [34] [35] . It steps forwards through the encoder frames one at a time, accumulating weighted information from the embedding until an activation threshold is reached, upon which a token is emitted. CTC and attention-based systems have previously been combined [36, 37] resulting in improved learning and decoding, and it is not uncommon to train AED models with a CTC auxiliary loss. We did not pursue the analogous combination of non-AR and AR decoders for Aligners, although it would be simple to implement. Another interesting, concurrent line of work is the Bayes-CTC [38] and Bayes-Transducer [39] models, which add a modulating factor to the respective losses in order to encourage earlier token emission in decoding; however, only our simpler model achieves full text alignment with no blank embedding frames between tokens.A common way to improve the efficiency of ASR systems is to downsample the time dimension within the encoder, known as frame reduction (i.e., T < T ; [40] ). In addition to reducing encoder complexity, which scales as O(T 2 ) with self-attention, frame reduction yields decoder efficiencies. It was necessary in the original Listen, Attend Spell (LAS) model [5] to enable learnability for the cross-attention by presenting a manageable number of frames to the decoder. In RNN-T it reduces the number of decoder steps, which scale with T , and can in some circumstances be carried out to an extreme, packing multiple labels per frame [41] . In the few cases we tried, we found Aligners to have similar robustness to downsampling as RNN-T, but we did not study this in-depth. Aligners are like CTC in that they cannot downsample the encoder to fewer frames than the length of the text sequence, although perhaps they could be trained to decode multiple tokens per frame.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We conducted a range of experiments to explore the performance of Aligners and compare them against previous models on an equal footing. After describing the common configurations and datasets used, we present three different sets of experiments. The first explores the performance of basic Aligner models, the second studies techniques to employ at test-time to attain long-form recognition, and the final set examines the alignment process occurring within the encoders.We experiment on three U.S. English datasets with very different characteristics. The first is LibriSpeech-960 hour (LS) [42] . The second is a Voice Search (VS) dataset comprised of real traffic, totalling over 100M utterances and nearly 500k hours of speech, with an average duration of 3.4 seconds; utterances are anonymized before processing. The majority of the utterances are pseudo-labeled by a teacher model [43] , and a small portion are human transcribed. Only 5% of the queries are greater than 7.6 seconds long. The test sets for VS include a main set which covers many common types of utterances, and several rare-word sets generated using TTS from specific domains-maps, queries, and news. Lastly, we include a long-form dataset drawn from random YouTube videos (YT). The training set is comprised of utterances with pseudo-labels generated by a previous ASR system [43] , cleaned to ensure no speaker overlap occurs, and ranging from 5-15 seconds each. The total training set size is roughly 670K hours. The test set includes 30 hours of long-form audio, at an average of 8 minutes per example, spanning a range of topic sources including lifestyle, entertainment, sports, gaming, and society. In all datasets, the audio input is represented using log-mel features with a 32ms window size at a 10ms stride.We used the same neural encoder architecture for every algorithm, with settings adjusted for each dataset as listed in Table 1 . Our networks begin with learnable 2-D convolutional subsampling layers, with kernel size 3 and stride 2, 128 features in the first layer and 32 thereafter. We employ state-of-the-art Conformer encoders [20] , which include in each layer: a feed-forward unit, a multiheaded self-attention layer, a 1-D convolution layer for localized processing, followed by an outgoing feed-forward unit and finally residual connection. The number of layers and model dimension vary by dataset, as shown in the table. We did not find RNN-T nor Aligner experiments to be highly sensitive to decoder dimensions-they can operate with surprisingly small prediction networks-although results sometimes did improve slightly with larger prediction networks when also using a wider beam search. All our models use a word-piece tokenizer [44] , as is common in state-of-the-art systems. Word pieces may be as small as a single letter, or could be an entire word, such as "wednesday". Unlike phonemes, word pieces of a given vocabulary exhibit a wide range in duration, not to mention complexity, of audio content associated with them. Although this likely makes the alignment problem more difficult, word piece vocabularies are very effective because they cover the text distribution efficiently. For label smoothing, we estimate the word-piece prior on-the-fly using batch-wide label counts.To enhance the beam search, we apply label smoothing debiasing [45] , which essentially removes low-probability tokens from the posterior and then re-normalizes it. Similar to [45] , we find that a relatively large debiasing parameter of 2 (so the threshold becomes 2/V ) is helpful, and we use it with a modest beam size of 6 in our experiments for a small but consistent improvement in accuracy. Table 2 shows results in Voice Search, including the rare-word test sets, and Aligners perform comparably to RNN-T in all cases. We also compare CTC and non-AR Aligners, which perform nearly as well in Voice Search. We found that increasing the vocabulary size to 32K improved the performance of the non-AR Aligner, yet it still suffers from many deletions on the NEWS rare-word set, which contains longer utterances. Our non-AR model performed relatively poorly on LibriSpeech and YouTube. For sequences beyond a very short length, an auto-regressive decoder, however small, is essential for producing a high quality final output from the encoder embedding.In LibriSpeech, we report results comparing CTC, RNN-T, AED, and Aligners in Table 3 . All models used the same 17-layer conformer encoder architecture and dimensions, except for differences in relative position attention, described in the following paragraph. Performance of our Aligner models was remarkably close to RNN-T, matched or beat AED, and clearly beat CTC. For each model, we report the best score from a small number of runs and checkpoints. Full training settings are provided in the appendix, along with a brief commentary on the relative performance between RNN-T and AED. It is not always practical to train with examples as long as the desired usage, and for any given Aligner-Encoder architecture there is likely some maximum alignable length. So it may be necessary to perform recognition on utterances longer than a trained model's capability. Our baseline method for comparing long-form recognition is "blind segmenting": 1) cut the audio into fixed-length, nonoverlapping segments, 2) transcribe them separately, and 3) concatenate the resulting text segments. Perhaps the main shortcoming of blind segmenting is that recognition at the boundaries is prone to error, where the audio might cut in the middle of a word. One applicable solution is to use overlapping segments with an appropriate routine to stitch the hypothesis together in post-processing [48, 49] .Here, we instead describe how to improve continuity using the model itself, using only inference configurations which require no further training.In our approach, cutting and re-concatenating the sequence happens within the model-called "chunking" to distinguish from the baseline. We preserve some continuity by cutting the audio only after the 2-D convolutional feature layers. The chunks are processed independently in parallel through the conformer, after which they are re-concatenated before decoding. The decoder processes the full-length embedding sequence into the full transcription hypothesis, using awareness of the chunk boundaries. Within each chunk, the decoder ignores frames after the first <EOS> emission, and it resumes decoding at the beginning of the next chunk. Ideally, the decoder's prediction network will carry its state forward across the chunk boundaries to evenly incorporate all tokens.When we experimented with chunk sizes close to the training length, however, it led to increased deletions near the chunk ends. It was actually better to reset the prediction network state. This is understandable as the decoder was only ever trained to begin each utterance with a blank state, and it may be helping to count frames until <EOS>. The best performance, however, came from a middle approach: at the chunk boundary we reset the prediction network state but then "prime" it by re-processing the last several tokens through it. We found state-priming to reduce the number of errors at the boundary without raising later deletions. For testing, we turn to the YouTube domain, where the training utterances ranged from 5-15 seconds long but test utterances spanned several minutes. Table 4 compares results from RNN-T and the Aligner-Encoder. They achieved equal WER (7.6%) with a 15-second blind segmenter, due to equivalent errors at the boundaries. In the unsegmented arrangement, the RNN-T, with local attention of width 256, generalized well to the full utterance length; its WER improved to 6.8%3 . Inside the Aligner, we applied a chunking period of 14 seconds (176 embedding frames). We found it best to reset the prediction network at the chunk boundaries and prime it with 10 tokens of history, which achieved 7.3% WER (7.5% without state-priming). While falling short of the RNN-T in this case, the Aligner did improve performance by smoothing the chunk boundaries, even without having been trained to do so. This result demonstrates our model's capability for long-form recognition, and could possibly be improved, such as by introducing additional training for this mode.To study how the encoder predicts the alignment, we examine the self-attention probabilities it computes during the forward pass. Figure 2 shows self-attention probabilities (every row sums to one) from the same head at different layers within one of our 17-layer encoders trained on LibriSpeech.A striking pattern emerges. One might expect the alignment to happen gradually in a network replete with residual connections, and indeed the self-attention in layer 4 is highly diagonal along the sequence; its output positions (vertical) draw from very near their own position from the input (horizontal). By layer 13, however, this pattern changes completely, and information concentrated in the front of the sequence appears to be distributed broadly. Suddenly, in the next two layers, the audio-to-text alignment is clearly visible. The label contains 12 word-pieces, and precisely that many output positions receive information from points distributed monotonically along the inputs to these layers. The remaining output positions are possibly populated with filler values-these positions in the final output embedding receive no training. The alignment is complete even before the final layer, where an unrelated distribution is seen. In this network, all the attention heads showed the same alignment operation happening in the 14th and 15th layers, for every input example we observed. Early to middle layers performed different mixtures of diagonal (local) and non-diagonal (global) attentions for different heads, which were less interpretable. Overall, the self-attention layers appear to be primarily performing audio encoding roughly inplace for many layers, and then they very explicitly perform the full alignment within as little as two layers. Interestingly, positions near the beginning and end of the sequence are perhaps used as margins for bookkeeping, where the training utterances often contain silence anyway.The alignment itself is useful in many applications, and can be estimated from token emission positions when decoding with RNN-T. Our model has obscured that information, but the observed behavior affords the possibility of extracting alignments during inference. The bottom subplot of Figure 3 shows the average of all attention heads in Layer 15. Seen side by side, they closely track the RNN-T alignment, which is likely close to ground truth.To further study the alignment process, we also examined the intermediate embeddings themselves by the following procedure. Given the converged Aligner-Encoder model, we trained an RNN-T model of the same size, which is randomly initialized except that the parameter values from the beginning layers of the Aligner are used in the corresponding layers of the new encoder. The Aligner weights were kept frozen, so that the portion of the model trained by RNN-T receives as input the intermediate Aligner embedding. Finally, we could measure alignments through the RNN-T decoder as usual-we computed the full forward-backward probabilities of the decoding lattice. By repeating this procedure using different numbers of Aligner-Encoder layers as the foundation, we traced the alignment progression of the internal representation through the forward pass of the original model.The result is shown in Figure 3 , where the suddenness of the process is apparent. Within a single layer the representation shifts from its original layout to become fully front-aligned with the output. This is the same layer that exhibited the most pronounced alignment self-attention probabilities previously. In the figure, each subplot is independently normalized, and the decoding lattice probabilities (computed from the forward and backward passes) are re-computed using a high temperature to exaggerate the tails of the distributions, which are tightly peaked. The bottom subplot shows the self-attention probabilities averaged over all heads in Layer 15, normalized by row but not temperature-adjusted.The hot spots track closely with the original RNN-T alignment-note the lattice calculation fills in probability mass where the blank token is to be emitted, whereas the Aligner self-attention ignores those segments. The close correspondence of the pauses is especially visible. Using the same diagnostic technique, we can examine the failure mode of poor length generalization. Figure 4 shows such a case, for an utterance that is 1.5x longer than any training example. The model is only able to align a portion of the utterance, explaining why sometimes many deletions occur from dropping the latter part of the text. Interestingly, the RNN-T model trained atop the 15 frozen Aligner-Encoder layers (as in the previous subsection) still assigns some probability to the remainder of the label sequence, in a somewhat aligned fashion but pushed to the tail of the embedding sequence. This part is not decoded successfully. As we increased the number of Aligner layers used, we observed that the RNN-T hypotheses began to lose tail words several layers prior to the apparent alignment layer.Along with the self-attention visualization of Figure 2 , this supports the conclusion that the Aligner works for several layers to prepare for the alignment operation; the over-length sequences already begin to be disrupted earlier. Separately, we note that for sequences which are near the length-capacity of the all-Aligner model, when deletions begin to occur, we observed them to often happen somewhere in the middle. We conducted one final experiment to demonstrate the likely applicability of Aligner-Encoders to tasks other than ASR. Specifically, we desired to show that our model has greater flexibility in aligning information than only the monotonic and increasing alignment relationship of standard ASR. To do so, we constructed the extreme case of decreasing alignment by completely reversing the audio input, so frames at the end of the audio input correspond to the beginning of the text output, and vice versa. Experimenting in LibriSpeech, the resulting model achieved similar performance as the regular model on utterances within the training length. Further, the selfattention weights exhibited the same behavior as in the forward model, except that in the aligning layer the weights trace from the upper-left to lower-right, showing the sequence reversal in action (compare against the bottom panel of Figure 3 ). We include alignment plots for this model in the appendix, Figure 5 . The equal ability to completely reverse the order of the information in the encoder embedding strongly suggests that our model could perform non-monotonic tasks, as well, which require more varied information re-ordering. Two prominent non-monotonic tasks where Aligner-Encoders have potential to simplify modeling include machine translation (MT) (e.g. [50] ) and automatic speech translation (AST) (e.g. [51] [52] [53] [54] ).Considering training speed, memory usage, and inference speed, we observed favorable comparisons for our model owing to its reduced complexity. While the exact numbers will depend on many hardware and implementation details, we present a representative case using our 100M-parameter encoder LibriSpeech models in Table 5 . Notably, the RNN-T spent roughly 290ms per training step in the decoder and loss (forward + backward propagation), whereas our model required only 29ms, a 10x gain. Our RNN-T loss implementation is highly optimized, so the majority of the gains came from eliminating the scan associated with the T dimension of the joint network output. The peak memory usage for our model decreased by 18% (by 1.4G per accelerator) 4 . The 4-layer AED decoder trained with a step time essentially as fast as ours, thanks to good parallelization of transformers, and did tend to converge in significantly fewer steps (see A.2). The inference speed of our model, however, was by far the fastest, especially relative to AED. We profiled all models using 11.5 seconds of audio, a batch size of 8, and beam search of size 6 (effective decoder batch size 48). The decode step for the Aligner and RNN-T 5 (LSTM + Joint network) measured a mere 190µs. As discussed earlier, we can estimate the total decode time as the step time multiplied by the text length (U ) for our model, versus multiplying by the sum of the audio and text lengths (T + U ) for the RNN-T, resulting in substantial savings. We approximate with T = 300, U = 100. Turning to AED, each step through the transformer decoder was almost two orders of magnitude slower, at 8.5ms (and U steps are executed). 6 The forward pass itself required 5ms per step, and re-ordering the decoder state as part of the beam search occupied the other 3.5ms. The re-ordering time was negligible in our model due to the minuscule LSTM state. Including the encoder, the total inference time of our model is estimated at 2x faster than RNN-T and 16x faster than AED in this scenario.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide a complete description of the model, and make an effort to convey the relevant settings for training and evaluation (see appendices). Furthermore, we strove to change settings as little as possible from those employed with previous models, showing that it should likely be relatively straightforward to adopt the new model. Lastly, one of the main benefits of the new model is its simplicity-since it uses standard deep learning components and removes complications relative to previous models, the description should suffice for reproduction. 
conclusion: 
 We have shown that transformer-based encoders are able to perform audio-to-text sequence alignment internally during a forward pass, using only self-attention (i.e., prior to decoding, unlike previous models). This finding enables the use of a much simpler ASR model, the Aligner-Encoder, which has a simple training loss and achieves lower decoding complexity than past models. A key design point of our model is to keep the benefit of auto-regressive modeling while removing as much computation as possible from the auto-regressive loop. Desirable extensions to our model for ASR that would require further study include: use in streaming recognition, ability for multilingual recognition, and incorporation of traditional pretrained encoders, to name a few. Fusion with an external language model [24] could be simplified relative to RNN-T owing to the absence of blank tokens, and for the same reason our model may be more receptive to training on other losses such as from text-only data.Looking beyond ASR, application to machine (text) translation would require some modification to permit output sequences longer than the input, although speech translation would not. Altogether, Aligner-Encoders and their newly identified ability to learn self-transduction present promising opportunities for future research and application.A Appendix / supplemental material A.1 LibriSpeech -Expanded Settings and Results  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: While historically AED has out-performed RNN-T on LibriSpeech, this appears to have changed with the introduction of the Conformer [20] . To this point, there are several relevant factors worth noting for our case. Both models receive positional encoding following the feature convolution layers. Our RNN-T is trained and operated in a non-streaming mode (e.g., with non-causal attention). We trained AED with label smoothing (weight of 0.1) to prevent overfitting. Our AED models did often converge much faster, sometimes in as few as 25k training steps; we conducted a small hyperparameter search over learning rate and schedule to ensure against overfitting. Lastly, our AED decoder is a 4-layer transformer with 18M parameters (total model parameters 128M), which is already significantly larger than the 3.5M-parameter LSTM decoder for the RNN-T. A 148M-parameter conformer AED was reported in [55] to achieve 2.16% and 4.74% on Test-Clean and Test-Other, respectively, which is better than our smaller AED baseline but still less good than our RNN-T, which we keep as the relevant SOTA for comparison.A.3 Reverse Alignment Experiment Figure NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We include the <EOS> in the label count U , without loss of generality.It is not automatically the case that RNN-T models exhibit good length generalization; it is by some combination of deliberate preparation and chance that our cases do.Total training time was roughly a day and a half.We disabled path merging for RNN-T.Our implementation included both self-attention and cross-attention in all layers, and a decode cache. 


Decision: This paper introduces the Aligner, a simple yet efficient end-to-end (E2E) model that merges the benefits of RNN-Transducer (RNN-T) and Attention-based encoder-decoder (AED) for automatic speech recognition (ASR). The authors found that the transformer-based encoder is sufficient to handle alignment between speech and text. Leveraging this finding, they utilized the proposed aligner encoder to reposition relevant information to the start of the embedding sequence, enabling them to employ frame-wise cross-entropy loss from AED during training instead of relying on the intricate dynamic programming required by RNN-T, thereby simplifying the training process significantly. During inference, the Aligner demonstrates reduced decoder complexity compared to AED. The new model achieves recognition accuracy similar to that of popular AED or RNN-T models.  

This paper presents a novel approach that is poised to make a significant impact on the speech recognition community. AED and RNN models currently dominate E2E models in ASR, and although numerous new architectures have been proposed recently, none has achieved the success of the Aligner introduced in this paper. It is impressive how this new model seamlessly integrates the strengths of AED and RNN-T models. The efficacy of the proposed method has been demonstrated across three distinct tasks: Librispeech from the public domain, an in-house voice search, and YouTube test sets, thereby reinforcing the credibility of the results. 

The reviewers unanimously agreed this is a high-quality paper and recommended it for acceptance. In the rebuttal, the authors addressed most of the reviewers' questions. Several reviewers questioned why RNN-T outperforms AED in this study, given that AED is generally seen as more powerful while RNN-T is better for streaming. The authors explained that both models have an identical speech encoder setup and cited the Conformer-Transducer paper to support their argument. However, Table 1 in "A Comparison of Sequence-to-Sequence Models for Speech Recognition" shows RNN-AED outperforming RNN-T. I believe the performance difference between AED and RNN-T architectures should be less dependent on the speech encoder, whether it's a Transformer or LSTM. 

In brief, this is an excellent paper featuring a new E2E architecture that merges the strengths of AED and RNN-T. It has the potential to significantly influence the speech community.

Reviews: review: paper_summary: paper_summary: A new simplified encoder-decoder model is presented without the attention. The decoder is generating the labels auto-regressively as usual until end-of-sentence (EOS). In contrast to attention-based encoder-decoder (AED) models, the attention is replaced by simply taking the same frame in the encoder - i.e. in decoder step u, it will take frame u also from the encoder. Thus the output sequence can never be longer than the input sequence. The idea is that the encoder with self-attention can already realign the information as necessary to output it label by label. The remaining encoder output frames after EOS are ignored, but of course all intermediate encoder frames are used due to self-attention.

It's an interesting test to see whether the self-attention is enough to already learn this. And the answer is yes, it can learn this.

Experiments are performed on three speech recognition tasks:

* Librispeech with 960h train data
* Voice Search with 500kh train data
* YouTube videos with 670kh train data

In all cases, a Conformer encoder is used. Word pieces are used as output labels.


The self-attention weights are analyzed and it is observed that the realignment happens in layer 14. This is also verified in another way, by freezing the first N layers of the encoder, randomly resetting the other encoder parameters, and adding a RNNT on top of it and training that, and then generating the RNNT soft alignment. With N>14, the alignment looks like the identity mapping.
strength_weakness: strength_weakness: No source code to reproduce the results?

No references are given to Voice Search and YouTube data, so it's impossible to reproduce and verify the results.
questions: questions: How is the convergence rate in comparison to CTC, RNNT, AED? How is the alignment behavior early in training?

Table 3, what is dev, is that dev-clean, dev-other, or both combined?

Table 3, it seems a bit weird to me that the RNN-T is so much better than the AED model. I would actually expect the opposite. For example:

* A Comparison of Sequence-to-Sequence Models for Speech Recognition,
https://www.isca-archive.org/interspeech_2017/prabhavalkar17_interspeech.html
* On the Comparison of Popular End-to-End Models for Large Scale Speech
Recognition, https://www.microsoft.com/en-us/research/uploads/prod/2020/11/template-5fa34dc776e7f.pdf
Both show that AED is better than RNN-T. And this is what I have seen on many other occasions as well. Was this expected to you? Why? Or if not, how do you explain it?

What happens when f_pred is just a feed-forward network without the recurrence (dependence on g_{i-1}), i.e. you would get a model with only the last label as context? For RNN-T, it has been shown that this performs equally well as using the whole history. It would be interesting to see how it behaves for this model. (For an AED model, this is not really possible because the cross attention mechanism needs it.)
limitations: limitations: -
paper_summary: paper_summary: This paper proposes a new speech recognition (or, more generally, sequence-to-sequence) architecture. This architecture performs an alignment process between input and output features via self-attention mechanisms in an encoder. The decoder network is a simplified version of the combination of the RNN-T or AED decoder, and its relationship is detailed in Section 2.2. The primary advantage of this method is reducing the computational cost by avoiding the dynamic programming to adjust the input and output length in RNN-T or cross-attention to consider all possible scores across the input frame and output token. The experiments show the comparable performance of the proposed method to RNN-T and AED, but it significantly reduces the computational costs.
strength_weakness: strength_weakness: - Weak reproducibility due to the use of non-public data for main ASR experiments and the lack of source code release. Note that I did not penalize this point in my initial judgment. But if this part is improved, I'll raise my score.
questions: questions: - Section 2.1: Do you need $U \leq T$? Can we apply this method for $T \leq U$? This would happen in the general sequence-to-sequence problem like MT or TTS.
- Section 4.2: Did you try it with the other encoder layer than the Conformer (e.g., vanilla transformer)? I'm curious because the alignment properties of this method might depend on the convolution operation in the Conformer.
- Section 4.5.2: I'm not sure how each attention head behaves. Can you discuss a bit more about how this behavior is different across the head?
- Section 4.5: Do you have some results on MT or AST?

Suggestions
- I recommend the authors emphasize the practical benefit of the computational complexity of this method in the abstract.
- Sections 4.5.3 and 4.5.5: These sections provide the main experimental results in the appendix, which are not recommended. This way, it breaks the page limit rule and is unfair compared with the other papers, which put all main results in the main body. These sections should be rewritten to avoid using the appendix results. Note that some supplemental use of the appendix is no problem (e.g., Table 5 is a good example. This is too detailed and may not be so crucial for understanding the main idea of this paper, but it is essential for reproducibility. So, it is adequate to be located in an appendix section.).
limitations: limitations: - Since this is not based on a hard alignment approach (RNN-T and CTC are based on hard alignment), I'm curious about the hallucination issues often observed in AED or decoder-only architectures. For example, OpenAI's Whisper is based on AED (soft alignment). It has a serious hallucination issue (despite its outstanding performance), and I think this is a potential limitation of the soft alignment-based approaches in general. I want the authors to discuss this aspect. This method probably has an advantage over AED due to its shallow decoder architecture, but I'm not very sure.
paper_summary: paper_summary: The paper introduces Aligner, which is to take the best parts from RNN-Transducer and AED (Attention Encoder-Decoder) models. The idea comes from the intuition that the transformer encoder with self-attention can already learns to align the input and the output -- which is explicitly modeled by previous approaches like applying techniques to ensure the monotonic alignment or doing dynamic programming to find the alignment. Aligner just simply train with cross-entropy loss, and the results surprisingly show that the encoder can internally learn to align the input and the output during the forward pass. Experimental results showed that Aligner can perform comparably with existing baselines, and show the computational efficiencies.
strength_weakness: strength_weakness: - The idea is yet confirmed in the model fine-tuning setups; as recent Whisper model [1] shows that large-scale pretraining can leads to performant ASR models, I hope this Aligner work applied to the large-scale pretraining setups and show the effectiveness as well. Note that I feel this is not a critical weakness of this paper.
- Honestly, I am not following the most recent state-of-the-art models for ASR systems, and not sure if there are some missing baselines that the authors should also compare with. I am willing to listen to other reviewers' opinions about this.

[1] Robust Speech Recognition via Large-Scale Weak Supervision, Radford et al, 2022
questions: questions: No questions for now.
limitations: limitations: Limitations are adequately addressed.
paper_summary: paper_summary: This paper proposes a new ASR model that connects a self-aligned encoder and the light text-only recurrence of RNN-T. The proposed model can be trained with label-wise cross entropy loss, which is computationally efficient than RNN-T training. The authors show the limitation of the model to inference on long-form audios and give a special inference configuration to mitigate the issue. Experiments on Librispeech and larger-scale ASR datasets demonstrated the close performance compared to other ASR models. The authors also show the audio-text alignment in the self-attention weights of a certain layer, which could be said to perform “self-transduction”.
strength_weakness: strength_weakness: The major concerns are the presentation and the limitation of the proposed model. 
1. The explanation of the aligner modification for long-form audio is somewhat hard to follow. It may be better to have figures, or equations involved in the paragraph.
2. As the claim of the paper is that Aligner encoder achieves better efficiency than RNN-T and AED, it would be better to summarize the experimental numbers in Section 4.6 into a table for reader's convenience. People might be interested in those strong numbers.
3. The proposed model may have limited use cases e.g. offline ASR because the encoder cannot be implemented for streaming with the current design.
4. A related work that replaces RNN-T loss with the cross-entropy loss and reduces the GPU memory usage. (https://arxiv.org/pdf/2307.14132).
questions: questions: Several additional questions:

1. If I remember correctly, the AED model can achieve no worse WER than RNN-T on Librispeech in literature, I am wondering did the authors look into other reasons why AED in Table 3 is much worse than RNN-T (except the long-form problem)?
2. Any results comparing rotary positional embedding and relative positional embedding for speech encoder on long-form audio? The authors mentioned that RNN-T used RPE, is it the reason why RNN-T is good at long-form audio?
limitations: limitations: The main limitation is the use case of the proposed method, aka, non-streaming applications. Even though, the method can be helpful for many different purposes.

Comments: title: AED vs RNN-T
AED vs RNNT
Reply to the rebuttal
Thanks for your explanation
Discussion period starts
comment: Thank you!  Indeed the AED vs RNN-T performance seems to be a valuable question, we're not sure we'll be able to answer it definitively, but will add more details and some discussion (please also see comments under reviewer JgvT).  In our experiments we used the same encoder for both, including global attention--so no streaming, which as you mentioned might otherwise put RNN-T at a disadvantage.  It might simply be that AED requires a larger decoder to do really well, where we only used a 4-layer, 18M parameter transformer (already much larger than the LSTM used for RNN-T, 3.5M parameters).
Good questions, on AED versus RNN-T, we used the same encoder architecture for each.  In fact they both include an absolute positional encoding, added into the embedding after the initial 2-D convolution layers, prior to the first conformer layer (we need to add this to Table 5)--some early ablations showed this might not be critical but I think we didn't try again with all the other final settings to be sure.  One possible difference is that the variational noise is applied to the LSTM and the text embedding variable in RNN-T, and this is helpful to get a last bit of performance improvement, whereas in AED we only applied it to the embedding variable, since the decoder is much bigger.  Another difference is that in AED we needed label-smoothing, but this doesn't apply in RNN-T.  In both models we gave the encoder global attention, so it can operate on the whole sequence (no streaming).  It is a bit strange that for our AED to work on the longest test utterances, we needed to concatenate training examples (as we described), and we think it is generally known that AED struggles to generalize to longer lengths, but the other references we've found don't mention this.  Separately from this, it's possible that to perform better AED simply needs a larger decoder, since we used only a 4-layer transformer, which already adds 18M parameters.

It's interesting that multiple reviewers have raised this question--seems worthwhile for us to add a short discussion about this.  Thank you.
Thank you, and I will keep my rating as the score is already 7.
Thanks for the rebuttal. The paper can benefit from the revisions proposed by the authors. I am willing to raise the score to 7 (Accept).

In terms of the AED vs. RNN-T, it would be great to clarify the settings and explain the gap. My impression is that RNN-T typically uses a streaming encoder for online purpose, while AED uses non-streaming decoder for offline ASR. AED would always be better in terms of WER. Maybe the authors used the same decoder, and I didn't check the paper again. The authors can use several sentences to clarify the gaps in the revised version of the paper.
The answers are valuable (especially for the head and hallucination discussions), but they are not related to the overall discussions. I already gave the accept score and I want to maintain it.

I'm looking forward to MT or AST experiments and the open-source implementation of this method.
Thanks for the rebuttal.

On AED vs RNN-T: Do you have exactly the same hyper-parameters for the encoder in both cases? Or were they tuned individually?

I guess in the original Conformer paper, both the architecture and its hyper-parameters were optimized always using the same RNN-T sequence modelling on top. So this gives maybe an advantage for RNN-T over AED.

RNN-T and CTC don't need any further positional information in the encoder output, while AED needs some information for the cross attention to work properly, such that it can know where to attend next. The architecture can indirectly learn absolute positional information, e.g. via the convolutional padding, but you can imagine that this is maybe not optimal. So maybe a different frontend, or explicitly adding absolute positional encoding would greatly help the AED sequence model.

Just some thoughts on this. My intuition tells me that AED is still more powerful than RNN-T when this is taken into account. And/or when the architecture and/or frontend is tuned for AED.

But studying this is probably out-of-scope for this work. Maybe the only reasonable simple experiment you could do now is adding absolute positional encoding to the encoder.
Dear Reviewers,

The authors have already posted their rebuttal to address review concerns. It is now the discussion period (Aug. 7 - 13). Please read the author response, and conduct the discussion. Thank you!

Thanks,
AC