Global Normalization for Streaming Speech Recognition in a Modular Framework
Authors: Ehsan Variani
Ke Wu
Michael Riley
David Rybach
Matt Shannon
Cyril Allauzen
Venue: NeurIPS 2022 Accept (2022)

Abstract: We introduce the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition. Our solution admits a tractable exact computation of the denominator for the sequence-level normalization. Through theoretical and empirical results, we demonstrate that by switching to a globally normalized model, the word error rate gap between streaming and non-streaming speech-recognition models can be greatly reduced (by more than 50% on the Librispeech dataset). This model is developed in a modular framework which encompasses all the common neural speech recognition models. The modularity of this framework enables controlled comparison of modelling choices and creation of new models. A JAX implementation of our models has been open sourced.

Global Normalization for Streaming Speech Recognition in a Modular Framework
Ehsan Variani, Ke Wu, Michael Riley, David Rybach, Matt Shannon, Cyril Allauzen
Google Research
{variani, w riley, rybach, mattshannon,, allauzen) @ google.com
Abstract
We introduce the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition. Our solution admits tractable exact computation of the denominator for the sequence-level normalization. Through theoretical and empirical results, we demonstrate that by switching to globally normalized model, the word error rate gap between streaming and non-streaming speech-recognition models can be greatly reduced (by more than 50% on the Librispeech dataset). This model is developed in modular framework which encompasses all the common neural speech recognition models. The modularity of this framework enables controlled comparison of modelling choices and creation of new models. A JAX implementation of our models has been open sourced."
Introduction
Deep neural network models have been tremendously successful in the field of automatic speech recognition (ASR). Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture [19], connectionist temporal classification (CTC) models [14] with recurrent architectures such as long short-term memory (LSTM) [20], and more recently sequence-to-see -o-suuecce (Seq2Seq) models like listen, attend and spell (LAS) [8], recurrent neural network transducer (RNN-T) [13], and hybrid autoregressive transducer (HAT) [40]. When configured in non-streaming mode, these neural ASR models have reached state-of-the-ar word error rate (WER) on many tasks. However, the WER significantly worsens when they are operating in streaming mode. In this paper, argue that main
of this WER gap is that all the existing models are constrained to be locally normalized which makes them susceptible to the label bias problem [36, 2, 24, 4]. To address this problem, we introduce new category of globally normalized models called Globally Normalized Autoregressive Transducer (GNAT). Our contributions are:
(1) Addressing the label bias problem in streaming ASR through global normalization that significantly closes more than 50% of the WER gap between streaming and non -trestremming ASR. algorithms for the exact computation of the global normal- ization under the finite context assumption.
(2) Efficient, accelerator-friendd
(3) A modular framework for neural ASR which encompasses all the common models (CE, CTC, LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalized counterparts.
httpss//ggitabb.co//gggeeee hotp.///eeeggggo-eeeeeecchc////e
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

2 Streaming Speech Recognition
For an input feature sequence x = 21 FT, usually represented as a sequence of real valued feature vectors (such as log mel), and finite output alphabet Σ, we wish to predict the correspondigg output label sequence y = 1/1 yu, Yi E E. We call each element z; in x a frame, and each element Y; in y an output label. Common ASR models do not directly predict y, but rather aligument label sequence z = 21 2V, & E EUA A. A is a finite alphabet of control labels, such as the blank the end-of-sequence label in LAS. There is deterministic mapping B(z)= (EUA)* + Ex for obtaining y from z (e.g. in RNN-T, we simply remove all the blank labels from z). An ASR model can then be broken down two tasks: (1) assigning a score II ( |x, z<s) to each alignment sequence z, where w w(z X, Zzi) > 0 is the alignment score of predicting a single alignment label; (2) linding (usually approximately) arg maxy Eu(B(r)yy II, wZiix, Zci).
label in CTC or RNN-T,
non-streaming ASR model's alignment score w [x, z<s) has access to the entire x for any i. In contrast, a streaming model's alignment score takes the form (Z)(xx<H))) z it only has access a prefix X <() of the input feature sequence, where t(i) is the frame to which z; is aligned. Streaming models can be seen as special case of non-streaming models with respect the alignment scores.
All the common neural ASR models use a locally normalized alignment score which satisfies the constraint Σsesus ((2; 2/x, i = 1. This is achieved by applying the softmax function to the last layer activations. The local normalization constraint makes w 2 x, Aez)) easily interpretable as a conditional probability distribution (a,2x, zzz), and thus II. ws x, Zei) easily interpretable as P. (z))). With samples from the true data distribution P(x,)) the modeling parameters are optimized by minimizing the negative coonditional-likeiioon loss Ep(xy)) !-rxx.y)_-oo...... log Pa (y|x)] = (P(xy)l-be log E (z]x)]]
2.1 Label Bias in Streaming ASR
For non-streaming. locally normalized model, the negative log-conndtiionallikllhhood loss is mini- mized in the limit (assuming enough model capacity) by setting w(z;)x, zez) to the true conditional probability P(=:|x, zei), leading to [[, zez) being equal to the true posterior P(z|x).
For a streaming model, is replaced by X<(()) in the alignment score (e.g. by using a unidi- rectional encoder). Here the negative log-conditionaal-ikeiiioo by setting w(2 - to P(z (x<(()) Zc)). As a result, the product II, (Zz][xxut)) Zes)
loss is minimized in the limit
II: P(= (=(0)x R<i) is in general not equal P(z) x) anymore. In other words, using a streaming locally normalized model means that the estimated alignment sequence posterior is the product of some locally normalized alignment scores which depend only on partial input X<t(t)- and as result can no longer accurately represent the true conditional distribution. This will bias the model towards predictions with low-entropy estimated posterior probabilities at each decoding step. This degrades the model ability to revise previous decisions, a phenomenon called label bias [24, 2].
2.2 Global Normalization
Traditionally, globally normalized models such as Conditional Random Fields (CRF) [24] are used to address the label bias problem. This paper seeks apply global normalization modern neural architectures that are more similar to CTC, RNN-T, or LAS, rather than traditional linear models, with the purpose of addressing label bias problem for streaming ASR models.
A globally normalized model does not constrain the alignment score w(2,/x, z<s) to be locally normalized; it only requires it to be any non-negative score, as long as the denominator Z(x) as a conditional probability distribution (z)x).). Minimum negative log-conditionall-ikee likelhoood training can be more expensive for globally normalized models due to the need to compute Z(x) and the corresponding gradients. However with our proposed modular framework, globally normalized model training can be made practical with careful modelling choices on modern hardware.
Σx II. [2, 2xs) is finite. The finite denominator allows us to interpret I. (2.lxxx22)))
2(x)
Any locally normalized model is by definition also globally normalized, whose 7(x) 1. While the reverse not true general, Appendix shows that under non-streaming settings with a mild condition, locally and globally normalized models actually express the same class of conditional

---

distributions. Based on this observation, we argue that under non- streaming settings, with adequately powerful neural architectures, maximum log- waa-cooniioaa-iieiiliie similar locally or globally normalized models, and thus similar WERs in testing. Results from [37, 17] and our own experiments in Section validate this.
training should yield behaviorly
3 A Modular Framework for Neural ASR
In this section, we introduce a modular framework for neural ASR, using the weighted finite state automaton (WFSA) formalism calculate the conditional probabilities via alignment scores. The modular framework clearly expresses the modelling choices and enables practical globlally normal- ized model training and inference. We use the WFSA formalism as the language for describing our framework because of its succinctness and precision. Our algorithms are implemented from scratch to run on TPUs, without using existing toolkits such OpenEst [1], Kaldi [33] or K2 [21].
3.1 Preliminaries
We begin with an introduction to the relevant concepts and notations.
semiring (K, & 8,,0, 0, 1) consists of a set IK together with an associative and commutative operation @ and an associative operation @, with respective identities 0 and I, such that @ distributes over tD, and 0&z = r80 = O. The real semiring (R++, +, x, 0, 1) is used when the weights represent probabilities. The log semiring (IR U {-o0), Blog +, -o, 0), isomorphic to the real semiring via the log mapping, often used in practice for numerical stability. 2 The tropical semiring (RR U [-o),, max, +, -0,00 0) is often used in shortest-path applications.
A weighted finite-state automaton (WFSA) A (E, Q, F. E) over semiring IK is specified by finite alphabet Σ, a finite set of states Q. an initial state E Q. a set of final states C Q, a final state weight assignment p: F -> IK, and finite set of transitions F Q x (E)( K Q (e denotes the empty label sequence). Given a transition e E E, píe] denotes its origin or previous state, n|e] its destination next state, ofc] its label, and w[e] its weight. A path π c1 .....r Cg is sequence of consecutive transitions e; E E: nfe:-]] = ple;], 8= -.... k. The functions n. p. and w on transitions can be extended paths by setting: n n|[]] - n1(ea] and p|ž] p[e1] and by defining the weight of path as the 8-product of the weights of its constituent transitions: w|r]] = w[e1] 1 8 .... @ w(ex]). An i, F, E) is simply WFSA whose transitions unweighted finite-state automaton (FSA) A = (E, Q, and final states are all weighted by I.
IIQ Q2) is the set of all paths from subset Q1 S Q to subset Q2 S. Q. II(Q1,y, Q2) is the subset of all paths of [T Q2) with label sequence = y1 yus Yi E E. A path in successfil. The weight associated by WFSA A to any label
[[()). F) is said to be accepting sequence y is given by A(y) = =eu((()) w[x] @ p(n(T)). The weight of A is the B-sum of weights of all accepting paths W (AA) = + ()((())) w w(T]] @ p(n[x)). For semiring K. where @ is also commutative, the intersection (or Hadamard product) of two WFSA A, and Ag is defined as: (Ai n (y) = (y & Az(y). [28] gives an algorithm to compute the intersection. We can view y WFSA that accepts only y with weight I, then A(y) W(Any).
3.2 Probabilistic Modeling and Inference on Acyelic Recognition Lattices
For any feature sequence T1 -.. .r., Fr,, we model with trainable parameters 0 to induce recog- nition lattice WFSA Ap,x = (I., Q0.x Fo.x: P0,x, Fxg,)) For a label sequence = y1 -i...yu yy, the weight associated with it, A9,((y), is interpreted under the log semiring as the unnormalized exp(Agx(y) (xp(W(ee.x)))
negative log conditional probability (o x) = (Ae,
exp(W
The recognition lattice Ag is designed to be acyclic, and therefore the weight of the automata in both the numerator and denominator above can be efficiently computed by visiting the states of the corresponding WFSA in topological order [27]. See Appendix C for our accelerator friendly version of this algorithm. We can thus train an ASR model by minimizing the negative log-conditional- likelihood on the training corpus D, and choosing 0* arg ming Exx log( (o x))] arg ming E(x.))DD(((A)) (Ag.x) - (A4.x Ny)].
aa Plog b- log(c" +e²)
3

---

In general, there can be more than one path in Ag.x that accepts the same y. During inference, finding the optimal y arg max, Po (y|x) requires running the potentially expensive WFSA disambiguation algorithm [30] on Aa,x. As cheaper approximation, we instead look for the shortest path स in A0 under the tropical semiring, and use the corresponding label sequence as the prediction, again using the standard shortest path algorithm for an acyclic WFSA [27].
3.3 Inducing the WFSA
Our framework decomposes the sequence prediction task in ASR into three components, each playing a specific role in inducing the recognition lattice Ag.x-
The context dependency FSA C (E, Qc. Fc, Ec) is e-free, unweighted FSA, whose states encode the history of the label sequence produced so far. C is fixed for a given model, independent of input
The aligument lattice FSA Lr = (E,Qr, br, Fr. Er) is an acyclic, unweighted FSA, whose states encode the alignment between input frames x and output labels y. Ly depends on only the length T of input x.
The weight function wa,x QT Qc (EUe) K. wax is the only component that contains trainable parameters and requires full access to x. This function defines the transition weights in the recognition lattice A0,x-
We will discuss how one can define these components in detail in the next section. With (C, LT, wgx)) given, the recognition lattice A is defined as follows:
Qa.x QT Qc
ig.x = (ir,ic)
Fø.x
FT x Fe
Ao..
((gar qe); M1 we x(gx ges v), (da d())
y E E, (4a, d'dd E Er, (ges v. do) E Ec
u (11a, e, e, wax los e), (Lar yo))
(qas e, e(a) E ET, qe E Qc}
PAo.x (4) = Vq Fø.x
In other words, the topology (states and unweighted transitions) of the recognition lattice Ag,x is the same as the FSA intersection LrnC (see Figure Ic for concrete example); and the transition weights are defined using wg.x. The e-frenness of C and the acyclicity of Lr implies that the recognition lattice Ag, is also acyclic.
4
Components of a GNAT Model
In this section, we define globally normalized autoregressive Iransducer (GNAT) through the frame- work above, by specifying each model component.
4.1 Context Dependency
GNAT uses an n-gram context-dependency defined by C, = (E, Qn Fn, E.), where Qn = S corresponds to a label history of length up to The initial state in = is the empty label sequence. The transitions E, C Qn x x Qn correspond to truncated concatenation: E, = {q, y. q') | E Q,y E } where q is the suffix of qy with length at most n. For example, when = 2, the transition (ab, c, bc) goes from state ab to state be with label c. All states are final: F, = Qn. See Figure la for the FSA C1 when Σ- {a, b}. Appendix C demonstrates how Cn be efficiently intersected during the shortest distance computation thanks to the highly regular structure of C.,.
Although not studied in this paper, our modular framework makes it easy to switch to more sophisticated context dependency, such as clustered histories often used for dontext-dependen phone models, or variable context length used in n-gram language models [29].

---

(a) n-gram context-dependency automaton C₁ for
(b) Frame dependent alignment lattice with T 3.
Σ b.
2,
3,6
0,e
1,a
2,4
3.0
b
2,6
3,5
(c) Recognition lattice topology: intersection of (a) and (b) with all paths for output ab highlighted.
Σ
Σ
Σ
E
Σ
2
(d) k-constrained label and frame dependent align-
(e) Label dependent alignment lattice with ((T))
ment lattice with 3, =
Figure 1: Examples of GNAT components
4.2 Alignment Lattices
Given the input feature sequence length T, the alignment lattice FSA L. defines all the possible alignments between the feature sequence and allowed label sequences. Since the feature sequence length T usually differs from the label sequence length U, many different alignments between the feature sequence and label sequence can be defined. The states in an alignment lattice FSA encode how the next label or e-transition corresponds to some position in the feature sequence.
We can choose different structures for Lt by encoding one both of the positions in the feature sequence and the label sequence. A simple example is the frame dependent alignment similar to [14], where each frame is aligned at most one label:
Q {0.... .O..... T
ir = 0 Fr (T}
Fr {(t ((t-1,y) (1,1), (ort)>eeRR BEEU 151ST}
Here any state / < T represents position in the feature sequence. We start by aligning to the initial frame of the feature sequence, and repeatedly shift to the next frame for every subsequent label or e-transition until all frames have been visited. Figure lb depicts frame dependent alignment lattice.
5

---

To allow a label sequence longer than the feature sequence, we can use the k-constrained label and frame dependent alignment similar [13]:
Qr {L,, 1,
åT = (0,0)
FT {T,,0)}
ET { 11,,,-1),,,,t,)))
V 0515, 1 u 1((( |15157]
Here, up l consecutive label transitions align to any single frame. An e-transition is then taken to explicitly shift the alignment to the next frame. The number of labels aligned to one frame is constrained by constant k solely in order to impose acyclicity. Figure 1d depicts a k-constrained label and frame dependent alignment lattice.
Some models may only depend on the position in the label sequence, similar to [8]. In this case we can bound the length of the label sequence by some function ((T), and use the following label dependent alignment:
Qr = {o. (o.....((TT) ((T}
iT 0
PT (((T)}
Er {u L, IvE 1s <us((T)) U ((( | a (())
Here, each label can be seen as aligning to the entire feature sequence, and the e-transition serves as an explicit termination of the label sequence. Figure Ie depicts a label dependent alignment lattice.
4.3
Weight functions
The weight function wa.x translates trainable parameters into transition weights for states in the recognition lattice Ao,x- The choice of the weight function depends the choice of the context and alignment lattice FSA, especially the alignment lattice where the meaning of a state directly affects how the weight function can access x.
The experiments discussed in this paper use frame dependent and k-constrained label and frame dependent alignment lattices. In these two types of alignment lattices, a non-final state qa in Qr contains a position T(%a) in the feature sequence (the state itself the case of frame dependent aligmment lattices; the first value in the state in the case of label and frame dependent alignment lattices). Weight functions can thus be defined in three steps,
1. Feed x into an encoder, such as unidirectional or bidirectional RNN, or self-attention encoder obtain the sequence of hidden units h of dimension D. In the experiments we compared streaming non-streaming encoders.
2. Map a single frame of hidden units h[t] and context state qe to a ([-) + 1 dimmenssioa vector, corresponding to the unnormalized transition weights for E (e)...
3. Optionally locally normalize transition weights across y E E {e) given (ga.ge).
For step 2, we experiment with the following concrete modelling choices with varying degree of parameter sharing,
Per-state linear projection (unshared) For every context state dc, we obtain a D x ( + 1) projection matrix WG, and (E + 1)-mim bias vector by. from 0, and define for y EU (c):
we.x(ga. ge. y) (W,. hl[(qa)] bbb.)
Shared linear projection with per-state embedding (shared-emb) We obtain from 0 (a) for every context state qu a D-dimensional state embedding F.g. (b) independent of context states a Dx ([+|+1) projection matrix W and a ([Σ) + 1)- bias vector b. and define for y EU {e}:
x (ga qe, y) (W tanh(((((q..)) Ege) + b) [y]

---

Shared linear projection with RNN state embedding (shared-run) Similar to shared linear projec- tion with per-state embedding but Eg. is obtained from running an RNN (e.g. LSTM) on the n-gram label sequence represented by Gc-
5 Discussion
A Modular Framework All the existing locally normalized models can be explained within the modular framework presented in Section with a particular choice of context size, alignment lattice and of course with constraining the weights to be locally normalized using softmax function. Appendix B presents how CE CTC, LAS, RNNT and HAT models can be expressed within this framework. This allows controlled comparison of different components models by mixing different modeling choices. Like locally and globally normalized models in general, when the weights are locally normalized, the denominator of models defined in our framework is one.
well as creating new
Finally, our framework is very different from traditional uses of finite state machines via a cascade of weighted finite state transducer compositions. The separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the depen- dency among alignment states, context states, and output label, which is impossible with composition cascades. The idea of weight function has been proposed before (e.g. [38], although as we shall see in experiments, the exact architecture of weight function plays crucial role in model accuracy.
Related Globally Normalized Models There is a rich literature on the applications of globally normalized models [5, 26, 35, 24], as well as theoretical studies on the label bias problem [24, 2, 12]. In ASR, there a lot of research on applying globally normalized models [3, 7, 6, 16, 25, 45, 18]. Among these, MMI [3, 7] is the most relevant to our work. In MMI, a sequence level score is factorized into an acoustic model (AM) score and language model (LM) score. The denominator is usually approximated over pruned lattice. Recently [34] introduced lattice-free MMI, which replaces the word level LM with a 4-gram phone LM during training. Both the GNAT and MMI are globally normalized. The GNAT model differs from MMI in several ways: (1) Unlike MMI, GNAT does not require an external LM, nor does it impose any constraint on how the sequence level scores are defined. (2) In MMI training. the LM is kept frozen while the AM parameters are optimized. In GNAT all the model parameters are trained together. (3) GNAT's recognition lattice encodes the same set of weighted alignment paths during training and inference, therefore the denominator computation in GNAT is exactly matched between training and inference. This does not hold for the standard MMI, because by using different LM topologies between training and inference, the denominator computation in training time only serves as an approximation of the denominator during inference.
poplemmentation-wiss.. GNAT trains from scratch without any need for initialization or special regular- ization techniques as used in the lattice-free version of MMI [34]. In addition we were able to train GNAT models with accelerators without any techniques discussed in [34].
The concept of global normalization has also been visited with deep neural networks [17, 10, 42, 44]. These models can be seen as special cases of MMI based on aspects such as how the sequence level scores are factorized, their use of fixed LM for training, and the mismatching training and inference LMs. Thus the comparison between MMI and GNAT applies here as well. Their use of WFST composition cascades results in weaker modelling power compared to our weight function formulation, and their use of non-streaming encoders combined with their model formulation makes globally normalized models equally expressive to the locally normalized counterparts (Appendix E).
Challenges The main challenge with the GNAT model is its scalability to a larger number of label contexts. At each training step, the model requires 01Q0al x [ΣI) semiring multiplications and additions. For n--ram context dependency, [Q (|2))-1 1)/((33) - 1), thus the computation scale exponentially by value of n. However shown in Appendix D. due to the particular structure of this space the practical computation and memory cost benchmarks do not scale exponentially with 0 S n < 2. We also note that large value of might also not be necessary: The HAT model [40] reports that a Seq2Seq model with a label history of just the two previous phonemes performs on par with a similar model with a full history trained on very large voice-search corpus. Similar observations are reported in studies with grapheme and wordpiece units [43, 11]. Due to the data sparsity there might not be enough training data to fully represent all n-gram context states, so increasing the value of might not necessarily lead to WER reduction. One way of dealing with
7

---

larger numbers of states is to use standard pruning techniques to keep only some of the most common states in the training data.
Experiments
Data We use the full 960-hour Librispeech corpus [32] for experiments. The input features are 80-dim. log Mel extracted from 25 ms window of the speech signal with a 10 ms shift. The SpecAugment library with baseline recipe parameters were used [15]. The ground truth transcription is used without any processing and tokenized by the 28 graphemes that appear in the training data.
Architecture Attention- architectures allows using the same parameterization for streaming and non-streaming models, thus for all the experiments we used 12-layer Conformer encoders [15] with model dimension 512, followed by linear layer with output dimension 640. The Conformer parameters are set such that the only difference between streaming and non-streaming models is the right context: each time frame the streaming models only access the left context (feature frames from 1 to f), while the non-streaming models can see the entire acoustic feature sequence. To enforce the consistency of the encoder architecture between streaming and non-streaming modes, we removed all the sub-archin aub-archite which behaved differently between these two modes. Specifically, we removed the convolution sub-sampling layer, and also forced the stacking layers only stack within the left context. The baseline experiments use shared-mn weight function defined in section 4.3. A single layer LSTM used with 640 cells. The experiments with the unshared weight function use linear layer of size (1Qcl x [ x 640 to project the encoder activation at each time frame into the transition weights of the recognition lattice. In our experiments, | = 32. For the n-gram context - 1). The experiments with the shared-emb weight function use an embedding table of size (Qel 128.
dependency, IQel = ([S)=+] 1)/((=)
Training All models are trained on 8 x 8 TPUs with a batch size 2048. The training examples with more than 1961 feature frames or more than 384 labels are filtered out. We used Adam optimizer [22] (81 0.9, A2 0.98, and 10-9) 90 with the Transformer learning rate schedule [41] (10k warm-up steps and peak learning rate 0.05/ 0.05/v512) 512).. We applied the same regularization techniques and the training hyperparameters used in the baseline recipe of [15].
Evaluation We report WER results on standard Librispeech test sets: test test_cllear clean and test other. The WER is either computed with sum-path algorithm or max-path algorithm. The sum-path algorithm approximately merges the alignment hypothesis corresponding to the same label sequence prefix after removal of epsilons. In ideal decoding, sum-pat man-path should result in the most likely output label sequence. The max-path algorithm computes the highest scoring alignment path using algorithms in Appendix C.
The RNN-T baselines are presented the rows corresponding to the "RNN-T" context dependency in Table la. Unlike the other rows in Table la, the decoder RNN in RNN-T in principle encodes an unlimited output history as context. The standard RNN-T model effectively uses the label frame dependent alignment lattice with an infinite k. The rows corresponding the frame dependent alignment lattice represent a slightly modified RNN-T model, where each frame is aligned to exactly one output (either blank, baselines in Table la and [15] are mainly due to our modifications to the Conformer encoder for a controlled comparison against streaming models.
Baselines
lexical label). The WER differences between non-streaming RNN-T
Choice of the context dependency Table la compares effect of n-gram context dependency for 1,2, 2,3 3 and baseline RNN-T models. The general observation is that increasing leads to better WER independent of the other choices of the modeling parameters. However, the model with 3-gram context dependency already performs par with RNN-T baseline. The 2- 2-gram context dependency perform almost on par as baseline on clean test set while still lagging on the other test set. This is consistent with the earlier observations in [40, 43, 11].
Choice of the alignment lattice The comparison of different alignment lattices in Table la suggests that this choice does not significantly contribute to the model performance. While there is a performance gap for 1-gram context dependency, we do not think there is a principal argument in favor of frame dependent alignment lattice. We speculate that this is more due to the choice of optimization parameters. However, the choice of lattice type can have some side effects. For example as & in k-constrained label frame dependent alignment lattice increases, the model has more
8

---

context dep.
weight function
WER
[%]
streaming
normalization
clean
other
context dep.
alignment lattice
weight in streaming
WER [%]
local
3.4
clean
other
global
3.3
8.4
frame
no
10.0
1-gram
local
7.0
17.4
1-gram
yes
7.1
16.0
yes
global
5.5
14.0
label frame
no
6.7
T0.2
local
2.8
6.7
yes
8.8
14.5
no
global
2.8
6.7
frame
no
2.8
6.0
2-gram
local
4.9
11.0
2-gram
yes
10.0
yes
global
3.8
9.5
label frame
no
2.5
5.6
yes
5.1
10.3
(b) Weight function parameters:
normalization and
frame
no
2.5
5.3
streaming. mmax-path decoding)
yes
4.9
9.7
3-gram
lahel frame
no
2.5
5.3
weight function
WER
%
yes
5.0
9.8
type
normalization
clean
other
frame
no
2.5
5.3
unshared
local
4.9
10.7
RNN-T
yes
5.1
9.8
global
4.2
10.6
label frame
no
2.5
5.5
shared-emb
local
5.4
13.1
yes
9.8
global
4.1
local
4.9
11.0
(a) Locally normalized baselines with different context- dependency and alignment lattice. All models used shared-mn weight function with without streaming
shared-rmn
global
3.8
9.3
(c) Comparison of different weight function types (max- path decoding).
encoder. (sum-path decoding)
Table 1: Experiment results
ability to delay its prediction to the end of the signal. This implicit lookahead can translate into performance gains particularly for unidirectional models. By limiting this quantity that performance on clean and other sets degrades by 34.9% and 36.7% respectively.
observed
Choice of the weight function normalization Table 1b examines the effect of weight normal- ization on non-streaming and streaming models. Here we present models with 1-gram and 2-gram context dependency with frame dependent alignment lattice. Note that for O-gram context depen- dency with a frame dependent alignment lattice it is easy to show that locally normalized and globally normalized models equivalent. For non-streaming models, the normalization seems to not have an impact on the performance quality neither for 1 1-gram nor for 2-gram context dependency experiments. This is validates our argument in the end of Section 2.2 that locally and globally normalized models express the same class of conditional distributions under this particular setting. On the other hand, streaming models significantly benefit from global normalization: For the clean test set, the globally normalized model outperforms the locally normalized model by about 21 % relative WER for 1-gram context dependency and by about 20% relative gain for 2-gram context dependency.
The globally normalized model with 2-gram context dependency also beat the baseline streaming RNN-T model in Table la and performs significantly closer to the non-streaming RNN-T baseline. The equivalent streaming RNN-T model performs 5.1% on test clean and the non-streaming model performs 2.5% same test set. The globally normalized model decoded with max max-path algorithm performs 3.8% on same test set. The globally normalized model effectively closed almost 50% the performance gap between streaming and non-streaming models.
The reported performance for the globally normalized models from max-path decoding, while the baselines benefit from sum-path decoding. Comparing the locally normalized models' WER from max-path decoding in Table lb and their counterparts in Table la, is clear that sum-path decoding leads to an extra WER gain. This gain is more significant on test_ othe_. So we expect the globally model performs even better when decoded with sum-path.
The standard num-path algorithms use several heuristics particularly for path merging and pruning. While similar merging techniques can be applied the globally normalized models, the pruning heuristics require several adjustments. This is particularly due to the nature of the globally normalized models where the transition weights are not constrained and can take any value, unlike locally normalized models where the transition weights are constrained to be positive number between O and 1 and sum to for all the weights leaving the same state in recognition lattice. We plan explore approaches to improve globally normalized model decoding in the future.

---

Choice of the weight function architecture Finally Table 1c compares different choices of the architectures for streaming model with 2-gram context dependency and frame dependent alignment lattice. While the unshared and shared-mn architectures are very different in terms of parameter sharing among states, both perform well, though the shared-rnn architecture performs slightly better. The shared-emb architecture performs significantly worst than shared-rnn architecture. Note that the shared-rnn model is able to learn common structures across states in the context dependency while shared-emb does not have such capability.
7
Conclusion
The GNAT model was proposed and evaluated with the focus on the label bias problem and its impact on the performance gap between streaming and non-streaming locally normalized ASR. The finite context property of this model allows exact computation of the sequence level normalization which makes this model differ from existing globally normalized models. Furthermore, the same property allows accelerator friendly training and inference. We showed that the streaming models with globally normalized criteria can significantly close the gap between streaming and non-streaming models by more than 50%. Finally, the modular framework introduced in this paper to explain the GNAT model encompasses all the common neural speech recognition models. This enables fair and accurate comparison of different models via controlled modelling choices and creation new ASR models.
10

---

References
[1] Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri Openf't: general and efficient weighted finite-state transducer library. In International Conference on Implementation and Application of Automata, pages 11-23. Springer, 2007.
[2] Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. arXiv preprinl 16Xiv:6003000
2016.
[3] Lalit Bahl, Peter Brown, Peter De Souza, and Robert Mercer. Maximum mutual information estimation of hidden markov model parameters for speech recognition. In ICASSP '6. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 11, pages 49-52.
IEEE, 1986.
[4] Léon Bottou.
Une approche théorique de I' d'apprentissage connexionniste et applications à la recommaissance de la parole. PhD thesis, Paris 11, 1991.
[5] Léon Bottou, Yoshua Bengio, and Yann Le Cun. Global training of document processing systems using graph transformer networks. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 489 4944. IEEE, 1997.
[6] John Bridle and L Dodd. An alphanet approach to optimising input transformations for continuous speech recognition. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, pages 277-280. IEEE Computer Society, 1991.
[7] Peter F Brown. The acoustic-modelin problem in automatic speech recognition. Technical report, Carnegie-Mellon University, Pittsburgh, PA, Department of Computer Science, 1987.
[8] William Chan, Navdeep Jaitly, Quoe V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint (rXiv:1508.0/22 2015.
[9] Zhiyi Chi. Statistical properties probabilistic context-free grammars. Computational Linguis- tics, 25(1):131-160, 1999.
[10] Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Wav2letter: an end-to-end convnet- based speech recognition system. arXiv preprint arXiv: (/609003/99 2016.
[11] Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, Rodrigo Cabrera, and Eugene Weinstein. Rnn-transducer with stateless prediction network. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7049-7053, IEEE, 2020.
[12] Kartik Goyal, Chris Dyer. and Taylor Berg- Kirkpatrick. An empirical investigation of global and local normalization for recurrent neural sequence models using a continuous relaxation to beam search. arXiv preprint arXiv:/904.0683 1904.08344, 2019.
[13] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint
1rX1v:1211.3711, 2012.
[14] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist
temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[15] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08 00000, 2020.
[16] Asela Gunawardana, Milind Mahajan, Alex Acero, and John C Platt. Hidden conditional random fields for phone classification. In Ninth European Conference on Speech Communication and Technology. Citeseer, 2005.
[17] Awni Hannun. Vineel Pratap, Jacob Kahn, and Nei-Ning Hsu. Differentiable weighted finite- state transducers. arXiv preprint arXiv:2010.0100) 2020.
[18] Yasser Hifny and Steve Renals. Speech recognition using augmented conditional random fields.
IEEE Transactions on Audio, Speech, and Language Processing, 17(2):354-365 2009.
[19] Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6): 2012.

---

[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. 18):11735-7880, 1997.
[21] k2-fsa authors. k2-fsa.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6980, 2014.
[23] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[24] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
[25] Martin Ian Layton. Augmented statistical models for classifying sequence data. PhD thesis, University of Cambridge, 2007.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (0111))227883333 1998.
[27] Mehryar Mohri. Semiring frameworks and algorithms for shortest-distance problems. Journal of Automata, Languages and Combinatorics, 7(3):321-350, 2002.
[28] Mehryar Mohri. Weighted automata algorithms. In Manfred Droste, Werner Kuich, and Heiko Vogler, editors, Handbook of Weighted Automata, pages 213-254. Springer, 2009.
[29] Mehryar Mohri, Fernando Pereira, and Michael Riley. Speech recognition with weighted finite-state transducers. In Jacob Benesty, M. Sondhi, and Yiteng Huang, editors, Handbook of Speech Processing, chapter 28, pages 559-582. Springer, 2008.
[30] Mehryar Mohri and Michael D Riley. On the disambiguation of weighted automata. In International Conference Implementation and Application of Automata, pages 263-278. Springer, 2015.
[31] Nelson Morgan and Herve Bourlard. Continuous speech recognition using multilayer percep- trons with hidden markov models. In International conference on acoustics, speech, and signal processing, pages 413-416. IEEE, 1990.
[32] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pages 5206-5210. IEEE, 2015.
[33] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In JEEE 2011 workshop on automatic speech recognition and understanding, number CONF IEEE Signal Processing Society, 2011.
[34] Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. Purely eequeence-trained neural networks for asr based on lattice-free mmi. In Interspeech, pages 2751-2755, 2016.
[35] Ronald Rosenfeld. A whole sentence maximum entropy language model. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 230-237.
IEEE, 1997.
[36] Noah A Smith and Mark Johnson. Weighted and probabilistic context-free grammars are equally expressive. Computational Linguistics, 33(4):477-491, 2007.
[37] Hao Tang, Liang Lu, Lingpeng Kong, Kevin Gimpel, Karen Livescu, Chris Dyer, Noah A Smith, and Steve Renals. End-to-end neural segmental models for speech recognition. IEEE Journal of Selected Topics in Signal Processing, (((8):12541125) 2017.
[38] Hao Tang, Weiran Wang, Kevin Gimpel, and Karen Livescu. Discriminative segmental cascades for feature-rich phone recognition. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 561-568, IEEE, 2015.
[39] Ehsan Variani, Tom Bagby, Erik McDermott, and Michiel Bacchiani. End-to-end training of acoustic models for large vocabulary continuous speech recognition with tensorflow. 2017.
[40] Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley. Hybrid autoregressive transducer (hat). In ICASSP 2020-2020 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6139-614 6133. IEEE, 2020.
12

---

[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint vrXiv:7766.33662 2017.
[42] Hongyu Xiang and Zhijian Ou. Crf-based single-stage acoustic modeling with ctc topology. In ICASSP 2019-2019 IEEF International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5676-5680 IEEE, 2019.
[43] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with trans- former encoders and rnn-t loss. In ICASSP 2020-2020 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833. IEEE, 2020.
[44] Huahuan Zheng, Wenjie Peng. Zhijian Ou, and Jinsong Zhang. Advancing ctc-crf based end- to-end speech recognition with wordpieces and conformers. arXiv preprint arXiv:2107.03007. 2021.
[45] Geoffrey Zweig and Patrick Nguyen. A segmental crf approach to large vocabulary continuous speech recognition. In 2009 IEEE Workshop on Automatic Speech Recognition & Understanding. pages 152-157. IEEE, 2009.
13


Reviews:
[{"review_id": "c9oitDHXM-c", "reviewer": "Reviewer_j8Qd", "summary": "This paper introduces the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition.\nAn exact computation of the denominator for the sequence-level normalization is realized in the WFSA framework. The WFSA based modular framework potentially can encompass some common neural speech recognition models.\nExperiments are conducted on Librispeech and show that GNAT can close 50% of the WER gap between streaming and non-streaming ASR. ", "questions": "see above", "limitations": "see above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-Strengths\n\nThe paper is well organized and written. The content is intensive.\n\nBased on the solid theory of WFSA, the GNAT model is new and successfully developed.\n\nIt is well known that the label bias problem suffered by locally normalized models can be naturally overcome by globally normalized models. It is nice to see that this paper provides a new insight that such advantage of globally normalized models in overcoming label bias can bring evident performance improvement in streaming speech recognition, with thorough empirical validations.\n\n-Weaknesses\n\nHowever, I have the following concerns.\n\nI) GNAT seems to be only feasible for small number of units.\n\nThe difficulty in training globally normalized models is the computation of the denominator (aka partition function in random field terminology). The core contribution of this paper is that based on theory of WFSA, it realizes an efficient algorithm for its exact computation, but still only for small number of units (28 graphemes, |\\Sigma|=32, in this paper) and limited context dependency.\n\nCurrent state-of-the-art ASR systems often use word-pieces, which usually contain thousands of units. It is not clear whether the proposed algorithm can be applied to such setting. (Line 247-258 only discusses the complexity w.r.t. n (the order of context dependency))\n\nLine 76-77: \"However with our proposed modular framework, globally normalized model training can be made practical with careful modelling choices on modern hardware.\"\n\nThis claim only works for small number of units. \n\nII) Imprecise comments about locally and globally normalized models.\n\nLine 78-79: For non-streaming settings, [33] shows that locally and globally normalized models express the same class of conditional distributions P(z|x).\n\nLine 319-320: the equal expressiveness of globally and locally normalized models under non-streaming setting [33].\n\nLine 245-246: all these models are non-streaming, where as explained multiple times in our paper, global normalization and local normalization are equally expressive.\n\nIn fact, the paper of [33] does not draw such a strong conclusion \"the equal expressiveness of globally and locally normalized models\"; it only compares weighted context-free grammars (WCFGs) and probabilistic context-free grammar (PCFG) - the two very particular models, and in the non-parametric setting.\n\nBasically, globally normalized models are also well known as random fields or undirected graphical models or Markov networks; locally normalized models as directed graphical models or Bayesian networks. It is well documented in Section 4.5 from [a] about the relationship between the two classes of models.\n\nAlso for your information, it is said in [b] that:\n\"...the label bias problem implies that globally normalized models can be strictly more expressive than locally normalized models.\"\n\nIII) The introduction to prior work on Globally Normalized Models should be improved. A number of statements are imprecise.\n\nLine 234-241: the paper claims 4 differences between GNAT and MMI.\nThe third difference is imprecise: the exact computation of the denominator is also taken in the lattice-free MMI [31].\n\nLine 242-246: These models can be seen as special cases of MMI thus all the differences between MMI and GNAT model applies here as well. ...\n\nThe CTC-CRF model [37,39] should not be classified as special cases of MMI. In fact, this work is closely related to the CTC-CRF model [37,39]. Similar to CTC-CRF, GNAT is a globally normalized model and is directly trained via conditional maximum likelihood, without any need for initialization or special regularization techniques. A main novelty in GNAT is that it uses WFSA based parameterization, which is more flexible.\n\nIt is important to establish clearer concepts (e.g. differing model definition and model optimization) and give the readers more precise picture.\n\nIV) Imprecise statements about the label bias problem in streaming and non-streaming settings (Section 2.1 and 2.2).\n\nIt is said in this paper: In non-streaming settings, score w(z_i | z_{<i}, x) can be directly expressed as probability p(z_i | z_{<i}, x). In streaming settings, \\prod w(z_i | z_{< i}, x_{<t(i)) is in general not equal to \\prod p(z_i | z_{<i}, x), so there is label Bias problem in streaming settings. \n\nHowever, in non-streaming settings, modeling with w(z_i | z_{<i}, x) (or say, via directed sequential models in general) inherently has the problem of label bias [a,b,c]. To be precise, the label bias problem is more severe in streaming settings, and less severe in non-streaming settings.\n\nV) About M-gram context dependency and the RNN-T baseline.\nThis part of content is hard to follow.\n\n\"M is equal to the length of the longest feature sequence in the training data (1961) and for label frame dependent M is equal to 1961 + 384.\"\nNo complexity analysis for computing the denominator in this case.\n\nMore introduction to the RNN-T baseline is needed. Is it implemented as a GNAT?\n\nMinors:\nLine 594: Sigma -> \\Sigma\n\n[a] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.\n\n[b] Andor, et al., \"Globally Normalized Transition-Based Neural Networks\", ACL, 2016.\n\n[c] Wiseman, et al., \"Sequence-to-sequence Learning as Beam-Search Optimization\", EMNLP, 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_AIxKAPPZhN", "reviewer": "Reviewer_tQ4B", "summary": "This paper introduces the Globally Normalized Autoregressive Transducer (GNAT) to address the label bias problem in streaming speech recognition. ", "questions": "see weakness", "limitations": "none", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1.This work mitigates the performance gap between the streaming and non-streaming systems by solving the label bias problem. It is a novel idea. \n\n2.Moreover, this work is unlike other works, which simply tune the structure of networks. It is concrete in both theory and implementation. The experimental results in streaming ASR through the proposed global normalization significantly close more than 50% of the WER to a non-streaming system.\n\n3.Theoratical soundness under WFSA framework\n\nweakness:\nHere,  the readers would have more expectations of this work.\nYou still need to compile the WFSA, which requires huge resources and is less flexible without an on-the-fly compiling mechanism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VWhe6JiCtFw", "reviewer": "Reviewer_wc8G", "summary": "The authors tackle in my opinion 2 main subproblems which are relevant: \n1. they propose a unified wFST theoretical framework to unify many of the common ASR architectures alignment aware architectures, namely, RNN-T, CTC, HAT, LAS ... and variations thereof as instances of the same theoretical framework. This was partially covered in [16], but not as well developed as in this paper. \n\n2. They show that approximately 50 % of the degradation that happens when going from offline models (Conformer encoders w/o limited attention) to causal encoders (causal convolutions and causal attention) can be reverted back when using globally normalized variants. They also propose an algorithm implementation as pseudo-code in the appendix.\n\nThe results are interesting and relevant for the ASR community. ", "questions": "\nI think the following points would have added a lot of value to the paper:\n\n* experiments on how the performance changes with the encoder look-ahead\n* analysis of different approaches for approximating the denominator sum, e.g. n-best, lattices ,,, \n* analysis/experiments on the conditions for which the label bias is  a major problem by for instance what would happen on different datasets or what would happen if we have noisy training data , ... etc, \n* the paper uses a very small vocabulary of 30 characters, should the same conclusions arise with word-level models composed of thousands of words/word-pieces ?\n\nFinally, despite the novelty of the globally normalized model, the paper seems related to K2 FSA (https://github.com/k2-fsa/k2), which seems to be aiming to similar goals or features.   \n\n ", "limitations": "\nThe authors have analysed the limitations of their proposal in terms of training time and cost which is one of the most important questions a model such as the presented one raises. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " The paper seems relevant but mainly  for the ASR community, which could be an strength or weakness depending on the readers background. \n\nThe proposed GNAT or general model seems as an application of classical ASR framework to the deep learning (DL) models that are currently the state of the art. The classical formalisms, well known from HMM-based AM models,  are introduced to generalize many DL models into the wFSA/T formalism similarly as done in ref [16].\n\nThe authors relate their proposed model with standard DL models. They provide examples of their proposed merged framework as well as mapping  well-known standard architectures within it.\nThey also  argue that the globally normalized models improve streaming models by fixing the label bias problem providing some empirical evidence.\n\nHowever, there are many questions and experiments missing for a full understanding of the main contribution of the paper, which is the globally normalized RNN-T like models for ASR. While from the engineering perspective it is clear the paper was a huge internal effort, the main contribution of an efficient way to compute a globally normalized model in a wFSA framework could have been analyzed further. \nIt is not clear to which extend the problems on label bias are major for the streaming ASR models or in which circumstances. For instance, there are encoders with small look-ahead in the literature or dual [1] streaming/non-streaming encoders that could also solve the label bias problem, etc -- see below--. It is not clear how this 2 potential solutions will interact for instance.\n\n\n[1] Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling ( https://openreview.net/forum?id=Pz_dcqfcKW8 )", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "INAxQUNmiMI", "reviewer": "Reviewer_KkKb", "summary": "This paper proposes a globally normalized speech recognition model. The model consists of two finite-state acceptors, one that provides context (reminiscent to a language model), and the other that provides the alignment from frames to labels. The paper argues that the performance of streaming is worse because of the label bias problem, and that training models with global normalization alleviates the problem. Experiments show that, in the streaming setting, global normalization generally performs better than local normalization.", "questions": "The FSA looks deceptively small, but the FSA actually describes paths of all possible lengths. It is probably worth noting this in the paper that once these FSAs are composed, the final FSA could have exponentially many edges.\n\nThe label bias seems to be a mismatch between training and test conditions, not so much about global versus local normalization. I do agree that globally normalized models would consider those partial paths during training, but wouldn't a locally normalized trained in streaming mode alleviates the problem too?\n\n> ... by minimizing the negative log-conditional-likelihood loss E_{P(x, y)}[...] = ...\n\nP(x, y) is not defined.\n\n> It is worth noting that any locally normalized model is trivially a globally normalized because Z(x) = 1 in this case.\n\nIt is not as trivial as it seems, and boils down to the distinction between Bayesian networks and Markov random fields. The community has been aware of this ever since the existence of discriminative training. I believe the first to discuss this in the end-to-end setting is Tang et al., 2017.\n\nTang et al., End-to-end neural segmental models for speech recognition, IEEE Journal of Selected Topics in Signal Processing, 2017\n\n> ... even though our algorithms cannot be directly implemented using existing toolkits such as OpenFst [1] or Kaldi [30].\n\nIn principle, we could. I'm not entirely sure what's the intention here.\n\n> ... under the log semiring can be viewed as the unnormalized negative log conditional probability P_\\theta(y | x) = \\frac{\\exp(-W(A_{\\theta, x}(y)))}{...}\n\nShouldn't W(A_{\\theta, x}(y)) be just A_{\\theta, x}(y)?\n\n> Our separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency among alignment states, context states, and output label, which is impossible with composition cascades.\n\nI believe this was proposed in Tang et al., 2015, called structured composition. There could be earlier papers.\n\nTang et al., Discriminative segmental cascades for feature-rich phone recognition, ASRU, 2015\n\n> The transcript truth is used without any processing and tokenized ...\n\nGround truth? Or just transcript?", "limitations": "The societal impact is the same as any other ASR work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The major strength is the formalism to put several ASR models under the same framework. The examples in Figure 1 are also very helpful for understanding the framework.\n\nOverall, this is a solid paper. There are several sections that are a bit notation heavy, but that's the nature of describing FSAs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept