Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a globally normalized speech recognition model. The model consists of two finite-state acceptors, one that provides context (reminiscent to a language model), and the other that provides the alignment from frames to labels. The paper argues that the performance of streaming is worse because of the label bias problem, and that training models with global normalization alleviates the problem. Experiments show that, in the streaming setting, global normalization generally performs better than local normalization.
strengths_and_weaknesses: The major strength is the formalism to put several ASR models under the same framework. The examples in Figure 1 are also very helpful for understanding the framework.

Overall, this is a solid paper. There are several sections that are a bit notation heavy, but that's the nature of describing FSAs.
questions: The FSA looks deceptively small, but the FSA actually describes paths of all possible lengths. It is probably worth noting this in the paper that once these FSAs are composed, the final FSA could have exponentially many edges.

The label bias seems to be a mismatch between training and test conditions, not so much about global versus local normalization. I do agree that globally normalized models would consider those partial paths during training, but wouldn't a locally normalized trained in streaming mode alleviates the problem too?

> ... by minimizing the negative log-conditional-likelihood loss E_{P(x, y)}[...] = ...

P(x, y) is not defined.

> It is worth noting that any locally normalized model is trivially a globally normalized because Z(x) = 1 in this case.

It is not as trivial as it seems, and boils down to the distinction between Bayesian networks and Markov random fields. The community has been aware of this ever since the existence of discriminative training. I believe the first to discuss this in the end-to-end setting is Tang et al., 2017.

Tang et al., End-to-end neural segmental models for speech recognition, IEEE Journal of Selected Topics in Signal Processing, 2017

> ... even though our algorithms cannot be directly implemented using existing toolkits such as OpenFst [1] or Kaldi [30].

In principle, we could. I'm not entirely sure what's the intention here.

> ... under the log semiring can be viewed as the unnormalized negative log conditional probability P_\theta(y | x) = \frac{\exp(-W(A_{\theta, x}(y)))}{...}

Shouldn't W(A_{\theta, x}(y)) be just A_{\theta, x}(y)?

> Our separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency among alignment states, context states, and output label, which is impossible with composition cascades.

I believe this was proposed in Tang et al., 2015, called structured composition. There could be earlier papers.

Tang et al., Discriminative segmental cascades for feature-rich phone recognition, ASRU, 2015

> The transcript truth is used without any processing and tokenized ...

Ground truth? Or just transcript?
limitations: The societal impact is the same as any other ASR work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition.
An exact computation of the denominator for the sequence-level normalization is realized in the WFSA framework. The WFSA based modular framework potentially can encompass some common neural speech recognition models.
Experiments are conducted on Librispeech and show that GNAT can close 50% of the WER gap between streaming and non-streaming ASR. 
strengths_and_weaknesses: -Strengths

The paper is well organized and written. The content is intensive.

Based on the solid theory of WFSA, the GNAT model is new and successfully developed.

It is well known that the label bias problem suffered by locally normalized models can be naturally overcome by globally normalized models. It is nice to see that this paper provides a new insight that such advantage of globally normalized models in overcoming label bias can bring evident performance improvement in streaming speech recognition, with thorough empirical validations.

-Weaknesses

However, I have the following concerns.

I) GNAT seems to be only feasible for small number of units.

The difficulty in training globally normalized models is the computation of the denominator (aka partition function in random field terminology). The core contribution of this paper is that based on theory of WFSA, it realizes an efficient algorithm for its exact computation, but still only for small number of units (28 graphemes, |\Sigma|=32, in this paper) and limited context dependency.

Current state-of-the-art ASR systems often use word-pieces, which usually contain thousands of units. It is not clear whether the proposed algorithm can be applied to such setting. (Line 247-258 only discusses the complexity w.r.t. n (the order of context dependency))

Line 76-77: "However with our proposed modular framework, globally normalized model training can be made practical with careful modelling choices on modern hardware."

This claim only works for small number of units. 

II) Imprecise comments about locally and globally normalized models.

Line 78-79: For non-streaming settings, [33] shows that locally and globally normalized models express the same class of conditional distributions P(z|x).

Line 319-320: the equal expressiveness of globally and locally normalized models under non-streaming setting [33].

Line 245-246: all these models are non-streaming, where as explained multiple times in our paper, global normalization and local normalization are equally expressive.

In fact, the paper of [33] does not draw such a strong conclusion "the equal expressiveness of globally and locally normalized models"; it only compares weighted context-free grammars (WCFGs) and probabilistic context-free grammar (PCFG) - the two very particular models, and in the non-parametric setting.

Basically, globally normalized models are also well known as random fields or undirected graphical models or Markov networks; locally normalized models as directed graphical models or Bayesian networks. It is well documented in Section 4.5 from [a] about the relationship between the two classes of models.

Also for your information, it is said in [b] that:
"...the label bias problem implies that globally normalized models can be strictly more expressive than locally normalized models."

III) The introduction to prior work on Globally Normalized Models should be improved. A number of statements are imprecise.

Line 234-241: the paper claims 4 differences between GNAT and MMI.
The third difference is imprecise: the exact computation of the denominator is also taken in the lattice-free MMI [31].

Line 242-246: These models can be seen as special cases of MMI thus all the differences between MMI and GNAT model applies here as well. ...

The CTC-CRF model [37,39] should not be classified as special cases of MMI. In fact, this work is closely related to the CTC-CRF model [37,39]. Similar to CTC-CRF, GNAT is a globally normalized model and is directly trained via conditional maximum likelihood, without any need for initialization or special regularization techniques. A main novelty in GNAT is that it uses WFSA based parameterization, which is more flexible.

It is important to establish clearer concepts (e.g. differing model definition and model optimization) and give the readers more precise picture.

IV) Imprecise statements about the label bias problem in streaming and non-streaming settings (Section 2.1 and 2.2).

It is said in this paper: In non-streaming settings, score w(z_i | z_{<i}, x) can be directly expressed as probability p(z_i | z_{<i}, x). In streaming settings, \prod w(z_i | z_{< i}, x_{<t(i)) is in general not equal to \prod p(z_i | z_{<i}, x), so there is label Bias problem in streaming settings. 

However, in non-streaming settings, modeling with w(z_i | z_{<i}, x) (or say, via directed sequential models in general) inherently has the problem of label bias [a,b,c]. To be precise, the label bias problem is more severe in streaming settings, and less severe in non-streaming settings.

V) About M-gram context dependency and the RNN-T baseline.
This part of content is hard to follow.

"M is equal to the length of the longest feature sequence in the training data (1961) and for label frame dependent M is equal to 1961 + 384."
No complexity analysis for computing the denominator in this case.

More introduction to the RNN-T baseline is needed. Is it implemented as a GNAT?

Minors:
Line 594: Sigma -> \Sigma

[a] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.

[b] Andor, et al., "Globally Normalized Transition-Based Neural Networks", ACL, 2016.

[c] Wiseman, et al., "Sequence-to-sequence Learning as Beam-Search Optimization", EMNLP, 2016.
questions: see above
limitations: see above
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors tackle in my opinion 2 main subproblems which are relevant: 
1. they propose a unified wFST theoretical framework to unify many of the common ASR architectures alignment aware architectures, namely, RNN-T, CTC, HAT, LAS ... and variations thereof as instances of the same theoretical framework. This was partially covered in [16], but not as well developed as in this paper. 

2. They show that approximately 50 % of the degradation that happens when going from offline models (Conformer encoders w/o limited attention) to causal encoders (causal convolutions and causal attention) can be reverted back when using globally normalized variants. They also propose an algorithm implementation as pseudo-code in the appendix.

The results are interesting and relevant for the ASR community. 
strengths_and_weaknesses:  The paper seems relevant but mainly  for the ASR community, which could be an strength or weakness depending on the readers background. 

The proposed GNAT or general model seems as an application of classical ASR framework to the deep learning (DL) models that are currently the state of the art. The classical formalisms, well known from HMM-based AM models,  are introduced to generalize many DL models into the wFSA/T formalism similarly as done in ref [16].

The authors relate their proposed model with standard DL models. They provide examples of their proposed merged framework as well as mapping  well-known standard architectures within it.
They also  argue that the globally normalized models improve streaming models by fixing the label bias problem providing some empirical evidence.

However, there are many questions and experiments missing for a full understanding of the main contribution of the paper, which is the globally normalized RNN-T like models for ASR. While from the engineering perspective it is clear the paper was a huge internal effort, the main contribution of an efficient way to compute a globally normalized model in a wFSA framework could have been analyzed further. 
It is not clear to which extend the problems on label bias are major for the streaming ASR models or in which circumstances. For instance, there are encoders with small look-ahead in the literature or dual [1] streaming/non-streaming encoders that could also solve the label bias problem, etc -- see below--. It is not clear how this 2 potential solutions will interact for instance.


[1] Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling ( https://openreview.net/forum?id=Pz_dcqfcKW8 )
questions: 
I think the following points would have added a lot of value to the paper:

* experiments on how the performance changes with the encoder look-ahead
* analysis of different approaches for approximating the denominator sum, e.g. n-best, lattices ,,, 
* analysis/experiments on the conditions for which the label bias is  a major problem by for instance what would happen on different datasets or what would happen if we have noisy training data , ... etc, 
* the paper uses a very small vocabulary of 30 characters, should the same conclusions arise with word-level models composed of thousands of words/word-pieces ?

Finally, despite the novelty of the globally normalized model, the paper seems related to K2 FSA (https://github.com/k2-fsa/k2), which seems to be aiming to similar goals or features.   

 
limitations: 
The authors have analysed the limitations of their proposal in terms of training time and cost which is one of the most important questions a model such as the presented one raises.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces the Globally Normalized Autoregressive Transducer (GNAT) to address the label bias problem in streaming speech recognition. 
strengths_and_weaknesses: Strength:
1.This work mitigates the performance gap between the streaming and non-streaming systems by solving the label bias problem. It is a novel idea. 

2.Moreover, this work is unlike other works, which simply tune the structure of networks. It is concrete in both theory and implementation. The experimental results in streaming ASR through the proposed global normalization significantly close more than 50% of the WER to a non-streaming system.

3.Theoratical soundness under WFSA framework

weakness:
Here,  the readers would have more expectations of this work.
You still need to compile the WFSA, which requires huge resources and is less flexible without an on-the-fly compiling mechanism.
questions: see weakness
limitations: none
review_title: null

Meta-review: metareview: Reviewers acknowledge that this paper has good contributions, including solving label bias problem, mitigating the performance gap between the streaming and non-streaming systems, and proposing a unified WFST theoretical framework, which should have value to the ASR community. However, reviewer j8Qd reported that he/she did not receive the notifications of the author response. He/she then posted new problems on the revised paper on Aug. 26, while the authors cannot respond. The problems are mainly on the comparison with prior works on the mismatching training and inference. The reviewer argued that there are on imprecise comments/claims. Given this situation, I give a recommendation of acceptance but strongly ask the authors to carefully look into these problems and address them with full effort. 
confidence: Less certain
Decision: Accept