Title: Revisiting the Entropy Semiring for Neural Speech Recognition
Authors: Oscar Chang
Dongseong Hwang
Olivier Siohan
Venue: ICLR 2023
Keywords: semiring
asr
ctc
rnn-t
entropy
regularization
distillation
streaming
speech recognition

Abstract: In streaming settings, speech recognition models have to map sub-sequences of speech to text before the full audio stream becomes available. However, since alignment information between speech and text is rarely available during training, models need to learn it in a completely self-supervised way. In practice, the exponential number of possible alignments makes this extremely challenging, with models often learning peaky or sub-optimal alignments. Prima facie, the exponential nature of the alignment space makes it difficult to even quantify the uncertainty of a model's alignment distribution. Fortunately, it has been known for decades that the entropy of a probabilistic finite state transducer can be computed in time linear to the size of the transducer via a dynamic programming reduction based on semirings. In this work, we revisit the entropy semiring for neural speech recognition models, and show how alignment entropy can be used to supervise models through regularization or distillation. We also contribute an open-source implementation of CTC and RNN-T in the semiring framework that includes numerically stable and highly parallel variants of the entropy semiring. Empirically, we observe that the addition of alignment distillation improves the accuracy and latency of an already well-optimized teacher-student distillation model, achieving state-of-the-art performance on the Librispeech dataset in the streaming scenario.

TL;DR: A numerically stable open-source implementation of the entropy semiring for CTC and RNN-T; obtained SOTA on Librispeech streaming.

Review discussion:
Pros:
* A novel regularizer and distillation objective based on entropy of alignment distribution achieving state of the art performance on Librispeech in streaming scenario.
* Open-source stable and parallel implementation for CTC and RNN-T
Cons:
N/A
Clarity: Very clear, pleasure to read
Novelty: Novel, significant contribution
Reproducibility: empirical results should be reproducible
NO.
n/a
Thank you for your kind words and helpful comments about our manuscript.
Strengths:
- The paper brings the mathematically backed up idea of semi-rings back to ASR such that they can be used in CTC or RNNT systems.
- The paper is clearly written.
- Experimental results show good WER performance in the Librispeech setting.
Weaknesses:
- In Table 3, does any of the papers utilize the LibriLight data? If yes, it might be better to denote that in the table. Current semiring knowledge distillation proposal might be benefiting from that external data and the comparison might not be fair enough.
- The appendix might be improved by giving some more details on the gradient back propagation over the semiring.
Clarity: Good. Clearly written except some typos.
Minor suggestions:
- $x_3 $ should be $ v_3$ in Example 2.10.
- Motvation -> Motivation in Section 5.1.
Quality: Good. Mathematical expressions are given correctly. Experiments show gains in both entropy regularization and in knowledge distillation experiments.
- In the regularization experiments, the batch size is 2048. How would this choice affect the regularization performance? Is the proposed model efficient memory-wise so that one can train with such a large batch size?
- A further analysis of the performance on sequences with different (short and long) lengths might have provided some further intuition for one of the motivations behind the proposed methods.
Novelty: Sufficiently novel. It revives the FST framework in the case of neural network training by building on the earlier studies on semi-rings.
Reproducibility: Even though some details such as the exact architecture of the RNNT model is missing (e.g. the structure of the predictor network), the paper is probably going to be accompanied by the open-source code (based on the claims in the text).
Hence, it should become reproducible.
NO.
Thank you for your kind words and helpful comments about our manuscript.
> "In Table 3, does any of the papers utilize the LibriLight data? If yes, it might be better to denote that in the table"
We have included this information in Table 3. We note that most prior work do not use the unlabeled Librilight data, and acknowledge the reviewer’s concern that the use of the additional unlabeled data makes a fair comparison difficult.
However, we note the following points:
1) With the exception of Hwang et al., none of the cited prior work does distillation, for which using unlabeled data is a natural thing to do. Hwang et al. does do distillation and also uses Librilight unlabeled data, and we do report better results.
2) The comparison with Chiu et al. should be considered a fair comparison, since we use the same architecture and the same training data (Chiu et al. uses Librilight as well). In this case, our resultant model significantly outperforms Chiu et al., with a 29% improvement on Test-Clean and a 21% improvement on Test-Other.
3) Our work also does not use any future context at all in making predictions, which is a handicap compared to prior work that does use it. Even a small amount of look-ahead can improve accuracy substantially, with a 12% improvement for Zhang et al. and a 10% improvement for Moritz et al.
> "The appendix might be improved by giving some more details on the gradient back propagation over the semiring."
In [1], Eisner notes that instead of deriving the inside-outside algorithm by hand, which can be extremely tricky, we can get it "for free" in a simple way by relying on backpropagation and automatic differentiation.
Similarly, our semiring implementation for CTC and RNN-T does not invoke any special code paths or customized over-rides for the backward pass. It simply defines a numerically stable forward pass, and relies on Tensorflow's automatic differentiation library to do the rest. We encourage the reader to lean into the "magic" of modern automatic differentiation, and not have to worry about what the semiring for the backward pass looks like.
[1] Inside-outside and forward-backward algorithms are just backprop. Jason Eisner.
> "Clearly written except some typos"
Thanks for pointing out the typos. We have corrected them.
> "In the regularization experiments, the batch size is 2048. How would this choice affect the regularization performance?"
We did not investigate how batch size affects regularization performance, but we do not think that a large batch size is necessary in general.
> "Is the proposed model efficient memory-wise so that one can train with such a large batch size?"
The short answer is yes.
Regarding memory efficiency, note that the overhead scales as a multiple of the size of the alignment lattice. Here’s a back-of-the-envelope estimate for the size of the memory overhead. For RNN-T, the shape of the table is [batch size, audio length, text length]. Say we have float32 tensors, batch size 16, audio length 1000, text length 1000, that amounts to 4 bytes * 16 * 1000 * 1000 * 2 * 2 = 256Mb. For regularization, the new table is twice as big, so a 256Mb overhead, and for distillation which is four times as big, a 768Mb overhead. For a 12Gb GPU, this amounts to only 2% and 6% of the available memory respectively.
Note that batch size 16 refers to the per-device batch, and the overall batch size is much larger due to data parallelism, i.e. multiple devices are used. (For reference, our experiments in Section 5.1 use a per-device batch size of 32, and those in Section 5.2 use a per-device batch size of 8.) Our calculation involves multiplying by 2 twice at the end. The first doubling is for the two sets of logits: the logit to produce the text token and the logit to consume the acoustic token. The second doubling is for the forward pass and the backward pass.
In general, the actual memory overhead is highly dependent on specific implementation details like the hardware, the parallelism strategy, gradient checkpointing, quantization, compiler and intermediate representations, etc.
But in most cases, it should be a tiny fraction of the memory usage, and not something to be concerned with.
> "A further analysis of the performance on sequences with different (short and long) lengths might have provided some further intuition"
The focus of this paper was on a numerically stable implementation, which required the mathematical proposal of different variants of the entropy semiring as well as the difficult coding implementation of the semiring framework for CTC and RNN-T. We have observed that even on short sequences (< 100), a naive implementation of the entropy semiring will not result in successful training due to NaNs, but did not determine exactly when it breaks. We agree that further theoretical and empirical analysis of the optimality of alignments and how that varies with sequence length is an important research topic, and we leave it to future work.
Strength: The idea of regularization or distillation based on alignment entropy is technically sound.
Overall, this paper is well structured, with detailed background introduction on semiring. Experiments are well-designed for validation purpose. Open-source contribution with this work makes it easier for speech community to experiment with and integrate this approach.
Weakness: I think experimental section could be made stronger with more details added to explain design and results. See questions in the summary section below.
Overall, this paper is clearly written, with well structured sections on background, methodology, experiments etc. It proposes new approaches to improve neural speech recognition, and it is reproducible with open-source implementation.
NO.
Thank you for your kind words and helpful comments about our manuscript.
> "I think experimental section could be made stronger with more details added to explain design and results"
We have included additional details in the description of the experimental setup. Please let us know if there are specific details we have omitted that you would like us to include. Unfortunately, due to space constraints, our descriptions might not be completely self-contained for readers less familiar with the literature. Where appropriate, we have cited specific literature for the reader to learn more.
For example, we report in Section 5.2 that we use the exact same causal Conformer architecture as Chiu et al., without describing how their random projection quantizer works. We believe that the interested reader will be able to learn more details about it by reading Chiu et al.
> "Adding std to those results in Table 1-3 to show if improvements are statistically significant"
We have included 95% WER confidence intervals in Tables 1 and 2. Unfortunately, none of the prior work in Table 3 report confidence intervals or do significance testing.
> "How to determine model hyperparameters"
We mostly followed the same architectural hyper-parameters that were used in prior work.
For example, the Conformer hyper-parameters used in Section 5.1 stick to the same hyper-parameters used in Gulati et al., and Section 5.2 adopts the exact same architectural hyperparameters as Chiu et al. (2022). Optimization hyper-parameters generally follow Vaswani et al. (2017) in using Adam, and a learning rate schedule that does a linear warm-up followed by annealing. The learning rate and batch size were chosen to enable speedy and successful training on our infrastructure.
> "Do authors also experiment with smaller models (e.g. similar as 30M or 80M #Params)?"
We conducted an additional experiment by shrinking the size of the student model in Section 5.2 to a 30M model. This is similar to the Conformer M model (30M) in Gulati et al. or the Conformer model (30M) by Yu et al. (2020) cited in Table 3.
We observe the following results:
Dev-Clean: 3.1 +- 0.2,
Dev-Other: 9.5 +- 0.4,
Test-Clean: 3.2 +- 0.2,
Test-Other: 9.5 +- 0.4.
These results compare favorably with Yu et al. (2020) who report:
Test-Clean: 3.7,
Test-Other: 9.2,
with significantly better performance on Test-Clean (14% improvement), and slightly worse performance on Test-Other (3% degradation).
Strength
- clear descriptions of each semiring operations
- novel entropy semiring formulation by considering the backpropagation capability and efficiency.
- reasonable experimental results showing the mitigation of overconfidence and the knowledge distillation
- archives the state-of-the-art performance in the Librispeech streaming setup
Weaknesses
- it could have more discussions about other applications than ASR (although they are briefly mentioned in the conclusion section).
- lack of the analysis
- there are some duplicated descriptions (e.g., the first paragraph in Section 3 and the first paragraph in Section 3 are highly overlapped with Section 1). These parts can be improved.
Clarity
- the paper is well written.
- semiring explanations are easy to understand
- some parts (e.g., Example 2.8) are difficult to follow, but later I found the solution in the appendix. It is better to add some pointers to the appendix.
- the motivation of corresponding experiments is also evident.
Quality
- The elegant formulation and reasonable experimental results make this paper's quality very high.
Novelty
- Although entropy semiring itself is not novel, I think the authors' contributions to the backprobable algorithm have enough novelty.
Reproducibility
- The method will be open-sourced. Together with the public Librispeech benchmarks, the paper has high reproducibility
NO.
Thank you for your kind words and helpful comments about our manuscript.
> "it could have more discussions about other applications than ASR"
In the Conclusion section, we have expanded our discussion of potential applications involving multiple modalities or domains, as well as suggested additional directions for future work. In the Related Work section, we have expanded our discussion about the use of semirings in relevant machine learning literature.
> "there are some duplicated descriptions (e.g., the first paragraph in Section 3 and the first paragraph in Section 3 are highly overlapped with Section 1)"
We have shortened the first paragraph of Section 3 and made our writing more concise.
> "It would be better to provide some more concrete examples for the semiring explanation"
We have included extended step-by-step derivations of Examples 2.8 and 2.10 in the style of Example 2.6 to the Appendix, and included pointers to them in the main text. This should make it easier for the reader to understand them.
> "Can we visualize whether the overconfidence is mitigated or not in Section 5.1?"
We agree that some kind of visualization of the alignments will be helpful. It is not clear what the best way of doing this is, but we think perhaps a heat map of some sort will be helpful. We will spend more time thinking about this, and include it in the camera-ready version of our paper should it be accepted.
> "It might significantly improve the performance in the out-of-domain conditions (e.g., noisy speech recognition, multilingual speech recognition)."
We agree with the reviewer that more work needs to be done to investigate the effectiveness of our proposed approach in different domains like noisy speech recognition and multilingual speech recognition. We hope that our open-source contribution will make it easy for future work to conduct such investigations.
Thanks for your response.
I'm looking forward to updates in the analysis part and the effectiveness in different domains.
We have now completed our visualization and done a further qualitative analysis on the effect of alignment entropy regularization. Interestingly, it seems like its main effect is to encourage more local exploration of alignment paths around the "peaks" in the lattice, rather than to avoid them altogether. This results in flatter but denser local optima, which helps on average.
However, on examples where the entropy regularized model does worse than the NLL model, we see that it is actually more over-confident and concentrates even more in the local vicinity of the misplaced peaks. This shows that even though alignment entropy regularization is helpful on average for reducing model over-confidence, it can actually backfire in some instances. This finding suggests important directions for future work (potentially a new semiring) that encourages more global, rather than / in addition to local, exploration of alignment paths that are much further away from existing local optima.
We thank the reviewer again for their suggestion to study empirical visualizations in the alignment space. This has provided stronger intuition for the effects of the regularization mechanism and uncovered potential directions for future work.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: The paper uses entropy semiring to implement entropy regularization and sequence knowledge distillation. The framework is mathematically clean and elegant. Experiments on automatic speech recognition show small but consistent improvements.

One reviewer has concern about whether the amount of improvement is significant. The improvement depends on entropy regularization and knowledge distillation, not the semiring itself. The experiments are proof of concepts, so I am not too concerned about the small amounts of improvement.

All reviewers unanimously praise the writing the elegance of the framework.
justification_for_why_not_higher_score: The novelty technically is not as significant as the reviewers have commented, and the authors implicitly acknowledge that by having the word "revisit" in the title. Eisner and Mohri have both independently propose semirings with tuples that can achieve interesting computations, such as for EM and for k-best decoding. The approach saves the effort of implementing dynamic programming, but does not actually save any computation.
justification_for_why_not_lower_score: The formalism of the paper has great merit, and can potentially spawn the design other training losses.

Decision: Accept: poster