Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to use Bayes Risk factors to control specific characteristics of CTC alignments. Specifically, the proposed method encourages the model to move the non-blank predictions towards the beginning of the output sequence. This shift of non-blank predictions enables speed-ups both in offline and online inference. In offline inference, since the non-blank predictions occur at the beginning of the output sequence, the paper proposes to trim all blank predictions after the last non-blank prediction. As a result, the length of decoder input is shorter by 60%, making the decoding process 47% relative faster. In online inference, the speed-up is also achieved by encouraging the model to produce non-blank predictions sooner, thus reducing the drift latency caused by delayed non-blank predictions.
strength_and_weaknesses: Strengths:

The paper proposes a novel technique for controlling specific characteristics of CTC alignments using Bayes Risk. Furthermore, the paper shows how to derive the Bayes Risk CTC loss function from the CTC loss function. The conducted experiments show that the proposed method can be used to speed up both offline and online inference.

Weaknesses:

The method is compared only against vanilla CTC. However, since the proposed method effectively reduces the length of the encoder output sequence for offline recognition, it would be good to compare it with naive uniform subsampling of encoder outputs with subsampling factors 2 and 3 to achieve the same down-sampling factor. Furthermore, in the case of online recognition, the paper should mention CTC delay constraints from [1], which uses reference alignments obtained with an external DNN-HMM model. Comparing the BRCTC method with the delay constraints method would be nice. This experiment can be implemented with GTN, which the authors used to implement BRCTC, and delay constraint FSA similar to Kaldi TimeEnforcerFst.

[1] Senior, Andrew, et al. "Acoustic modelling with CD-CTC-sMBR LSTM RNNs." 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015.
clarity,_quality,_novelty_and_reproducibility: The paper proposes a novel technique for controlling specific characteristics of CTC alignments. The experiments are conducted on popular public datasets, and all relevant hyper-parameters are listed in the appendix. A reference implementation of the BRCTC loss function is also included, which makes this work reproducible, for example, with the ESPnet toolkit.

The paper is relatively clearly written but contains many minor issues discussed below. Two problems in terms of clarity are:

- Abstract and introduction state that "BRCTC achieves up to 47% inference cost reduction for offline systems without degradation in transcription performance." This statement on its own is misleading. Just changing the loss function does not lead to any speed-up. The speed-up is achieved by architectural changes enabled by BRCTC. Please clarify this in the paper.

- The use of $2u$ in eq. 8 is unintuitive at first read and needs to be explained in the main body of the paper. Please explain that $2u$ is used because $l_u = l'_{2u}$ as you do in Appendix C.

Minor comments:

- The authors cite everything with \citet, but most of the citations are parenthetical and should have been made with \citep.

- $ali(l, x)$ in Sections 2.1 and 3.1 use different formatting conventions.

- There is an extra period after footnote 1 in the abstract. Also use consistent placing of footnotes.

- Figure and section references are inconsistent (Fig 1.a, Fig. 1.b, Fig.3.c, section 4.3, sec.4.2).

- In the sentence "it is common to ask when the prediction of each token $l_u$ is finished within this path." I would recommend replacing "each token $l_u$" with "a given token $l_u$".
summary_of_the_review: This paper proposes an interesting method which could be very useful for improving the inference speed of ASR models. Therefore, after including the naive down-sampling baseline experiment and fixing the writing/formatting issues mentioned above, I am inclined to accept this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new variant of CTC that can encode conditions to achieve different preferable properties. The authors gave two examples, one is to force the model to emit the last non-blank token as early as possible; the other is to avoid the drift latency of non-blank tokens to provide better trade-off between latency and transcription quality. Both the two examples have practical merit for ASR systems. The authors did intensive experiments on both streaming and non-streaming ASR, ST and MT to show the control ability of the proposed BRCTC. The results are convincing given the experimental settings. 
strength_and_weaknesses: Strength:
1. Novelty; There are previous studies on different CTC variants but the perspective explored in this paper has not been seen in literature. The idea to 
2. Of clear practical benefits; Though most of commercial ASR systems are based on Transducer or LAS, CTC is still widely used as a standalone ASR system or integrated into Transducer/LAS training to provide auxiliary functions. And other fields like MT/ST are also relying on CTC in some scenarios. The reviewer believe the work in this study will result in more flexible CTC based designs.
3. Convincing results; The authors did intensive experiments and show both numeric results and visualization to validate the effectiveness of the proposed BRCTC.

Weakness
1. For online experiments, I think CTC-attention training is unnecessary. It's better to show pure CTC training results. 
2. Some are some variants of transducer like (alignment-restricted transducer, prunned Transducer) are also based on modification of forward-backward algorithm to achieve specific properties (latency, computational cost). It's better to cite these papers.
3. For the limitation discussion, the author mentioned the increasing training cost of BRCTC but didn't provide much details.
clarity,_quality,_novelty_and_reproducibility: Clarity: could be better.
Quality: good
Reproducibility: code is provided. Hope it will be made public.
summary_of_the_review: Overall, the reviewer think this is a very good extension to vanilla CTC and can  benefit multiple seq2seq tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: CTC is a fundamental method for many sequence-to-sequence tasks. It works by maximizing the summed probabilities of paths that can be mapped to the target sequence. As CTC treats all paths equally, it cannot generate accurate alignments between its predictions and input sequences. This work proposes a Bayes Risk CTC method (BRCTC) to make CTC alignment controllable by first dividing paths into groups and then specifying preference weights for groups. For offline ASR, BRCTC was shown to effectively reduce the inference cost by down-sampling the hidden vectors. For streaming ASR, BRCTC can accelerate the recognition by forcing the model to emit non-blank spikes as early as possible.


strength_and_weaknesses: Strength
- The method is novel and well-motivated.
- As far as I can see, the method is technically sound and well-justified by the experiments.

Weakness
- The work does not specify a general condition that the grouping function f(\phi) has to meet to enable the forward-backward algorithm to be applied. Instead, it gives an example f(\phi) which is the largest index of non-blank elements in the path. If it is the only case to which the proposed method can be applied, the main value of the work is to speed up CTC inference. Although accelerating CTC inference is also important, the writing of the paper gives readers the impression that it is a general method to control CTC alignment.
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
Novelty: medium
Reproducibility: good
summary_of_the_review: Making CTC output controllable alignment prediction is an interesting topic. The proposed method is novel and technically sound. My main concern is whether the proposed method is applicable to other useful grouping functions except the example given in Sec. 3.2 and if there is a general rule for making a legal grouping function. However, the paper does not include any discussion on it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper re-formulates the widely used CTC criterion for ASR such that the alignments learnt are controllable. This is done by forcing a preference of desirable alignment paths in the CTC loss function. Furthermore, a bayesian risk factor controls this preference. To this end, the forward and backward variables are chosen such that the above preference is satisfied. The authors report the effectiveness of the proposed technique on two tasks: 1) downsampling of speech frames for offline ASR such that inference time is reduced without performance degradation; 2) speeding up online ASR by having earlier emissions without loss in performance. Results on speech translation have also been reported.
strength_and_weaknesses: The paper is well written and the proposed techniques are very well formulated mathematically. The authors provide abundant materials in the appendix and also release their code. CTC alignment is an important problem for ASR and may have consequences on a variety of downstream applications. The authors investigate two of them and the results are impressive. The proposed technique itself is quite elegant and mathematically grounded. I appreciate the effort put in by the authors to build the mathematical framework and providing all the details of the experiments in the appendix.

Comments:
1. There shouldn’t be a ‘t’ in the subscript for ‘p’ in equation (2)
2. I may be missing something here, but shouldn’t the final result in equation (5) be a product over all ‘t’s.
3. The arg max operation defined in the first paragraph of page 5 is not very clear. There is a boolean operation inside the argmax(.) where elements will be either 0 or 1, what will argmax return in such a case?
4. Typo: page 6 last line, DFS -> DSF

Questions:
1. Why did the authors not compare against some simple baselines:

    a) Downsampling: Using a CNN for subsampling a speech sequence such that the new sequence length is similar to the one obtained by the controlled alignment. The CNN is like a feature extractor from speech. Fox example, wav2vec2.0 uses 20ms windows, what happens if we use a >20ms window and lower the sequence length? How much performance is degraded and how does it compare with the proposed model?

    b) Online Latency reduction: Plantinga et al. [1] have a simple solution for emitting phones sooner where they have an “alignment loss which encourages outputs only when features do not resemble silence.” It would also be a good idea to cite the above paper.

2. Can such an alignment controlling mechanism be used to train RNN transducers which are also very popular and use a quite similar algorithm to compute the loss?

3. What are some other applications apart from the two explored where controlling CTC alignment will be useful?

Reference:
[1] Plantinga et al. Towards Real-Time Mispronunciation Detection in Kids' Speech. ASRU 2019.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, high quality, novel and the authors have released their code for reproducibility.

summary_of_the_review: This is a good paper which provides a mathematically sound technique to control CTC alignments. This is backed by impressive results on two tasks. The paper may benefit by running some simple baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a Bayes risk version of CTC for training end-to-end automatic speech recognizers. Usually the purpose of discriminative training is to close the gap between the training loss (log likelihood) and the actual task loss (for example character error rates). This paper repurpose the risk functions for training streaming recognizers, proposing one for emitting the labels early and one for downsampling.

All reviewers unanimously praise the writing and the approach. Some reviewers question the choice of baselines, for example, not choosing naive downsampling to compare against. Some reviewers (including me) feel that the repurposing of Bayes risk is a little awkward, and there are probably better, simpler approaches to solve the problems of concern, for example, coming up with a different loss function that is significantly different from CTC or through architectural changes.

Overall, the paper receives positive reviews.
justification_for_why_not_higher_score: Bayes risk is a loss that has a long history. Even in the context of CTC, something similar has been done, so technically, the novelty is limited.
justification_for_why_not_lower_score: The approach itself has merit. The repurposing of the approach is awkward, but is good to have.
Decision: Accept: poster