BAYES RISK CTC: CONTROLLABLE CTC ALIGNMENT IN SEQUENCE-TO-SEQUENCE TASKS
Authors: Jinchuan Tian
Brian Yan
Jianwei Yu
CHAO WENG
Dong Yu
Shinji Watanabe
Venue: ICLR 2023 poster (2023)
Keywords: CTC
alignment
sequence-to-sequence
speech recognition

Abstract: Sequence-to-Sequence (seq2seq) tasks transcribe the input sequence to a target sequence. The Connectionist Temporal Classification (CTC) criterion is widely used in multiple seq2seq tasks. Besides predicting the target sequence, a side product of CTC is to predict the alignment, which is the most probable input-long sequence that specifies a hard aligning relationship between the input and target units. As there are multiple potential aligning sequences (called paths) that are equally considered in CTC formulation, the choice of which path will be most probable and become the predicted alignment is always uncertain. In addition, it is usually observed that the alignment predicted by vanilla CTC will drift compared with its reference and rarely provides practical functionalities. Thus, the motivation of this work is to make the CTC alignment prediction controllable and thus equip CTC with extra functionalities. The Bayes risk CTC (BRCTC) criterion is then proposed in this work, in which a customizable Bayes risk function is adopted to enforce the desired characteristics of the predicted alignment. With the risk function, the BRCTC is a general framework to adopt some customizable preference over the paths in order to concentrate the posterior into a particular subset of the paths. In applications, we explore one particular preference which yields models with the down-sampling ability and reduced inference costs. By using BRCTC with another preference for early emissions, we obtain an improved performance-latency trade-off for online models. Experimentally, the proposed BRCTC reduces the inference cost of offline models by up to 47% without performance degradation and cuts down the overall latency of online systems to an unseen level.

Published as a conference paper at ICLR 2023
BAYES RISK CTC: CONTROLLABLE CTC ALIGNMENT IN SEQUENCE-TO-SEQUENGG TASKS
Jinchuan Tian, Jianwei Yu", Chao Weng, Dong Yu
Tencent LAB ftyriontian, tomasyu, cweng, dyuj@tencent.com com
Brian Yan & Shinji Watanabe"
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA [byan, swatanab ( @andrew. emu, edu
ABSTRACT
Sequence-to-Sequeees (seq2seq) tasks transcribe the input sequence to target sequence. The Connectionist Temporal Classification (CTC) criterion is widely used in multiple seq2seq tasks. Besides predicting the target sequence, side product of CTC is to predict the alignment, which is the most probable input-long sequence that specifies hard aligning relationship between the input and target units. As there are multiple potential aligning sequences (called paths) that are equally considered in CTC formulation, the choice of which path will be most probable and become the predicted alignment is always uncertain. addition, it is usually observed that the alignment predicted by vanilla CTC will drift com- pared with its reference and rarely provides practical functionalities. Thus, the motivation of this work is to make the CTC alignment prediction controllable and thus equip CTC with extra functionalities. The Bayes risk CTC (BRCTC) crite- rion is then proposed in this work, in which customizable Bayes risk function is adopted to enforce the desired characteristics of the predicted alignment. With the risk function, the BRCTC is general framework adopt some customizable preference over the paths in order to concentrate the posterior into particular subset the paths. In applications, we explore one particular preference which yields models with the down-sampling ability and reduced inference costs. By us- ing BRCTC with another preference for early emissions, we obtain an improved performance-latene trade-off for online models. Experimentally. the proposed BRCTC, along with trimming approach, enables us to reduce the inference cost of offline models by up 47% without performance degradation: BRCTC also cuts down the overall latency of online systems to an unseen leve
INTRODUCTION
Alignment
Reference Alignment
CTC eritterier
Input
Input Index-
(a) CTC paths
(b) Vanilla CTC Posterior
(c) BRCTC Posterior (ours)
Figure 1: (a) An intuitive explanation of CTC paths. 2 is the blank symbol. Each path suggests hard alignment between the input and target. (b) Posterior of an offline vanilla CTC ASR system. Different colors mean different units. The predicted alignment drifts away from its reference? but the predicted non-blank token sequence is correct. (c) Posterior of a BRCTC ASR system that adopts the method in section 3.3 All blank spikes squeezed to the earlier time stamps.
Centence-to-Sequeeee (seq2seq) tasks have attracted broad interest and achieved great progress in multiple applications in the past few decades. Connectionist Temporal Classification (CTC) (Graves) et al. 2006) is fundamental criterion for seq2seq tasks. The CTC criterion was initially proposed
means corresponding authors. Reference alignment ohtained by deep neural setwork-hidden Markov model (DNN-HMM) system.
Cooee release:

---

Published as a conference paper at ICLR 2023
for automatic speech recognition (ASR) but its usage has been extended to many other tasks like machine translation (MT) (Qian et al. 2021 Gu & Kong 2020; Huang et al. 2022). speech transla- tion (ST) (Yan et al.) [022] Chuang et al.] 2021; Liu et al. (020), sign language translation (Wang eet al. 2018, Guo et al. 2019; Camgoz et al., 2020). optical character recognition (OCR) (Graves & Schmidhuber] 2008). lip reading (Assael al. (017). hand gesture detection (Molchanov al., 2016) and even robot control (Shiarlis et al. 2018). Research on CTC is of wide interest, as many advanced systems for seq2seq tasks are based on CTC (Yao et al. 202I), its extensions (Graves) 2012 Sak et al., 2017, HHiuuchi et al. 2020 Qian et al. 2021) and its hybrid with irrention-baser architectures (Watanabe et al. [017]] Yan et al. 2022).
In CTC, each input unit is explicitly aligned to either a target unit or a blank symbol. During training, all of these potential aligning sequences (called paths) are enumerated and their posteriors are summed and maximized, which is equivalent to maximizing the posterior of the target sequence. Fig[aa gives an explanation of the paths in CTC. Besides predicting the target sequence, another functionality of CTC is to predict the input-target alignment. Unlike the attention-based methods Chaa et al., 2016; Vaswani et al., 2017) that softly predict the aligning relationship by attention weights, CTC predicts hard alignment. Usually, there is path whose posterior is dominantly larger than the others (Zeeer et al. (022)). so this dominant path is considered the predicted hard alignment between the input and the target sequences. In CTC implementation, unit-level classification over all possible target units is conducted for each input unit to obtain the posterior of each path. Fig D demonstrates the dominant posterior of the predicted alignment by plotting the unit-level posteriors.
Since predicting any path will yield the correct target sequence, the vanilla CTC is designed to treat all paths equally. However, this equality for paths will result in uncertainty about which path will be selected as the predicted alignment. Also, both our experiments (see Appendix K} and literature (Sak) al. 2015) show that there a disagreement between the predicted alignment and its reference (see F b), which limits its usage in real applications. Thus, the motivation of this work is to control the CTC alignment prediction, making it certain and functional. Specifically, instead of pursuing the accuracy of alignment prediction, e.g., for CTC segmentation (Kürzinger et al. (020). this work intentionally selects the path with customizable characteristics as the predicted alignment.
This paper proposes novel Bayes risk CTC (BRCTC) criterion to make CTC alignment prediction controllable. To express our preference for the paths with the desired characteristics, a Bayes risk function is adopted to weigh all paths during training. To be more detailed, the forward- forward-backward algorithm of the original CTC is revised into a divide-and-conquu divided into several exclusive groups according to a customizable property, and the path groups with more preferred property will receive larger risk values during training. Same as the vanilla CTC, BRCTC can preserve the models' transcription ability, as it considers the posterior of all paths during training. However, the alignment prediction from BRCTC will additionally obtain the desired characteristics due to the adoption of the risk function. Note the designs of how the paths are grouped and what risk value is assigned to each path group are customizable, so the exact functionalities can be ailor-made according to specific applications, such as offline and online scenarios.
manner: the paths are firstly
In applications, the BRCTC provides novel solutions to two key problems of seq2seq tasks. For offline systems, BRCTC can help to down-sample the intermediate hidden representations so that the mismatch between the input and target lengths is alleviated and the inference cost is significantly reduced. For online systems like streaming ASR, BRCTC provides a better trade-off between tran- scription quality and latency. Besides ASR, the proposed BRCTC criterion can also be generalized to other seq2seq tasks like MT and ST. Experimentally, BRCTC can cooperate with a trimming approach to achieve up to 47% inference cost reduction for offline systems without degradation in transcription performance; it can also build online systems with extremely low overall latency that can hardly be achieved by vanilla CTC.
Our main contributions are listed as follows: (1) Bayes risk CTC (BRCTC), an extension of CTC, is proposed customizable approach to achieve controllable CTC alignment prediction. To the best of our knowledge, this is among the first works which achieve alignment control for CTC-based models without external information. (2) With various intentional designs of the risk functions, BRCTC can significantly reduce the inference cost (by up to 47% relative) and overall latency (to 302ms, up to 30% relative) for offline and online models respectively. (3) Strong experimental evidence is provided in this work to show that high-quality CTC target predictions can be obtained from CTC/ / BRCTC posteriors which do not necessarily encode accurate alignment prediction.
2

---

Published as a conference paper at ICLR 2023
2 REVIEW ON CONNECTIONIST TEMPORAL CLASSIFICATION (CTC)
Seq2seq tasks are transcribe the input sequence x I..... Xy] to the target sequence [[,,.., I,, where any is vector (e.g., features token embeddings) while any token 4, belongs to a known vocabulary L. T and are the lengths of input and target respectively, Unless other specified, our discussion is temporarily restricted to ASR. Generalizing the concept of CTC to MT and ST tasks needs more discussion, which is presented in section
2.1 TRAINING PROCESS AND ALIGNMENT PREDICTION
CTC (Graves al. 2006 is a widely used criterion seq2seq tasks. Following the Bayesian decision theory, CTC tries to maximize the posterior P(Ixx) during training. Instead of maximizing directly, CTC maximizes the summed probability of all paths (see Fig(Taa) Note as the blank
symbol and extend the vocabulary L' = C J ( {Ø). Any symbol sequence π = [TI..... कr E L is path if (xx)) 1, where B is a deterministic mapping that removes all Ø and repetitive tokens but preserves the repetitive tokens separated by Ø (e.g., ((BaaZabb) = aab). Thus, to maximize the posterior P(][x) is equivalent maximizing the summed posterior of all paths:
P(][x) =
Σ p(T()) =EB-1())
(1)
where (1) is the set of all paths. Next, to compute the posterior p( p(]]x) of each path æ, unit-level posterior over L' is computed for each input unit, which yields the CTC posterior y [y' .y.....y"]] Here each element y' is distribution over L' at step and yn, is the posterior for element xy. So the path posterior is formulated
xxxx))=
II P( x) R II p(mdx)
II y'. t=1
t=1
t=1
where the context Ti:t-1 is discarded in the approximation concerning the conditional independence assumption of CTC.
As demonstrated in Fig any path represents an aligning relationship between the input se- quence 1 and the target sequence x. Commonly, the CTC posterior is peaky (Zeyer et al. 202T] and the posterior of one certain path is dominantly larger than all others. To this end, the path with the highest posterior is usually considered the predicted alignment during training: ali(1, x) = arg max 10
2.2 FORWARD-BACERN BACKWARD ALGORITHM
Since the number of possible paths BI) will grow exponentially with and U increasing. directly enumerating all paths and their corresponding posteriors is impractical. As an alternative, the forward-buckward algorithm enables the training objective of CTC to be computed efficiently.
The first step of the an extended sequence I' = [Ø, ..., Ø, lu, Ø] by inserting a Ø between every two non-blank tokens as well as the start and the end of the sequence so that | 2U + 18 Then, define the forward variable att,, v), (1 < t < T, < v < 2U + 1) as the summed posterior all path prefix The that are aligned with the prefix of the expanded sequence I' symmetrically, define the backward 1) as the summed posterior of all path suffix ver that aligned with the suffix of the expanded sequence I' f'.:2++1:
algorithm is to extend the label sequence 1 [4...., [u] into
variable B11 v). (1 < S T, < < 20 + 1)
T II U
a(t,v)= = Σ
EI
II
D
B(t,o)
Σ
(3)
d=1
15 1-
41)
f'tt
ne-th
With any fixed and v, the summed probability of all paths whose t-th elements He exactly V. can be represented by the forward and backward variables as below, which is also termed as the occupation probability:
T
a(t, -B(t,
Σ p() FE8-1m
Σ
(4)
y/.
(1) 1'=l
T
$1 is the length function.
3

---

Published as a conference paper at ICLR 2023
In addition, for any constant 2 t I T. the choices of for any possible path E B B='(0) (I) restricted to the elements in I' and these choices are exclusive Thus, enumerating all index e with Eq will consider all possible paths and then the training objective of CTC can be written as:
20+1
20+1 alt, v) 3(t,
P(Ix))
Σ p(T - Σ Σ #EB-
p( = Σ
(5)
Dig
REB-1a
T
The computation of the att, v) and B(t, v) is recursive and the CTC gradients can also be computed in close form using the forward and backward variables. Details are presented in Appendix A
3 BAYES RISK CTC
In this part, a general formulation of the proposed Bayes risk CTC (BRCTC) criterion is presented in section 3.1 Examples about how paths be grouped fit the presented in section 3.2 Finally, we demonstrate how the proposed BRCTC with customizable risk designs can be used to tackle two different practical problems in section 3.3 and section 3.4
process
3.1 GENERAL FORMULATION
CTC prediction has two functionalities: predicting the target sequence I and predicting the hard alignment ali(1, x) between the input and the target sequences. The former is implemented by dis- criminating all paths π E (I) from the sequence set L' since feeding each path into B will yield the target I. In vanilla CTC, however, no constraint posed to the latter since Eq treats all paths equally and the choice of the dominant path, a.k.a. the alignment, is left unpredictable. To make the alignment prediction controllable is exactly to break this equality and intentionally select the desired paths among B B='(1) (I). To this end, Bayes risk function r(T) is adopted to enforce the characteristics of the desired paths. The modified CTC objective can be written as:
Jueee (l, (l,x) = Σ [p(T|x) r[[T]]
(6)
xEB-1())
The revised objective is termed Bayes risk CTC (BRCTC) due to the adoption of the Bayes risk function. Note when r(T) = 1, BRCTC becomes equivalent to vanilla CTC. Since BRCTC is
still maximizing the posteriors of the paths in (I) ideally it will preserve the models' target
prediction performance.
Directly applying the risk function to each path is still prohibitive as enumerating all possible paths is computationally impractical. To this end, the forward-backward algorithm in the vanilla CTC inherited to efficiently compute the BRCTC objective. Inheriting the forward-backward process will lead the design of the risk function to a divide-and-conquer paradigm. Under this paradigm, designing the risk function equivalent to specifying two things: 1) how the paths are divided into groups and 2) what risk values are assigned to each path group. Here the only requirement to achieve compatibility between BRCTC and the forward-backward process is that the summed posterior within any group can be fully represented by the the desired characteristics are about some specific properties of these paths (e.g.. the largest index of all non-blank elements within the path). Assume f(x) is the concerned property of path R, then all paths that satisfy (TT = r can form a group, where T is a possible value of the concerned property. Then, all paths in the same path group will receive the same risk value. Note (T) as a function of r. which is the shared risk value within the group as replacement of r(T). Thus, the BRCTC objective is to enumerate all path groups with the corresponding risk value ry (T) like below. A more detailed explanation is provided in Appendix B
variables. Formally,
Jorene (1, x) = Σ Σ [[(T(x)-(TT]]
Σ
Σ
[[(T|x))r Eba(7)
Σ
[(T[x)]
1mB-1m
-EB-1(0
[())=
3.2 EXAMPLES ABOUT HOW PATIIS ARE GROUPED
Here we provide two examples of how the paths are grouped to be compatible with the forward- backward process. The first example is in Eq[4 in which paths can be grouped by their choices of t-th element ,. So the occupation probability is also the summed posterior of path group, which can be naturally represented by the forward and backward variables.
Here we should assume the blank symbols in 1' are different from each other avoid confusion.
4

---

Published as a conference paper at ICLR 2023
Ref. Ali.
Encoderr
BRC TC
Trinming
Decoder
BRCTC
Farter
00
Stan
Prodition
(a) Offline
(b) Online model inference
(c) BRCTC achieves earlier emissions than vanilla CTC
down-sampling
Figure 2: (a) Down-sampling process using BRCTC criterion. h is trimmed before being fed into the decoder. (b) Inference process of the online model and its three exclusive sources of latency. DCL: data collecting latency. CL: computational latency. DL: drift latency. (c) Posteriors of online vanilla CTC BRCTC systems and the reference alignments.
Secondly, more complicated but useful example to group the paths according to the ending point of certain non-blank token. Since any path is aligning relationship between the input and target sequences, it is common to ask when the prediction of given non-blank token ly 2u finished within this path. Formally, with the known constant u, the concerned property of can be defined as T = (T)) = arg max; s.t. Te = Lu 2u If so, the summed probability each path group can be formulated by the forward and backward variables below. A detailed explanation for this formulation provided in Appendix C
p( a(T, 2u) [T, 2u)
s.t. 3(T,2u) = [B(T,2u) B(r + 1, Vn if (8(T,
Σ FEB-im (ujjrr
Otherwise
Combine Eq for any constant u. path groups with different T and the corresponding risk values (T) are enumerated as below. This strategy of grouping paths is adopted in section 3.3 and 3.4
(r())) alT, 2u) B(T,2w)
Juneie (1, (1,x)
VT.
3.3 APPLICATION: DOWN-SAMPLS
A key problem for series of offline seq2seq tasks (like ASR, ST) is that the input sequence is much longer than the output sequence, a.k.a., x > |I (Gado et al. 2021). If an encoder is used to process into the encoder hidden output h, this can partially be interpreted as |h] [1]. For the mainstream encoder-decoder architectures in seq2seq tasks, the inference cost of the decoder is highly correlated with h/. Thus, the over-length of h leads to redundancy in the inference computation. The proposed BRCTC is capable of reducing the length of the h to save inference costs. The workflow is shown in Fig 2 a. The blanb spikes of CTC posterior y are pushed to the earlier time-stamps using BRCTC, then the posterior is adopted as a reference to trim the h into the shorter h'.
Pushing all non-blank spikes to the early input units (like in Fi a) requires the predicted alignment to finish its prediction of i early as possible. Consider the latter situation in section 3.2 and set u = U, the concerned property t is the input index where all non-blank elements have completed in the paths. Subsequently, paths with smaller are more preferred so the risk values should be larger. We adopt (T) - as the risk function in this application, where À is an adjustable hyper-parameter called risk factor. Finally, the training objective updated as:
Σ - a(T, 20) B(T, 20)
Joer x1.x)=
(10)
yh.
Provided the continuous and highly confident blank predictions in like in Fig it is reasonable
assume the corresponding elements in h contain nearly no useful semantics and can be trimmed
Formally, h is trimmed to h' [h,.... .... hm+ nD). where is the maximum value of s.t. V L'
t, v. 99%; D 5 is a small integer to keep safety margin for the trimming. The trimmed hidden output is fed into downstream architectures as replacement of h.
'There might be some repetitions I, but we still consider all tokens I are different for simplicity. The correlation between any non-blank nı and l, is clear so this notation will not lead to confusion. "Usually, transforming into y only adopts simple linear classifier and the softmax function.
5

---

Published as a conference paper at ICLR 2023
3.4 APPLICATION: PERFORMANCE-LATENTT TRADE-OFF
Another problem for online seq2seq systems (e.g.. streaming ASR) is the trade-off between the transcription performance and the system latency. For online systems with constrained context, better transcription quality requires more context, which, however, will result in longer lateney. Assume the input sequence is fed into the system chunk-by-chunk (Shi et al. 202T), this paper defines the total latency as the sum of three exclusive sources as shown in Fig b: Data collecting latency (DCL): the time to wait before the input signal forms a chunk. This depends on the model design. Computational latency (CL): the time consumed by model inference. Only this latency depends on the hardware performance. Drift latency (DL): the difference between the input indexes when an event starts and when its prediction ends" This is learned during the model training. The formal definition of the latency sources and further explanation are in Appendix F)
The proposed BRCTC can guide the model to emit non-blank spikes at early input indexes (see c) so that the drift latency (DL) can be reduced. As a benefit, it provides a better overall performance-lateney trade-off than the vanilla CTC systems (more discussion is in section 4.3 Formally. if a non-blant token lu is required to be emitted earlier, the concerned property т is exactly the ending point of its prediction within the path. Thus, a tailor-made training objective for the token l is:
Hecco (1, x, u) = Ee ((TTTT)/ a(T, B(T,, vi,
a(T, 2u) BB(T, 2u) (11)
s.t. T arg max
y
The design of the group risk function (T) is still the exponential decay function but with an extra bias r'. Without this bias, the absolute values of Jnee (1, x, u) will be unbalanced for different token Iu To guide every token lu to emit earlier requires the considerations of all u. So the global training objective to maximize is then transformed into:
U
Jorese (1, x) = U
Σ
log Jhruc (1, x, u)
(12)
4 EXPERIMENTS
Our experiments are mainly designed to examine the two applications of the proposed BRCTC criterion. The experimental setup is introduced in section 4.1 The BRCTC down-sampling method
and performance-latenny trade-off are validated in section 4.2 and section 4.3 respectively. BRCTC
is generalized to MT and ST in section 4.4 Visualization and its analysis are section 4.5
4.1 EXPERIMENT SETUP
Datasets: Experiments are mainly conducted for ASR, but MT and ST are also included. For ASR, the experiments are on Aishell-1 (Bu et al., 2017). Aishell-2 (Du al. 2018). Wenetspeech (Zhang ett al. 2022) and Librispeech (Panayotov et al.) 2013). The volumes of these datasets range from
178 hours 10k hours. Librispeech is in English and the others are in Mandarin. For MT and ST,
IWSLT14 (Cettolo et al. 2012) and MuST-C-v2 En-De (Di Gangi et al. 2019) are adopted.
Models: For all tasks, Hybrid CTC/Attention model (Watanabe et al. 2017 Yan et al. 2022) is evaluated. For offline ASR, Transducer (Graves) 2012) plus CTC architecture is also evaluated.
Training and Decoding: For training. a two-stage method is proposed for the offline down- sampling application and vanilla CTC is directly replaced by BRCTC in the online systems. For offline decoding, default algorithms with decoder recurrence (Watanabe et al., [017]] Graves [012]
Yan et al. 2022) are used to demonstrate the reduction of inference cost; for online decoding, CTC greedy search is adopted to better measure the transcription performance and latency. Our BRCTC implementation depends
the differentiable finite-state transducer" 19) (Hannun et al. 2020))
Evaluation Metrics: To compare the transcription performance. CER (for Mandarin) and WER (for English) are reported for ASR task; detokenized case-sensitive BLEU (Post 2018) is reported for MT and ST tasks. To compare the computational cost of offline models, the real-time factor (RTF), the down-sampling factor (DSF, a.k. [h'//h]] /hh) and its oracle (a.k.a., |1/||h] are reported. To ana- lyze the latency, time is measured for hardwarr--nddeendde of hardware latency. Implementation details are in appendix D and E for reproducibility.
latency while the RTF is the indicator
7 "See Fig e, this latency is only computed with the input unit indexes, not on the real-world timeline. UUsually, the path groups with r being equal close r' make most of the contributions in Eq 11 For different u, the T', along with the risk values for these major path groups, is different. So different 7' leads
unbalance in the absolute values of Jeree (1, x, u). As remedy, taking 7' as a hias ensures that the path group with maximum posterior always receives the risk of 1.0 regardless of u.
Our implementation is based K2 toolkit: http:///ithub..om//2faaau/

---

Published as a conference paper at ICLR 2023
26
05
Vonila CTC
Tinimed
DGF
Oroole
Vanille CTC
BROTO
CERNEERS
SPOTII
1975
Waralapaech Librapaach
(a) CER/ / WER (1%)
(b) DSF (L) & Oracle
(c) RTF (4) vs. Beam Size
Figure 3: Transducer performance on various ASR tasks w/o BRCTC down-sampling,
4.2 RESULTS ON BRCTC DOWN-SAMPLING FOR OFFLINE ASR SYSTEM
This part evaluates the effectiveness of BRCTC down lowm-sampling method on the offline ASR task. Our main results of the Transducer plus CTC model are plotted in Fig The tendency for the Hybrid CTC/Attention model is similar. The complete results are in Appendix G Firstly, the transcription performance of the Transducers plus CTC or BRCTC is reported in Fig.ja. As suggested, with datasets in varying volumes and different languages, adopting BRCTC does not result in notice- able variance in the transcription quality. Secondly, Fig b. demonstrates the effectiveness of the down-sampling process. The encoder hidden output h trimmed by least 60% and the trimmed h' is roughly as 1.5~2.5 times long as the I only To So the length mismatch of input and output sequences is significantly alleviated. Thirdly, since the inference cost of the decoder highly depends on the length of h, replacing h by h' will reduce the inference cost remarkably. For the label- synchronous decoding of hybrid CTC/Attention (Watanabe et al., 2017), the computational cost of each decoding step is reduced since the expense of attention computation will be smaller. For the name-synchronout decoding of Transducers (Graves) 2012). the inference cost is reduced since the number of frames in decoding is reduced linearly along with h// As the inference cost of the decoder is also sensitive to the beam size, we sweep the beam size from 2 to 15 on Librispeech test-other set. As Fig suggests, the relative inference cost reduction becomes larger with the beam size growing.
4.3 RESULTS ON BRCTC PERFFORAANCE-LTTENCY TRADE-OFF FOR ONLINE ASR SYSTEM
This part evaluates the trade-off between the transcription performance and the latency w/o the adoption of BRCTC on streaming ASR task. Our main observation is shown in Fig Fig 4 a reflects the trade-off between the DCL and DL. Firstly, the models designed with smaller DCL will encounter larger DL since the models are not confident with the highly restricted context and will wait for longer input before decisions. This DCL-DL relationship suggests the system with extremely low overall latency is not feasible by only designing low DCL (a.k.a,, small chunk size). Secondly, the adoption of BRCTC achieves consistent DL reduction since tries to push the spikes of all non-blank tokens to be emitted earlier. Thirdly, the DL gap between vanilla CTC and BRCTC increases along with DCL increasing, since a longer chunk provides higher performance ceiling for BRCTQ while the DL for vanilla CTC is always larger than Oms to explore all accessible contexts. 4b reflects the trade-off between the DCL and CER. As expected, CTC / BRCTC systems with smaller DCL degrade more in their transcription quality due to the more restricted context. The adoption of BRCTC also results in CER degradation compared with its baseline, since earlier emissions will also restrict the accessible context.
Summarizing Fig a&b leads to Fig c, in which the trade-off between all hardware-independent latency (DCL + DL) and the transcription quality (CER) is demonstrated. The adoption of BRCTC provides several benefits: 1) building the system with extremely low total latency (only 280ms, the pink circle), which is not feasible for vanilla CTC due to the scesaw-like relationship between DCL and DI. 2) Achieving both lower CER and smaller latency than vanilla CTC (the green circles). Specifically, BRCTC provides an alternative solution for online applications: increasing DCL with a larger chunk size and reducing DL using BRCTC to meet the latency budget and achieve better overall performance-latenccy trade-off.
The alternative solution of using a larger DCL (larger chunk size) also provides an extra benefit from the perspective of hardware. As suggested by CCen et al. (2021) and also presented in Fig d. a larger chunk size can better explore the advantage of parallel computing and then achieve lower RTF.
10 Note that x has heen sub-sampled by 4 times when being encoded into Ré et al., 2018 Then our BRCTC down-sampling method is conducted on the h. The two down-sampling methods are used together. TThe DL. can he negative due the look-ahead mechanism of the model.
7

---

Published as a conference paper at ICLR 2023
400
200 Al(ma)
CERI/S
CER/N)
Vanill CTC
Venilla CTC
BRCTO (ours)
BRCTC (ours)
DCLima)
640
0
DCL(mS) 640
DCL DL/ms)
640
DCLims)
640
(a) DCL vs. DL
(b) DCL vs. CER
(c) DCL+DL vs. CER
(d) DCL vs. RTF
Figure 4: Aishell-2 trade-off' between transcription performance and latency w/o BRCTC.
This means the hardware may have more time being idle switching among multiple processes during serving. Using a larger chunk size will slightly increase CL. but our observation from Fig Q still holds. Appendix H provides more detailed results.
4.4 GENERALIZING BRCTC TO ST
This part demonstrates that the proposed BRCTC can also be generalized to other seq2seq tasks like ST. The ST results are in table [ Consistent with our ASR experiments, adopting BRCTC town-sampling in offline ST 1) achieves comparable transcription quality with the vanilla system; 2) reduces the length of h by 42% and accelerates the inference by 27%. The proposed BRCTC is also generalized to MT task in Appendix
Transcription Quality (BLEU1)
Down-sampling Effectiveness
System
COMMON
IEE
DSF())
RIF()
Atlention Vanilla CTC. Yan
2022
24.3
28.3
0.51
Attention BRCTC (ours)
29.1
28.7
0.58 /0.25
0.37 (-27%)
Table I: ST performance on MuST-C-V2 English-Germant dataset w/o BRCTC down-sampling
4.5 VISUALIZATION
Following Graves et al. (006)_ Fig 5 compares the evolution of the CTC distributions and the their gradient. the beginning of training, the distribution is roughly unified and the gradient is smooth along the time-axis. Note the gradient for all non-blank tokens is similar only except for the last token (the green line and the red arrow): for the last token, the BRCTC gradient for a larger frame index will be penalized in order push the last token to the earlier indexes. After epochs of training, the gradient will localize. Although the models are not sure what the non-blank tokens ex- actly are, the places where the non-blank tokens will spike roughly determined. At this stage, the BRCTC model has already known the spikes should all happen at the very left. After convergence, the gradient is close to zero and the CTC distributions become peaky. For BRCTC, the spikes all concentrated on the left as expected. We also find the down- lown-sampling process is implemented mainly by the last two encoder layers. More visualization is provided in Appendix
DISCUSSION
Correlation between target prediction and alignment prediction: predictions in high quality can be obtained from CTC posteriors which do not necessarily encode accurate alignment prediction. Theoretically, all paths in Eq Jrre treated equally in vanilla CTC and the convergence of vanilla CTC can be achieved with any path being dominant, there are mul- tiple solutions for the alignment prediction sub-task. Additionally, selecting each of these solutions can hardly interfere with the transcription quality, since each of the paths will yield the correct target prediction after the blank and repeat removal. Experimentally, our experiments suggest that vanilla CTC systems with significantly drifted alignment prediction can still preserve the transcription qual- ity (see Appendix Kj experimental in section 44. further demonstrate that BRCTC achieves com- petitive transcription results like vanilla CTC with extremely unreasonable alignment prediction. Monotonic assumption, MT & ST tasks and rearranging ability: An underlying assumption of CTC is the monotonic assumption, which requires that, if any E; and zy are mapped to two non- blank tokens 4, and l,.' respectively with f < f', then there must have u < u'. Conventionally, this assumption restricts CTC from being applied to seq2seq tasks whose alignment is not monotonic, like MT and ST. However, this constraint can be softened by deploying self-attention encoder archi- tectures which allow implicitly reorder the semantics of h and make it roughly monotonic with respect to (Chuang et al. 2021). Besides, our method in section 3.3 can also be viewed as a pro- cess to rearrange the semantics of h even though the relative order of non-blank spikes is kept. To
This work observes that target
8

---

Published as a conference paper at ICLR 2023
Fht Stant
10
(0.03
0.0
25 Cocche
0.00
Figure 5: Evolution of CTC distribution y and the corresponding gradients on logy. BRCTC with the down-sampling method is used.
sum up, although this work mainly addresses the CTC training criterion, we believe the rearranging ability of attention- based neural networks is key factor in BRCTC's functionality.
RELATED WORKS
The proposed BRCTC is extension of CTC. There are several existing criteria and frameworks that can be partially viewed as CTC extensions. To alleviate the conditional independence assump-
tion of CTC yields Transducer (Graves] 2012), RNA (Sak et al.] (007) and their path-modine
ex- tensions (Mahadcokar et al. [202]] Kuang et al. 2022 Shinobara & Watanabe [022] Yu et al.) [202]]
[021]; ; to equip CTC with discriminative ability yields LF-MMI with CTC topology (Ha-) dian et al. 22018); to exploit the non-auto- regresseeeee nature of CTC yields the non-auto-regressive
Kim et al.
ASR (Higuchi et al. 2020) and MT (Qian et al., 2021 architectures; to explore various topologies
yields multiple CTC variants (Zhao & Bell| 2022 Laptev et al. 2022). To exploit partially labeled
and multi-labeled data yields STC (Pratap et al. (022) and GTC (Moritz et al. 202T). None of the works aforementioned try to control the alignment prediction of CTC. Some pioneer works can be interpreted as preliminary attempts to control CTC alignment prediction. [hhorbani et al.] (2018); Kurata & Audhkhasi (2019) achieve aligned CTC posterior spikes among heterogeneous models by learning the spike time-stamps of teacher model. However, the spikes of the teacher model itself cannot be controlled. Zeyer al. (2020); /Senior et al. (2015); Plantinga & Lossier-uusierrpp (2019) im- prove CTC and Transducer models by injecting external alignment supervisions, but obtaining these alignments consumes extra effort. By contrast, the proposed BRCTC is customizable for general purposes in an end-to-end fashion and depends on neither teacher models nor additional informa- tion. Previous literature has also discussed the basic properties of CTC from multiple perspectives, like its peaky behavior (Zeyer et al. 2021), alignment drift (Sak et al. 2015) and the properties of blank symbol (Zhao & Bell| 2022. Bluche et al. 2015). To our knowledge, The correlation between the target prediction and alignment prediction has not been scriously discussed before this work.
7 LIMITATION
The proposed BRCTC has a known limitation. The gradient of BRCTC is obtained by the naive chain rule (as the barkard process only adopts addition and multiplication operations, this is feasible). Thus, the gradient computation needs trace back the forward-backward process, which then results in an increase in the training cost.
8 CONCLUSION
This work is motivated by our experimental observation that a CTC system with drifted alignment prediction still preserves competitive transcription ability, which inspires us the possibility of mak- ing CTC alignment prediction controllable to serve the needs in various applications. An extension of CTC called BRCTC is then proposed to select the predicted alignment among all possible paths and the design of the risk function is left customizable to fulfill the task-specific needs of different applications. For the offline model, the adoption of BRCTC leads to inference cost reduction since the length of the intermediate hidden output can be down-sampled. For the online model, BRCTC provides better performance-lateme trade-off. We verify the effectiveness of the proposed BRCTC on multiple sequence-to-sequeeee tasks with various datasets, languages, and model architectures.
9 ACKNOWLEDGEMENT
This work used the Extreme Science and Engineering Discovery Environment (XSEDE) TTwws let al.) (2014). which is supported by National Science Foundation grant number ACI- 1C1-52...... Specifically, it used the Bridges system (Nystrom et al. 2015)), which is supported by NSF award number ACI-1445606, the Pittsburgh Supercomputing Center (PSC).

---

Published as a conference paper at ICLR 2023
10 REPRODUCIBILITY STATEMENT
We are taking various measures to ensure the reproducibility of our experiments:
Code is released as the complementary material of this submission.
Details of all experiments are clarified in Appendix B
REFERENCES
Yannis M Assael, Brendan Shillingford. Shimon Whiteson, and Nando de Freitas, Lipnet: End-to- end sentence-level lipreading. GPU Technology Conference, 2017.
Théodore Bluche, Hermann Ney, Jérôme Louradour, and Christopher Kermorvant. Framewise and ctc training of neural networks for handwriting recognition. 2015 13th international conference on document analysis and recognition (ICDAR), pp. 81-85. IEEE, 2015.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), pp. 1-5, 2017.
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6224 6228, 2018.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language trans- formers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit3: Web inventory of transcribed and translated talks. In Conference of european association for machine translation (EAMT), pp- 261-268, 2012.
William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960-4964, 2016.
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang. Xin Jiang, and Qun Liu. Litevl: Efficient video-language learning with enhanced spatial-temporal modeling- In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, 2022, pp- 7985-7997. Association for Computational Linguistics, 2022.
Xie Chen, Yu Wu, Zhenghao Wang, Shujic Liu, and Jinyu Li. Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. In ICASSP 2021 202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5904- 5908, 2021.
Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. M3ST: mix three levels for speech translation. CoRR, abs/2212.03657. 2022.
Shun-Po Chuang, Yung-Sung Chuang, Chih-Chiang Chang, and Hung-yi Lee. Investigating the reordering capability in CTC-based non-autoregressive end-to-end speech translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1068-1077, Online, August 2021. Association for Computational Linguistics.
Siddharth Dalmia, Dmytro Okhonko, Mike Lewis, Sergey Edunov, Shinji Watanabe, Florian Metze, Luke Zettlemoyer, and Abdelrahman Mohamed. Legonn: Building modular encoder-decoder models. arXiv preprint arXiv:2206.03318 2022.
10

---

Published as a conference paper at ICLR 2023
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST- C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume oong and Short Papers), pp. 2012-2017, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. Aishell-2: Transforming mandarin asr research into industrial scale. arXiv preprint arXiv: /000./888..00 10583, 2018.
Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. CTC-based compression for direct speech translation. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pp. 690-696. Association for Computational Linguistics, April 2021.
Shahram Ghorbani, Ahmet E. Bulut, and John H.L. Hansen. Advancing multi-accented Istm-ctc speech recognition using domain specific student-teacher learning paradigm. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 29-35, 2018.
Alex Graves. Sequence transduction with recurrent neural networks.
arXiv preprint
2012.
Alex Graves and Jürgen Schmidhuber. Offline handwriting recognition with multidimensional re- current neural networks. Advances in neural information processing systems, 21, 2008.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Jiatao Gu and Xiang Kong. Fully non non-autoregressive neural machine translation: Tricks of the trade. arXiv preprint arXiv:2012.15833 2020.
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: annvolution-uugmentn Transformer for Speech Recognition. In Proc. Interspeech 2020, pp. 5036-5040, 2020.
Dan Guo, Shuo Wang, Qi Tian, and Meng Wang. Dense temporal convolution network for sign language translation. In IJCAI, pp. 744-750, 2019.
Hossein Hadian, Hossein Sameti, Daniel Povey, and Sanjeev Khudanpur. End-to-end speech recog- nition using lattice-free mmi. In Proc. Interspeech 2018, pp. 12-16, 2018.
Awni Hannun, Vineel Pratap, Jacob Kahn, and Wei-Ning Hsu. Differentiable weighted finite-state transducers. arXiv preprint arXiv:2010.01003: 2020.
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask ctc: Non-autoregressive end-to-end asr with etc and mask predict. arXiv preprint arXiv:2005.08700, 2020.
Chenyang Huang, Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. Non-autoregressive translation with layer-wise prediction and deep supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10776-10784, 2022.
Jaeyoung Kim, Han Lu, Anshuman Tripathi, Qian Zhang, and Hasim Sak. Reducing Streaming ASR Model Delay with Self Aligmment. In Proc. Interspeech 2021, pp. 3440-3444 2021. doi: 00.214377nnterspeechh200111 322.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. Pruned RNN-T for fast, efforr-efficiiet ASR training. In Proc. Interspeech 2022, pp. 2068-2072, 2022. doi: 10.21437/Intesspeecch.o 20330.000.0

---

Published as a conference paper at ICLR 2023
Gakuto Kurata and Kartik Audhkhasi. Guiding etc posterior spike timings for improved posterior fusion and knowledge distillation. In Proc. Interspeech 2019, 2019.
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. Cic- segmentation of large corpora for german end-to-end speech recognition. In International Con- ference Speech and Computer, pp. 267-278. Springer, 2020.
Aleksandr Laptev, Somshubra Majumdar, and Boris Ginsburg. CTC Variations Through New WFST Topologies. In Proc. Interspeech 2022, pp. 1041-1045, 2022.
Jaesong Lee and Shinji Watanabe. Intermediate loss regularization for cte-based speech recogni- tion. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 0224-6228, 2021.
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengging Zong. Bridging the modality gap for speech- to-text translation. arXiv preprint arXiv:2010. 14920, 2020.
Jay Mahadeokar, Yuan Shangguan, Duc Le, Gil Keren, Hang Su, Thong Le, Ching-Feng Yeh, Chris- tian Fuegen, and Michael L. Seltzer. Alignment restricted streaming recurrent neural network transducer. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 52-59, 2021.
Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks. In CVPR, pp. 4207-4215 2016.
Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Semi-supervised speech recognition via graph- based temporal classification. In ICASSP 2021 2021 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 6548-6552. 2021.
Nicholas A Nystrom, Michael J Levine, Ralph Z. Roskies, and Ray Scott. Bridges: a uniquely flexible hpe resource for new communities and data analytics. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, pp. 1-8, 2015.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Daniel Park, William Chan, Yu Zhang, Chung-Chenn Chiu, Barret Zoph, Ekin Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019, Sep 2019.
Peter Plantinga and Eric speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 690-696, 2019.
Towards real-time mispronunciation detection in kids'
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, October 2018. Associ- ation for Computational Linguistics.
Vineel Pratap, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. Star temporal classification: Sequence classification with partially labeled data, arXiv preprint arXiv:2201.12208, 2022.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang. Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li.
Glancing transformer for non-autoregressive neural machine translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1993-2003, Online, August 2021. Association for Computational Linguistics.
Haşim Sak, Andrew Senior, Kanishka Rao, Ozan Irsoy, Alex Graves, Françoise Beaufays, and Jo- han Schalkwyk. Learning acoustic frame labeling for speech recognition with recurrent neural networks. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2280-4284 IEEF, 2015.
12

---

Published as a conference paper at ICLR 2023
Haşım Sak, Matt Shannon, Kanishka Rao, and Françoise Beaufays. Recurrent neural aligner: An encoder-decoder neural network model for sequence to sequence mapping. Proc. Interspeeck 2017, pp. 1298-1302, 2017.
Andrew Senior, Haşim Sak, Félix de Chaumont Quitry, Tara Sainath, and Kanishka Rao. Acoustic modelling with ed-cte-smbr Istm mns. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 604-609, 2015.
Yangyang Shi, Yongqiang Wang. Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le. and Mike Seltzer. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6783-6787, 2021.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning. pp. 4654-4663. PMLR, 2018.
Yusuke Shinohara and Shinji Watanabe. Minimum latency training of sequence transducers for streaming end-to-end speech recognition. In Proc. Interspeech 2022, pp. 2098-2102, 2022. doi: 10.21437/neerspeech.2222 10989.
J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, R. Scott, and N. Wilkins-Diehr. Xsede: Accelerating scientific discovery. Computing in Science & Engineering, 16(5):62-74, Sept.-Oct. 2014. ISSN 1521-9615. doi: 10. 20.100//CSE.201..8880. URL/doi. www.ccctttteeeeecctttccooo////// 11C9/MCSE. 2014.80
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- fion processing systems, 30, 2017.
Shuo Wang. Dan Guo, Wen-gang Zhou, Zheng-Jun Zha, and Meng Wang. Connectionist temporal fusion for sign language translation. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1483-1491, 2018.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1188)12400-225, 2017.
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, and Shinji Watanabe. Ctc alignments improve autoregressive translation. arXiv preprint arXiv:22/0.05200. 2022.
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen. Lei Xie, and Xin Lei. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. In Proc. Interspeech, Brno, Czech Republic, 2021. IEEE
Jiahui Yu, Chung-Cheny Chiu, Bo Li, Shuo-yiin Chang, Tara N. Sainath, Yanzhang He, Arun Narayanan, Wei Han, Anmol Gulati, Yonghui Wu, and Ruoming Pang. Fastemit: Low-latency streaming with sequence-level emission regularization. In ICASSP 2021 2021 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6004-6008, 2021.
Albert Zeyer, André Merboldt, Ralf Schlüter, and Hermann Ney. A new training pipeline for an improved neural transducer. Proc. Interspeech 2020, pp. 2812-2816, 2020.
Albert Zeyer, Ralf Schlüter, and Hermann Ney. Why does ctc result in peaky behavior? arXiv preprint arXiv:2 105. 14849, 2021.
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen. Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6182-6186. IEEF, 2022.
13

---

Published as a conference paper at ICLR 2023
Zeyu Zhao and Peter Bell. Investigating sequence- levell normalisation for etc-like end-to-end ast. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7792-7796, 2022.
Zhihong Zhu, Weiyuan Xu, Xuxin Cheng, Tengtao Song, and Yuexian Zou, A dynamic graph interactive framework with label-semantic injection for spoken language understanding. CoRR, abs/2211.00023, 2022.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX: DETAILS OF THE FORWARD-BACKWAN PROCESS AND THE CLOSE-FORM GRADIENT OF VANILLA CTC
This appendix describes the recursive computation of the forward-backward algorithm and the close form gradient of vanilla CTC.
For the forward variable a(t, v), the recursive process is:
all, v) [a(t - 1) + att - 1, - 1)] ui. [a(t- - 1, alt 1, v 1) + alt - 1- -2] -
if = Ø or = 1
-2]]
Otherwise
(13)
with the initial condition:
1)1, y
a(1,2) = y a(1, =0 Yv
(14)
Symmetrically, for the backward variable (tt v), the recursive process is:
(((+11,0)+ 1)] y/-
if 2 or 1'
3(t, v) = L))
3(t 1. 22] vh.
Otherwise
(15)
with the initial condition:
BT, 2U - 1) T.
B(T, 2U) = vt
3(T, v) x) 0,
Ve <
20
(16)
After all forward and backward variables are computed, the CTC gradient can be computed in close form. Firstly, for the posterior of any path its gradient w.r.t. the y is:
Op(Tx) Oyk.
T DT
II
= Tec
(17)
Oyl
1=++1
otherwise
Then consider the occupation probability in Eq and include the two products above into the for- ward and backward variables, its gradient w.r.t. the output y is:
p(T(x)
alt, v) ((t if E
(18)
Oyk
otherwise
Finally, enumerate all v like in Eq
the gradient of CTC is computed as:
p p(|x)
Σ
v)t,v) (t,,
(19)
Oyt.
vk
vElab(V,K)
where lab(I' k)) indicates where the i occurs: lab(l', {u l' k}
15

---

The first line the original definition of BRCTC, in which each path along with its risk value r(T) is considered and the product is summed for all paths. The second line is to group the paths with identical concerned property т. In this stage, the risk value for each path still r(T). The third line Published as a conference paper at ICLR 2023
B FURTHER EXPLANATION OF BRCTC GENERAL FORMULATION
This appendix provides a more detailed explanation of the BRCTC general formulation. The equa- tions provided below.
Jureie (1, x)
Σ (T(T(x))((1))
FEB (()
Σ [p(T|x) - [(T]]
REB-1(U)
[(H)-r
Σ
[p(x/x) (T)]
NEB 10 s(nyar
Eira(T)
[(T[x)]
(20)
WCB-1a
suggests the risk values for all paths within group are identical and only depend on the concerned property т. So the risk value changed from r(x) to (T(T) within the group. The final line is to
extract the common factor (T) out of the summation within the group. This suggests we can first
compute the summed posterior of all paths within the group and then apply the risk value for that group in one go.
we further provide a naive example to illustrate how paths are grouped and how the risk values assigned to each group. Assume the input length is 3 (index from I to 3) and the target sequence is [A, B). Then, all possible paths and their posteriors are listed in the first and the second rows of the table below respectively.
Example:
We also assume the concerned property - = f(T)) is the time-stapp when the prediction of token B finishes. Thus, the value of the concerned property for each path is listed in the third row of the table below.
Paths with the identical value should form a group. So the path ABØ forms a group with single element while the remained paths form another group. For paths within a group, the risk value should be identical. The risk value for each group Fi (T) user-defined. We simply set r, (2) = 1.0 and rg (3) = 0.8. Then the risk for each path r(T)) is then set in the fourth row of the table below.
Path(T)
ABØ
ABB
AAB
AØB
ØAB
p( x)
0.3
0.1
0.2
0.0
0.1
3
3
r(T) or (T(T))
1.0
0.8
0.8
0.8
0.8
Finally, formulating all these processes in equations will be:
J Jarae = (P(AB2)-r(AB)) + p(ABB) r(ABB) + P(AAB) r(AAB)
+P(A2B) ( (((A)) r(0)B) #Initial formulation
((ABB))r(ABB))
+ p(A2B) r(ABB) + p(AAB) ((((AAB)-(AABB))) r(@AB)
+ (I(AAB2) #grouping
(AAB) PAAA)) (3( + p(A0B) rr(3) + p(ZAB) ( (3)
+ +(p(ABZ)-r, #replace r(T) by (T)
= =[P(ABB) +p(AA))+((aBB p Ø AB)] (3)
+ +P(AAB2)) (2(() #extract common factors
=(0.1 + 0.2 0.0 0.1) 0.8 + 0.3 1.0 0.62
(21)
16

---

Published as a conference paper at ICLR 2023
C APPENDIX: THE EXPLANATION FOR GROUPING STRATEGY FORMULATION
This appendix explains the formulation in section 3.2 The property function is set to T = fa(T)- = arg max, S.L. n. Lu l' 2u-
Firstly, for the scenario T < T, the property function can be translated into the condition ,i 22
and + # l' 2mc Thus, the summed probability of the path group is formulated
Σ p(T
Σ
p(Txx)= =
Σ
II
TEB 1 (u))
TEB mm ey cy 2n 18 20
TEB 00)
Be 2 2 7-1 2ll2a
T
T
Σ
II
Σ
II
Uay
Vxy
CCB-1() P=11
0==1
71
2
a(T, 2u) B((,, 2n)
aa(T22u) B((T + 1,)2
(22)
Vz,
Set:
3(r,2u) 2u) B(T, 2u) B(T 1,,20) Vz.
(23)
The summed probability then becomes:
Σ p(a[x)= = a(T, 2u) B((T 2u)
(24)
y7,
KEB 10 (=(=)r
Secondly, T T suggests that:
p(xxx) = a(T, a((T.uu) 3((T, 2u)
Σ
(25)
Vi.
WCB 10
Ju(n)yy
To sum up, the summed probability of the path group with the given and u is
Σ p()[x)= a(T, 2u) B(T, 2u) 7
(26)
ve.
fu(s)=r
with
[8(r,2u) 2u) B(T + 1,2u) VT, if < T (8(T,200 2u),
A(r, 2u)
(27)
Otherwise
17

---

Published as a conference paper at ICLR 2023
D APPENDIX: THE TWO-STAGE METHOD FOR BRCTC DOWN-SAMPLING
Verilla
Encoder 2
Encoder
bi
Bayes Riee CTC Loss
Cross-Enispy
Loss
Trmming
y1
Beyes Rask CTC Loss
Downstream Loss
bi
Encoder
Atention Decodar
Encoder
Downstream Medules
First stage
Second Stage
Cross-Er
Transducer Loss
Vanilla CTC Lees
Cross-Erirve dtropy Loes
Jani Netwerk
Trans alion Eneader
Altantion Decandr
Decander
hg
hg
hg
tgt
Hybrid CTC/Attention (ASR, MT)
Transducer (ASR)
Hybrid CTC/Attention (ST)
Figure 6: An illustration of the proposed two-stage method for BRCTC down-sampling. This appendix describes the proposed two-stage method for BRCTC down-sampling. Details are described in o The first stage mainly follows the architecture in Watanabe et al. (2017): the encoder-decoder model is jointly optimized by both CTC loss and cross-entropy loss. The decoder part is adopted in order to be reused in the second stage. Intermediate CTC technique (Lee & Watanabe 202T) consistently adopted so that the whole encoder split into two parts with an identical number of layers, and the hidden outputs from both the intermediate layer (a.k.a., h2) and the final layer (a.k.a., h2) are supervised by the CTC criterion. In this stage, all CTC criteria are our proposed BRCTC. In the second stage, the weights of the two encoder parts and the CTC classifier are preserved, but several things are changed. First, the h1 is trimmed into h' using the CTC posterior y1 the reference, the CTC loss for h2 is the vanilla CTC loss since h2 does not need to be trimmed again. Thirdly, the h2 is then fed into downstream modules and losses.
the length of the intermediate hidden output is reduced. Second,
The design of downstream modules and losses depends on the specific tasks and system architec- tures. For ASR and MT tasks with the Hybrid CTC/Attention architecture, the attention decoder and the cross-entropy loss are adopted and the weight can be inherited from the first stage. For ASR with the Transducer architecture, the prediction network and the joint network are randomly initialized and the Transducer loss is adopted. For the ST task with the Hybrid CTC/Attention architecture, the translation encoder and the attention decoder are randomly initialized and two extra losses are adopted: the vanilla CTC loss and the cross-entropy loss. Note in ST tasks, the text labels from both the source language and the target language are provided. CTC loss for Encoder and Encoder still adopts the source language labels like in the first stage, but the translation encoder and the at- tention decoder are supervised by the target language labels (tgt.I). For all architectures, the global training objective is the weighted sum of all loss items.
Note that, in the second stage, it is possible to use vanilla CTC or BRCTC consistently. However, when vanilla CTC is consistently adopted, the down-sampling factor will degrade gradually along with the second-stage training; when BRCTC is consistently adopted, a slight performance degra- dation is observed. All hyper-parameters of this two-stage method presented in Appendix E
18

---

Published as a conference paper at ICLR 2023
E
APPENDIX: REPRODUCIBILITY
This appendix describes the details of all experiments for reproducibility.
Datasets: all statistics of the datasets are in table
Models and Features: all model architectures and the input features are presented in table table 4
table 6 and table
Training and BRCTC settings: Adam (Kingma & Ba 2014) optimizer with the learning rate being inverse square root decaying (Vaswani et al. (017) is adopted. Details for the op-
timization and the setting for BRCTC shown in table All experiments conducted on Nvidia V100 GPUs.
Decoding:
For the online ASR model, adopt greedy CTC decoding. For all other offline models, the decoding configurations are in table 8 All the inference jobs are conducted on Intel(R) Xeon(R) Platinum 8255C CPU (2.5GHz). The RTFs are calculated by the first 1/88 data in the corresponding test sets. External language models in any form are not used in inference. Checkpoints from the last 10 epochs (for ASR) or the 10 epochs with the best validation accuracy (for ST) are averaged for evaluation.
Dataset
Task
#Hours
#Pairs
Language
Units
Aishell-1 Bu ct al
2017
ASR
178
120k
Mandarin
4231 Char.
Aishell-2
ASR
1k
962k
Mandarin
5214 Char.
Wenetspeech
2022
ASR
10k
14M
Mandarin
6267 Char.
Librispeech
Panayotov at
2015
ASR
960
281k
English
500 BPE
IWSLT De-En En (Cttolo et a
2012
MT
160k
Femman-English
10k BPE (shared)
IWSLT Es-En
(Cellolo el
2012
MT
160k
Spanish-Enge -nggiss
10k BPE (shared)
MuST-C-V2 En-De (Di Gangi al.
2019
450
250k
English- German
500 4k BPE
Table 2: Dataset description of ASR / MT ST tasks.
Acoustic input features: FBank Pitch
Frage_Jength
25ms 80
Frame_ste shift
10ms 3
Fbank_ding
Pitch_dim
Text imput: Embbedding
Embedding size
512
Acoustic data augmentation: speed perturhation and specaugment Park al. 2019)
Num time masks
Time mask length
Num frequency masks
2
Frequency mask length
Max time warp
5
speed perturbation factors [0.9, 1.0, 1.1]
Fncoder: Conformer Gulati al. 2020
Num layer
12
Num atlention head Feed-forward dim
4
Attention dim
512
2048
Num CNN module kemel
31
CNN down sample
4x
Decoder for hybrid CTC/Aticntion architecture:
Transforner (Vaswani et al. 2017
Num layer
Num attention
Attention dim
512
Feed-forward dim
2048
Decoder for Transducer (Graves 2012 plus CTC architecture:
Prediction network
LSTM
LSTM hidden size
512
LSTM num layer
Joint network
Linear
Joint network dim
512
Hybrid CTCattention Watanabe et al. 2017 architecture:
CTC loss weight
0.3
Attention loss weight
0.7
Attention label smooth
0.1
Transducer (Graves [0112] CTC architecture:
CTC loss weight
0.5
Transducer loss weight
1.0
Others:
Inter. CTC [Le & Watanabe 2021
6-th 0.1
Inter. CTC weight
0.3
Dropout rate
Table 3: Offline ASR system configuration
19

---

Published as a conference paper at ICLR 2023
Acoustic input features: FBank + Pitch
Frame Jength
25ms
Frame_shin
10ms
Fbank_dim
Pitch_dim
Text imput: Embedding
Embedding size
512
Acoustic data augmentation: speed perturhation and specaugment Park 2019)
Num time masks
2
Time mask length
40
Num frequency masks
2
Frequency mask length
30
Max time warp
speed perturbation factors
[0.9, 1.0, 1.1]
Encoder: Emformer (Shi et al. 2021
layer
Num atlention head Feed-forward dim
Attention dim
512
2048
Memory bank length
4
Left context
320ms
Chunk size: Right Context
2:1
1:1
Chunk size
[80 160, 320, 480, 640)ms
Decoder for hybrid CTC/attcntion architecture:
Transformer ( (Vaswani ct al.
2017
Num layer
6
Num attention head
Attention dim
Feed-forward dim
2048
Ilybrid CTCAAtlention Watanabe el al.
2017
architecture:
CTC loss weight
0.3
Attention loss weight
0.7
Attention lahel smooth
0.1
Others:
Dropout rate
0.1
Table 4: Online ASR system configuration
Acoustic input features: FBank
Frame_Jenglh Jength
25ms 80
Frame-shift
10ms
Fbank_din
Text input: Embedding
Embedding size
512
Acoustic data augmentation: speed perturbation and specaugment Park (aar et al. 2019
Num time masks
Time mask length
5% T
Num frequency masks
Frequency mask length
Max time warp
speed perturhation factors
[0.9, 1.0. 1.1]
Encoder: Conformer (Gulati et al. 2020
Nur layer
12
Nur attention head Feed-forward dim
Attention dim
256
2048
Num CNN module kernel
CNN down sample
Translation Encoder: Conformer (Gulati et al.
2020
Num layer
Num attention head Feed-forward dim
Attention dim
512
2048
Num CNN module kernel
Decoder for hybrid CTClatiention architecture:
Transformer (Vaswani et al. 2017
Num layer
Num attention head Feed-forward dim
Attention dim
512
2048
Hybrid CTC/attention (Watanabe et al., 2017)
architecture:
ASR CTC weight
ST CTC weight
ST Attention loss weight
0.49
Attention label smooth
0.1
Others:
Inter. CTC (Lee & Watanabe 2021
6-th
Inter. CTC weight
0.3
Dropout rate
0.1
Table 5: ST system configuration
Text input: Embedding
Embedding size
512
MT encoder: LegoNN Encoder (Dalmia al. 2022
Num layer before up-sampling
Num layer after up-sampling fred-forward dim
6
Attention dim
1024
Num attention head
Up-sampling rate
Decoder hybrid CTClatention architecture:
Transfomer (Vaswani al. 2017
Num layer
Num uttention head
Attention dim
512
Peed-forward dim
1024
Hybrid CTC/atcention (Watanabe al. 2017 CTC loss weight
architecture:
0.3
Attention loss weight
0.7
Attention label smooth
0.1
Others:
Inter. (Lee & Watanabe 2021
6-th
Inter. CTC weight
Dropout rate
Table MT system configuration
20

---

Published as a conference paper at ICLR 2023
Dataset
(offline)
À (online)
#epochs
peak
#warmup iter.
#GPU
Max Global Barch size
Aishell-1
10
20
50+50
3c-4
25k
8
400 seconds
Aishell-2
10
20
50+50
3e-4
25k
8
400 scconds
Wenetspeech
30
50+50
3e-4
25k
32
3200 seconds
Librispeech
100
50+50
3e-4
25k
8
400 scconds
IWSLT
50
150+50
2o-3
10k
8M bin
MuST-C-V2
50
60+40
le-3
25k
8
6M bin
Table 7: Optimization strategy and BRCTC settings. Epochs 50+50 means 50 epochs for both the first and the second stages. If there is only one stage (baseline offline systems, online systems), the number of epochs is the sum. Bin for each example: bin input length output length input dimension.
Dataset
Beam Size
CTC weight
Attention weight
Length reward
Aishall-T
I0
0.5
0
Aishell-2
10
0.5
0.5
Wenetspeoch
10
0.5
0.5
Librispeech
10
0.5
0.5
0
TWSLT
5
{0, 0.2, 0.4, 0.6, 0.8)
1 CTC weight
{0, 0.2, 0.4, 0.6, 0.8)
MuST-C-V2
{0. 0.2, 0.4, 0.6, 0.8)
1 CTC weight
{0. 0.2, 0.4, 0.6, 0.8)
Table 8: Offline decoding configurations. Numbers in curly braces 0 indicate grid search. For grid search we report the best results.
F APPENDIX: FORMAL DEFINITION OF THE LATENCY SOURCES AND FURTHER EXPLANATION
This appendix provides the formal definitions for the three latency sources in section 3.4 All kinds of latency are computed as the expected value and are at the token-level. Note we are using the Em- former (Shi al. (021), which defines three parts of context: left conteco,,,,, chunk and right_context. The left context is the observed history and is irrelevant to the latency. The chunk is the current context. The adoption of the right context allows look-ahead mechanism.
Data Collecting Latency (DCL): DCL chunk_size/2 + right which is the mean time to wait before the collected frames can form a chunk.
Computational Latency (CL): CL chunk size RTF, which represents the rough infer- ence time for chunk.
Drift Latency (DL): = +, where - is the ending unit index of the token prediction in the path; is the starting unit index of that token obtained by DNN-HMM systems. To ensure every token has its reference, we only consider the tokens in the longest common subsequence between the reference transcription and predicted hypothesis. The DL can be negative due to the look-ahead mechanism.
We further provide an example to explain that the three latency sources are exclusive and should be accumulated. Assume event happens at the step with the input index of f. The input unit for the i step will not enter the model for inference until the following units are collected enough to form chunk, which results in the DCL. After the chunk forms, the computation on the model will take some time, which is the CL. Even though the inference process of the f step has been finished, the output posterior yf usually will not predict the event due to the index drift shown 2 Instead, the model may predict the event at another input index So the Event prediction cannot be emitted until the inference process the unit has been finished, which also a latency. The gap between and f is the DL. Note the DL only depends on the input index, not the real-world timeline.
would also like to note that this decomposition of latency sources ideally ignores the other sources like communication, feature extraction, model loading, etc., as these latency sources are usually marginal or out of the scope of seq2seq tasks.
21

---

Published as a conference paper at ICLR 2023
G
APPENDIX: DETAILED EXPERIMENTAL RESULTS ON OFFLINE DOWN-SAMPLING
Aishell-1
Aishell-2
Wenetspeech
Librispeech
178h, Mandarin
1kh, Mandarin
10kh, Mandarin
960h. English
System
dev test
android / mic
dev meeting / net
t-clean 1-other
Atlention + CTC
4.26 4.74
6.33/ 6.29
9,44 15.97
3.02/7.72
Attention BRCTC (nurs)
4.30/4.75
6.13/5.34 6.03
9.59 16.86 9.04
2.15/7.63
Transducer CTC
4,47 74.92
6.3975.47 6.18
9.20/17.34/ 8.61
3.05/7.79
Transduder BRCTC (ours)
4.33 / 4.73
6.36 5.35 6.30
9.20 17.41 8.87
3.44/7.41
Table 9: ASR results on the models' transcription performance w/o BRCTC down-sampling method. All models adopt auto-regressive decoding algorithms. CER/WER (4) are reported. The beam size is set to 10 consistently.
AishelI-T
AishelF-2
Wenetspeach
Librispeech
lest
test-tos
test-net
1-other
System
RIF
1XF77Omele
RTF
RTF
DSF7Orade
RTF
OOace
Attention CTC
1.07
Attention BRCTC (ours)
0.94
0.21/0.12
0.95
0.34/0.12
1.53
0.37/0.21
1.38
0.35/0.21
Transducer + CTC
0.37
0.42
0.48
0.31
Transducer BRCTC (ours)
0.21
0.20/0.12
0.22
0.29 0.12
0.33
0.40/0.21
0.36
Table 10: Evaluations results on the models' inference cost w/o BRCTC down-sampling method. The real-time factor (RTF 4), the down-sampling factor (DSF 4) and oracle are reported. The beam size is set to 10 consistently. The maximum inference cost reduction happens in Aishell-2 Transducer case, in which the RTF is reduced from 0.42 to 0.22 (47% relative reduction).
H APPENDIX: DETAILED EXPERIMENTAL RESULTS ON ONLINE PERFORMANCE-LATENCO TRADE-OFF
DCI+DDI+CT (ms)
Hardware-lndepenne
Hardware- Deppendee
CER% Greedy
Marker
(CL)
(ms)
DCL+ (mS)
(Ls)
Aishell test
0.176
6.88
480
120
336
456
0.305
24
7.19
0.128
6.28
850
720
80
800
0.106
50
5.77
5.55
315
240
47
287
0.176
28
8.10
315
7.97
20
440
480
800
400
0.128
40
7.23
570
720
200
$20
0.106
50
6.31
Aishell 2 test-android
403
0.175
9.97
465
80
363
443
0.283
22
10.25
9.44
632
480
103
583
0.104
49
9.03
8.57
302
80
200
280
0.283
22
11.63
20
349
320
-8
312
0.116
37
10.33
9.65
532
640
-166
474
0.09]
58
9.16
Table 11: Trade-off between the transcription performance and the latency for online CTC mod- els. A: risk factor of BRCTC: DCL.: data collecting latency; CL:: computational latency; DL: drift latency; CER: character error rate; RTF: real-time factor. Latency data computed for greedy search only. and ** represent cases for comparison. A represents the extremely low latency cases that cannot be achieved by vanilla CTC. The minimum overall latency achieved by BRCTC only 302ms (Aishell-2). Compared with its vanilla baseline whose minimum overall latency is 43 lms, the BRCTC achieves a 30% overall latency reduction relatively.
22

---

Published as a conference paper at ICLR 2023
1 APPENDIX: DETAILED EXPERIMENTAL RESULTS ON MT TASKS
This appendix presents the experimental results on MT tasks. As suggested in table T2 BRCTC can reduce the length h to 63% and save 27% inference cost. At the same time, the model with BRCTC still preserves the competitive transcription ability (very close BLEU scores).
However, several things noticeable. For MT tasks, the lengths of and Commonly, there is no need to conduct the down-sampling on h in MT tasks. This experiment follows the setting in (Yan et al. 2022), where CTC is integrated as an auxiliary criterion. To ensure [h| 2 [I] in CTC computation, the x are up-sampled for 3 times when being encoded into h. In this case, the down- sampling process will be needed.
usually close.
Transcription Quality (BLEUT)
Dowa sampling Effectiveness
System Attention + Vanilla CTC (Ya et 2022
De.FEn
Fs.P5n
DESTD Oracle Rel. Interence Time (I)
31.9
37.9
1.00
Attention + BRCTC (ours)
31.7
38.0
0.63 6,34
0.73
Table 12: MT performance on IWSLT14 dataset w/o BRCTC down-sampling
J APPENDIX: MORE VISUALIZATION
This appendix provides more visualization results on 1) the gradient analysis of BRCTC in online applications and 2) the attention analysis of BRCTC down-sampling process.
7 compares the evolution the CTC distributions and their gradients on the online performance- latency trade-off application. Most of the observations are similar to those in F 1) at the beginning of training. the gradients for each non-blank token, rather than the last non-blank token only, are interfered with by the adoption of Bayes risk function. The gradient peaks of BRCTC are shifted to the left. 2) after 2 epochs of training, the gradients will also be localized, but BRCTC has not learned the places where the emission will happen. After convergence, the emissions of BRCTC will be earlier. We add the input indexed to show its difference from vanilla CTC.
Vanilla CTC Posterior
BRCTC Posterior
Vanilla CTC Gradient
BRCTC Gradient
0.03
Flat Start
0.00
0.03
0.03
Epochs
0.00
10 14
10
0.03
0.03
20 Epochs
0
0.00
0.00
6
10
16
2
6
13
Figure 7: Evolution of CTC distribution y and the corresponding gradients on logy. BRCTC with emission latency alleviation method is used.
Fig demonstrates the encoder self-attention weights from different layers and attention heads. As shown in the figure, the down-sampling process is completed mainly in the last two layers before the BRCTC criterion (the 5-th and 6-th layers). The attention weights shown in the red boxes suggest
how the semantics of input units with large input indexes are aggregated to the output units with small indexes. Observations like this are mainly in the last two layers before BRCTC so we assume the global context is fully explored in other layers.
23

---

Published as conference paper at ICLR 2023
Outpue Indox
50
20
Olutzut lindx
20
20
120
220
:00
220
Outtuu
Clttuut
150
:30
350
210
220
220
200
320
250
250
250
210
Head 1 Input Index
Head Input Index
Head 3 Input Index
Head Input Index
Layer-1
Output Index
Output Index
Output Index
5o
Outpat Index
200
350
350
50
L0C
220
J00
30
003
130
J00
200
50
100
-35
200
200
50
100
330 J0u
Imput
Imput
Input
Imput
Layer-2
Outpue Indox
50
Outpue Indox
20
820
220
200
200
250
250
250
132 20
290
103
150
200 352
50
100
253
108
350 200
Head 1 Input Index
Head 2 Input Index
Head 3 Input Index
Head 4 Input Index
Layer-3
Ouiput Indox
50
10
Outquet Index
2o
20
100
100
azo
100
:zo
350
220
200
200
200
200
250
250
250
250
Head 1 Input Index
Head Input Index
Head 3 Input Index
Head Input Index
Layer-4
Output index
Output Index
Output Index
Outpat Index
a00
130
50
10C
220
200
250
30
002
120
200
200
50
102
-30
200
250
30
100
230 200
Imput
Imput
Input
Imput
Layer-5
Oucput Indox
Ourput Indox
820
120
320
200
300
250
250
250
250
102 10
103
150
200
50
100
273
200 Head 4 Input Index
350
Head 1 Input Index
Head 2 Input Index
Head 3 Input Index
Layer-6
Figure 8: Encoder attention plots from different layers and attention heads. Data from Librispeech test-clean set, utterance 1089-1 34686-0000. Down-sampling is conducted in the last two layers before BRCTC only (see the red boxes).
24

---

Published as a conference paper at ICLR 2023
K APPENDIX: ALIGNMENT DRIFT OF VANILLA CTC SYSTEM
This appendix demonstrates that systems even trained with vanilla CTC are predicting the align- ment that drifts significantly. We draw several CTC posteriors and their reference alignment ob- lained by DNN-HMM systems (the colored bars) in Fig All figures are obtained from Aishell-2 test-android set.
CTC Posteriors
Input index
Utterance: Aishell-2 ATOOLIW0009
CTC Posteriors
Input Index
Utterance: Aishell-2 ATO01 WW0200
CTC Posteriors
Input Index
Utterance: Aishell-2 AT001 AT001IW002. WW020
CTC Posteriors
Input Index
Utterance: Aishell-2 AT001 W0020
Figure 9: CTC posteriors and their reference alignment predicted by vanilla CTC systems. Colored bars are the reference alignments obtained by DNN-HMM systems. The predicted alignment drift significantly compared with the DNN-HMM reference alignment.
25


Reviews:
[{"review_id": "nPYKTJExN_", "reviewer": "Reviewer_TGbL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new variant of CTC that can encode conditions to achieve different preferable properties. The authors gave two examples, one is to force the model to emit the last non-blank token as early as possible; the other is to avoid the drift latency of non-blank tokens to provide better trade-off between latency and transcription quality. Both the two examples have practical merit for ASR systems. The authors did intensive experiments on both streaming and non-streaming ASR, ST and MT to show the control ability of the proposed BRCTC. The results are convincing given the experimental settings. ", "summary_of_the_review": "Overall, the reviewer think this is a very good extension to vanilla CTC and can  benefit multiple seq2seq tasks. ", "strength_and_weaknesses": "Strength:\n1. Novelty; There are previous studies on different CTC variants but the perspective explored in this paper has not been seen in literature. The idea to \n2. Of clear practical benefits; Though most of commercial ASR systems are based on Transducer or LAS, CTC is still widely used as a standalone ASR system or integrated into Transducer/LAS training to provide auxiliary functions. And other fields like MT/ST are also relying on CTC in some scenarios. The reviewer believe the work in this study will result in more flexible CTC based designs.\n3. Convincing results; The authors did intensive experiments and show both numeric results and visualization to validate the effectiveness of the proposed BRCTC.\n\nWeakness\n1. For online experiments, I think CTC-attention training is unnecessary. It's better to show pure CTC training results. \n2. Some are some variants of transducer like (alignment-restricted transducer, prunned Transducer) are also based on modification of forward-backward algorithm to achieve specific properties (latency, computational cost). It's better to cite these papers.\n3. For the limitation discussion, the author mentioned the increasing training cost of BRCTC but didn't provide much details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QTXwN3tWf4", "reviewer": "Reviewer_YKWK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use Bayes Risk factors to control specific characteristics of CTC alignments. Specifically, the proposed method encourages the model to move the non-blank predictions towards the beginning of the output sequence. This shift of non-blank predictions enables speed-ups both in offline and online inference. In offline inference, since the non-blank predictions occur at the beginning of the output sequence, the paper proposes to trim all blank predictions after the last non-blank prediction. As a result, the length of decoder input is shorter by 60%, making the decoding process 47% relative faster. In online inference, the speed-up is also achieved by encouraging the model to produce non-blank predictions sooner, thus reducing the drift latency caused by delayed non-blank predictions.", "summary_of_the_review": "This paper proposes an interesting method which could be very useful for improving the inference speed of ASR models. Therefore, after including the naive down-sampling baseline experiment and fixing the writing/formatting issues mentioned above, I am inclined to accept this paper.", "strength_and_weaknesses": "Strengths:\n\nThe paper proposes a novel technique for controlling specific characteristics of CTC alignments using Bayes Risk. Furthermore, the paper shows how to derive the Bayes Risk CTC loss function from the CTC loss function. The conducted experiments show that the proposed method can be used to speed up both offline and online inference.\n\nWeaknesses:\n\nThe method is compared only against vanilla CTC. However, since the proposed method effectively reduces the length of the encoder output sequence for offline recognition, it would be good to compare it with naive uniform subsampling of encoder outputs with subsampling factors 2 and 3 to achieve the same down-sampling factor. Furthermore, in the case of online recognition, the paper should mention CTC delay constraints from [1], which uses reference alignments obtained with an external DNN-HMM model. Comparing the BRCTC method with the delay constraints method would be nice. This experiment can be implemented with GTN, which the authors used to implement BRCTC, and delay constraint FSA similar to Kaldi TimeEnforcerFst.\n\n[1] Senior, Andrew, et al. \"Acoustic modelling with CD-CTC-sMBR LSTM RNNs.\" 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HWEZRKdjv2z", "reviewer": "Reviewer_NBYg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CTC is a fundamental method for many sequence-to-sequence tasks. It works by maximizing the summed probabilities of paths that can be mapped to the target sequence. As CTC treats all paths equally, it cannot generate accurate alignments between its predictions and input sequences. This work proposes a Bayes Risk CTC method (BRCTC) to make CTC alignment controllable by first dividing paths into groups and then specifying preference weights for groups. For offline ASR, BRCTC was shown to effectively reduce the inference cost by down-sampling the hidden vectors. For streaming ASR, BRCTC can accelerate the recognition by forcing the model to emit non-blank spikes as early as possible.\n\n", "summary_of_the_review": "Making CTC output controllable alignment prediction is an interesting topic. The proposed method is novel and technically sound. My main concern is whether the proposed method is applicable to other useful grouping functions except the example given in Sec. 3.2 and if there is a general rule for making a legal grouping function. However, the paper does not include any discussion on it.", "strength_and_weaknesses": "Strength\n- The method is novel and well-motivated.\n- As far as I can see, the method is technically sound and well-justified by the experiments.\n\nWeakness\n- The work does not specify a general condition that the grouping function f(\\phi) has to meet to enable the forward-backward algorithm to be applied. Instead, it gives an example f(\\phi) which is the largest index of non-blank elements in the path. If it is the only case to which the proposed method can be applied, the main value of the work is to speed up CTC inference. Although accelerating CTC inference is also important, the writing of the paper gives readers the impression that it is a general method to control CTC alignment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3E0sTnlpZT", "reviewer": "Reviewer_GviL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper re-formulates the widely used CTC criterion for ASR such that the alignments learnt are controllable. This is done by forcing a preference of desirable alignment paths in the CTC loss function. Furthermore, a bayesian risk factor controls this preference. To this end, the forward and backward variables are chosen such that the above preference is satisfied. The authors report the effectiveness of the proposed technique on two tasks: 1) downsampling of speech frames for offline ASR such that inference time is reduced without performance degradation; 2) speeding up online ASR by having earlier emissions without loss in performance. Results on speech translation have also been reported.", "summary_of_the_review": "This is a good paper which provides a mathematically sound technique to control CTC alignments. This is backed by impressive results on two tasks. The paper may benefit by running some simple baselines.", "strength_and_weaknesses": "The paper is well written and the proposed techniques are very well formulated mathematically. The authors provide abundant materials in the appendix and also release their code. CTC alignment is an important problem for ASR and may have consequences on a variety of downstream applications. The authors investigate two of them and the results are impressive. The proposed technique itself is quite elegant and mathematically grounded. I appreciate the effort put in by the authors to build the mathematical framework and providing all the details of the experiments in the appendix.\n\nComments:\n1. There shouldn’t be a ‘t’ in the subscript for ‘p’ in equation (2)\n2. I may be missing something here, but shouldn’t the final result in equation (5) be a product over all ‘t’s.\n3. The arg max operation defined in the first paragraph of page 5 is not very clear. There is a boolean operation inside the argmax(.) where elements will be either 0 or 1, what will argmax return in such a case?\n4. Typo: page 6 last line, DFS -> DSF\n\nQuestions:\n1. Why did the authors not compare against some simple baselines:\n\n    a) Downsampling: Using a CNN for subsampling a speech sequence such that the new sequence length is similar to the one obtained by the controlled alignment. The CNN is like a feature extractor from speech. Fox example, wav2vec2.0 uses 20ms windows, what happens if we use a >20ms window and lower the sequence length? How much performance is degraded and how does it compare with the proposed model?\n\n    b) Online Latency reduction: Plantinga et al. [1] have a simple solution for emitting phones sooner where they have an “alignment loss which encourages outputs only when features do not resemble silence.” It would also be a good idea to cite the above paper.\n\n2. Can such an alignment controlling mechanism be used to train RNN transducers which are also very popular and use a quite similar algorithm to compute the loss?\n\n3. What are some other applications apart from the two explored where controlling CTC alignment will be useful?\n\nReference:\n[1] Plantinga et al. Towards Real-Time Mispronunciation Detection in Kids' Speech. ASRU 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster