Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework
Authors: Eliya Segev
Maya Alroy
Ronen Katsir
Noam Wies
Ayana Shenhav
Yael Sapir Ben-Oren
David Zar
Oren Tadmor
Jacob Bitterman
Amnon Shashua
Tal Rosenwein
Venue: ICLR 2024 poster (2024)
Keywords: ASR
CTC
seq2seq
low latency
WER
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A framework to add a controllability to the predicted alignment in CTC training according to a desired property

Abstract: Connectionist Temporal Classification (CTC) is a widely used criterion for training supervised sequence-to-sequence (seq2seq) models. It learns the alignments between the input and output sequences by marginalizing over the perfect alignments (that yield the ground truth), at the expense of the imperfect ones. This dichotomy, and in particular the equal treatment of all perfect alignments, results in a lack of controllability over the predicted alignments. This controllability is essential for capturing properties that hold significance in real-world applications. Here we propose Align With Purpose (AWP), a general Plug-and-Play framework for enhancing a desired property in models trained with the CTC criterion. We do that by complementing the CTC loss with an additional loss term that prioritizes alignments according to a desired property. AWP does not require any intervention in the CTC loss function, and allows to differentiate between both perfect and imperfect alignments for a variety of properties. We apply our framework in the domain of Automatic Speech Recognition (ASR) and show its generality in terms of property selection, architectural choice, and scale of the training dataset (up to 280,000 hours). To demonstrate the effectiveness of our framework, we apply it to two unrelated properties: token emission time for latency optimization and word error rate (WER). For the former, we report an improvement of up to 590ms in latency optimization with a minor reduction in WER, and for the latter, we report a relative improvement of 4.5% in WER over the baseline models. To the best of our knowledge, these applications have never been demonstrated to work on this scale of data. Notably, our method can be easily implemented using only a few
lines of code and can be extended to other alignment-free loss functions and to domains other than ASR.

Published as a conference paper at ICLR 2024
ALIGN WITH PURPOSE: OPTIMIZE DESIRED PROPER- TIES IN CTC MODELS WITH A GENERAL PLUG-AND- PLAY FRAMEWORK
Eliya Segev*
Maya Alroy"
Ronen Katsir
Noam Wies
Ayana Shenhay
Yael Ben-Oren
David Zar
Oren Tadmor
Jacob Bitterman
Amnon Shashua
Tal Rosenwein
ABSTRACT
Connectionist Temporal Classification (CTC) is widely used criterion for train- ing supervised sequence-to-sequees (seq2seq) models. It learns the alignments between the imput and output sequences by marginalizing over the perfect align- ments (that yield the ground truth), at the expense of the imperfect ones. This di- chotomy, and in particular the equal treatment of all perfect alignments, results in lack controllability over the predicted alignments. This controllability is essen- tial for capturing properties that hold significance in real-world applications. Here we propose Align With Purpose (AWP), a general Plug-and-Plaz framework for enhancing a desired property that by complementing the CTC loss with an additional loss term that prioritizes alignments according to a desired property. AWP does not require any interven- tion in the CTC loss function, and allows to differentiate between both perfect and imperfect alignments for a variety of properties. We apply our framework the domain of Automatic Speech Recognition (ASR) and show its generality in terms property selection, architectural choice, and scale the training dataset (up to 280,000 hours). To demonstrate the effectiveness of our framework, apply it two unrelated properties: token emission time for latency optimization and word error rate (WER). For the former, latency optimization with minor reduction in WER, and for the latter, we report relative improvement of 4.5% in WER over the baseline models. To the best of our knowledge, these applications have never been demonstrated to work on this scale of data, Notably, our method can be easily implemented using only few lines of code and can be extended to other alignment-fre loss functions and to domains other than ASR.
models trained with the CTC criterion. We do
report improvement of up 590ms in
INTRODUCTION
oulput ((lls)
Property-Designed function
Vocabutary Saze
fars
4
4g
Creates an improved
43
cample N aligmments from P(ak))
alignment based
(w.r1 the property)
ax
C. = fex
(ave(r.)) mar(Pr(a))a)) Pr(ajls) A,
Figure I: The Align With Purpose flow: alignments are sampled from the output of a pre-trained CTC model which f fprop is applied to create N pairs of alignments. Then, hinge loss with an adjustable weight applied on the probabilities of each pair alignments, trained jointly with a CTC loss. See full details in section 2.2
"Equally contributed, email: last ) @ @Forcam.com
'The code will be made publicly available in the supplementary materials.

---

Published as a conference paper at ICLR 2024
srquencc-to-sqqeere (seq2seq) tasks, in which the learner needs to predict sequence of la- bels from unsegmented imput data, are prevalent in various domains, e.g. handwriting recognition (Graves & Schmidhuber 2008), automatic speech recognition (Collobert et al., 2016;) Hannun al.) 2014), audio-visual speech recognition (Afouras et al.) 2018), neural machine translation (Huang et 2022), protein secondary structure prediction (Yang et. (022), to name a few. For years, optimizing a seq2seq task required finding suitable segmentation, which is an explicit alignment between the input and output sequences. This is severe limitation, as providing such segmentation is difficult (Graves) 2012)
Two main approaches were introduced to overcome the absence of an explicit segmentation of the input sequence, namely soft and hard alignment. Soft aligmment methods use attention mechanism
(Chan et al., 2016,) Vaswani et al. 2017) that softly predict the alignment using attention weights.
Hard alignment methods learn in practice an explicit alignment (Graves et al. 2006; Graves 2012
Collobert et al. 2016), by marginalizing over all alignments that correspond to the ground truth (GT)
labels.
As streaming audio and video become prevalent (Cisco 2018). architectures that can work in a streaming fashion gain attention. Although soft alignment techniques can be applied in chunks for streaming applications (Bain et al. 2023), their implementation is not intuitive and is less computa- tionally efficient compared to hard alignment methods, which are naturally designed for streaming processing. Among the hard alignment methods, the CTC criterion (Graves et al. (006) is a com- mon choice due to its simplicity and interpretability. During training. CTC minimizes the negative log- logliielihood of the GT sequence. To overcome the segmentation problem, CTC marginalizes over all possible input-GTT output pairings, termed perfect alignments. This is done using an efficient algorithm, which is the core algorithm in CTC.
herward-backwee
CTC has by-product of learning to predict an alignment without direct supervision, as CTC poste- riors tend to be peaky (Zeyer et al.) [021 Tian et al. (022), and hence the posterior of few specific alignments dominant over the others. While this implicit learning is useful, it comes at the cost of the inability to control other desired properties of the learned alignment. This can be explained by the inherent dichotomy of the CTC, which leads to lack of additional prioritization within perfect or imperfect aligmments.
However, many real-world seq2seq applications come with a property that can benefit from or even require such prioritization. For example, in the contexts of ASR and OCR, standard metric to test the quality of a system is the word error rate (WER). Therefore, prioritizing imperfect alignments with low WER can improve the performance of a system measured by this metric, thereby reducing the gap between the training and testing criteria (Graves& & Jaitly| 2014). Another example is low-latency ASR system. Here, even perfect CTC score can only guarantee a perfect transcription while completely disregard the latency of the system. Clearly, in this setting, for an application that requires fast response, prioritizing alignments with fast emission time is crucial. F visualizes the aforementioned properties. In general, there are many other properties that also necessitate prioritization between alignments, whether perfect or imperfect.
GT: the cat
0
Pred:
tha cet
CER
28.5%
28.5%
WER
100%
50%
L4
Pred 1: CATTTØ0
Pred 2: @CC@ATT
Figure 2: A visualization of two properties that are not captured by CTC. (a) Emission Time: Two alignments that yield the same text. but the green alignment emits the last token of "CAT" at times- tamp (t_3) while the purple alignment emits it at t.6. (b) Word-Erre Rate: two imperfect predic- tions with the same CER but different WER.
2

---

Published as a conference paper at ICLR 2024
To exemplify the importance of prioritization, Table shows that CTC score is not a good proxy for some properties of the predicted alignment. shows two different models with a similar training loss that have different WER and emission time, although trained on the same data.
Table 1: CTC score is not good proxy for WER and latency of a system. The reference and the streaming models are CTC models with different architectures, which results in different predicted alignments. The latency of the streaming model is the delay in token emission time in comparison to the reference model. The Results are shown on the LibriSpeech test clean dataset.
ASR Model Type
CTC Score
WER ( (%)
Latency (ms)
Reference Streaming
0.0033
4.28
0
0.003
5.43
217
To complement the CTC with an additional prioritization, we propose Align With Purpose (AWP) a Plug-and-Play framework that allows enhancing a given property in the outputs of models trained with CTC while maintaining their transcription abilities. We add loss term, LAwP, that expresses a more subtle differentiation between alignments so that the final loss becomes I. = Lorc + J. Specifically, for a given property, design function fprop that receives an alignment as an input, and outputs an improved alignment with respect to the property. Then, we sample N alignments based on the output probabilities of the pre-trained CTC model, apply Sprop on the sampled alignments, to create N pairs of alignments. Finally, we implement LAw p as hinge loss over the N pairs, thus encouraging the model to increase the probability mass of the preferable alignments, as described in Figure
Previous research has proposed controllability to model predictions, Liu et al. (002)) introduced an additional loss term that prioritizes a ranked list of alternative candidates during the training of gen- erative summarization models. Specifically, for the case of hard alignment criteria like CTC, many proposed solutions are restricted to handling perfect alignments only, and some require interven- tion in the forward-bace backward algorithm (Tian al. 2022 Yu et al. [02T]] [hinohara & Watanabe, 2022 Yao et al. 2023 Laptev et al. 2023). as opposed to AWP. Alternative approaches address the imperfect alignments through additional loss terms, seen in Prabhavalkar et al. (2018); Graves & Jaitly (2014). The aforementioned frameworks are less straightforward for implementation and might require considerable amount of development time and optimization. In contrast, AWP offers a relatively simple implementation, requiring only few lines of code.
To summarize, our main contributions are as follows: (1) Align With Purpose a simple and general Plug-and-Play framework enhance general property the outputs of CTC model. (2) We show promising results in two properties that are independent of each other- we report an improvement of up to 590ms in latency optimization, and a relative improvement of 4.5% WER over the baseline models for the minimum WER (mWER) optimization. (3) We demonstrate the generality of our framework terms of property selection, scale of the training dataset and architectural choice. To the best of our knowledge, these applications have never been demonstrated to work on scale of data as large as ours. (4) The framework enables prioritization between both perfect and imperfect alignments.
We apply our approach to the ASR domain, specifically to models that are trained with CTC crite- rion. However, this method can be extended to other alignment-free objectives, as well as to other domains besides ASR.
2 CTC AND ALIGN WITH PURPOSE
The outline of this section is as follows: We start with description of the CTC loss in subsection 2.1 followed by a detailed explanation of the proposed " "Align With Purpose" method in subsection 2.2
Finally, we showcase two applications: low latency in subsection 2.3 and mWER in subsection 2.4
3

---

Published as a conference paper at ICLR 2024
2.1 CTC
The Connectionist Temporal Classification criterion (Graves al. 2006) is common choice for training seq2seq models. To relax the requirement of segmentation, an extra blank token that represents a null emission added the vocabulary V, so that V V UUUR {0}.
[a1, [[]]] (e.g. audio), the model outputs T vectors et RIV", each of which normalized using the softmax function, where Da can be interpreted as the probability of emitting the token & at time t. An alignment is 7 length sequence of tokens taken from V', and P(a is defined by the product of its elements:
Given a T length input sequence
T P(a|x) II p(a,)
The probability of given target sequence (e.g. text) of length U, = [y1....,, where U s T, the sum over the alignments that yield y::
P(ylæ)
Σ
p(alx),
a: (W)
where B is the collapse operator that first removes repetition of tokens and then removes blank tokens.
The CTC objective function minimizes the negative log-likelihood of the alignments that yield as seen in Eq.
LCTc(#) log P(y|æ))
By definition, the CTC criterion only enumerates over perfect alignments and weighs them equally. This means that CTC considers all perfect alignments as equally good TTia et al. (2022) and all imperfect alignments as equally bad Graves & |aitly (2014)
2.2 ALIGN WITH PURPOSE
In this section, we present the suggested method, Align With Purpose (AWP). AWP complements the CTC loss with additional loss term which aims enhancing desired property by adding more subtle prioritization between alignments.
Given desired property to enhance, define property-specife function fprop V' VT that takes as input an alignment a and returns an alignment with the same length. fprop is designed to output better alignment
the property. During training, at each step we sample N random alignments according to the distribution induced by the output of the seq2seq model, such that at v, for E [1..7] and i [1..N] (see Appendix ( for more details on the sampling method). We then apply a' Frrpp (a') to obtain better alignments. This results in N pairs of alignments (a a²), where a' is superior to a' in terms of the property. Finally, enhance the desired property the model is encouraged to increase the probability mass of a', by applying hinge loss on the probabilities of the alignment pairs:
N
Law =
1 N
Σ mar
P(a1lz) + A,0),
(4)
where À is a margin determined on a validation set. See Fig. for an illustration of the proposed framework.
Prabhavalkar et al., 2018), sampling from a randomly initialized model is less effective since the outputs are completely random. Therefore, we train the model to some extent with a CTC loss as in Eq. and proceed training with the proposed method.
As pointed out (Graves & Jaitly 2014
Putting all together, the training loss then becomes:
L(x) = LCrc(x) + aLAwp(z),
(5)
where a is a tunable the CTC loss.
that controls the trade-off between the desired property and
4

---

Published as a conference paper at ICLR 2024
2.3 APPLICATIONS: Low LATENCY
Streaming ASR systems with low latency an active research field, it serves key component in many real world applications such as personal assistants, smart homes, real-time transcription of meetings, etc. (Song et al. 2023). To measure the overall latency of system, three elements should be considered: data collection latency (DCL) which is the future context of the model, computational latency (CL) and drift lateney (DL), as defined by Tian et al. (2022). For the latter, we slightly modified their definition, see Appendix A.2 for more details. We also leave the CL. component out of the scope of this work as is sensitive architectural choice, hardware, and implementation. Thus, denote by TL=DCL+DL the total latency of the system.
Several techniques were suggested to reduce the TL: input manipulation (Song et al. 2023). loss modification (Tian et al., 2022), loss regularization (Yu et al. 202I] Yao et al., 2023 Shinohara & Watanabe 2022 Tian al.j (023), and architectural choice (Pratap al.) 2020). These methods
are specific to low latency settings, or require intervention in the
algorithm. See
Appendix D for more details and comparison other works.
Probabiiity of non fon-blank tokens
TH
.5
.DEM
CN
TRA TROM
TIOM
704
.6
.0MM
DN
TRA
1.0
0.8
0.6
0.4
0.2
0.0
10 20
30
40
50
60
70 80
Time (Frames)
Figure 3: Drift in emission time in CTC model. Bottom purple text: An offline Stacked ResNet model with symmetric padding, with 6.4 seconds of context divided equally between past and future contexts. Top green text: An online Stacked Resnet with asymmetrie padding, with 430ms future context and 5.97 seconds past context. It can be seen that the output of the online model has a drift 2000 ms.
One way to reduce the DCL is by limiting the future context of the model. In attention based models can be achieved by left context attention layers Yu et al. 202I), and in convolutional NN it can be achieved using asymmetrical padding (Pratap et al.) 2020). However, Pratap al. (2020) have shown that training with limited future context results a drift (delay) the emission time of tokens explained by WWang et al. (2020), who made the observation that less future context deteriorates performance. Therefore, by delaying the emission time, the model effectively gains more context, which in improves its performance.
(DL), as be seen in Fig. 3 The cause of the drift
Sampled Alignment
Xtthhheeeacoe
cceaett
1n lntonn (a)
a
tthhheee
cc teod)
Figure 4: Defining flow Joweneyy obtain a we shift the sampled alignment a one token the left, starting from random position (second token in this example) within the alignment, and pad â with a trailing blank token, marked by a black rectangle
To mitigate the DL using AWP, given an alignment a, we sample random position within it, and shift one token to the left from that position to obtain seen in Fig. 4 Clearly, tokens emission time of a is one time step faster than a starting from the random position. By limiting the initial shift position to correspond tokens that repetitions, the same as a. To make à T length aligmment, we pad it with a trailing blank token.
ensure that the collapsed text of remains
Formally, we define the function flow Jateney- Given an alignment a, define the subset of indices [j1, fr C [2..T] all the indices such that ajj, = a 1, meaning that aj, is repetition of the
5

---

Published as a conference paper at ICLR 2024
previous token. Then sample random position ý from [fi: fr and obtain a:
a2
if < -
a2
0++1
if - - T ift==
(6)
2.4 APPLICATIONS: MINIMUM WORD ERROR RATE
The most common metric to assess an ASR system is the word error rate (WER). Nevertheless, training objectives such as CTC do not fully align with this metric, resulting in gap between the training and testing criteria. Therefore, the system's performance could improve by adding prioritization over the imperfect alignments w.r.t. their WER. This gap was previously addressed by Graves Jaitly (2014), who suggested approaching by minimizing the expected WER but this method requires extensive optimization. Prabhavalkar al. 2018 suggested a similar objective for training attention models with the cross-entropy (CE) loss.
GT text: the cat Sampled Aignment
Collapsed
WER
a
thoooaaaee ccceeetttt
tha cet
100%
FIBER
(megin)
a
ttthooegaaee cccaaatttte tha cat
50%
Figure 5: Defining fMWER. Given target transcription 'the cat', the (upper) sampled alignment yields the text 'tha cet', which has 100% WER. Substituting the occurrences of the token 'e' with the token 'a' produces the text 'tha cat', which has 50% WER.
As illustrated in Figure 5 to apply AWP for m mWER training, define fMWER. Given a sampled imperfect alignment a and a GT transcription y. to obtain a we seleet the word in the collapsed text B(a) which requires the minimum number of substitutions in order for it to be correct. Then we fix the alignment of this word according to the GT, so that the number word errors (aaa) is reduced by 1.
3 EXPERIMENTAL SETUP
We evaluate AWP on two end-tasks: low latency and mWER, by conducting experiments using multiple architectures and different scales of datasets. The general settings listed below. For more details on the setup see Appendix[A]
Datasets. We examine our framework on 3 scales of the data, ranging from 1K to 280K hours. The small scale dataset is LibriSpeech (Panayotov al. (2015) (LS-960). The medium scale dataset consists of 35K hours curated from Vox (LV-35K). The large scale an internal dataset of 280K hours of audio-tranne -udio-transcript pairs (Internal-2800 which, the best of our knowledge, is the largest dataset that was used to train a low-latency model. We test our framework on the test splits of LibriSpeech.
Architecture. To validate that our method is invariant architecture selection, we trained 3 dif- ferent architectures: Stacked ResNet (He et al., 2016), Wav2Vec2 (Baevski et al. 2020) and Con- former (Gulati et al. 2020 model.
We used pre-trained Wav2Vec2 base model with 90M parameters, available on HuggingFace This architecture was used in the mWER training experiments.
The Conformer employed is medium-sized model with 30.7M parameters. As offline model, given its attention layers, its future context was not limited. To transition from an offline to an online model. during inference, the right context (DCL) restricted to 430ms. The left context was also limited to 5.57s, resulting in a 6s of context in total. The model consumes the input in chunks, similarly to Tian et al. 2022
https://huggingfaass co/
http://wwggggggooooooooo/oooaa/a wtt//a/cbbcc//a//eeee/ee--eeaaae 100k-

---

Published as a conference paper at ICLR 2024
Lastly, the Stacked ResNet consists of 66M parameters. This architecture can be implemented in a streaming manner and can be highly optimized for edge devices. Therefore, it's a good fit for an online system in a low-resource environment. In our implementation, the model has 6.4s of context. In the offine version of this model, the 6.4s are divided equally between past and future context, i.e. it has DCL of 3.2s. The online version, implemented with asymmetric padding by Pratap et al. (2020), has also 6.4s context, but its DCL is only 430ms, which makes it feasible to deploy it in online ASR systems. We used the offline implementation in both end tasks- baseline in the mWER training and as an offline model in the low latency training.
suggested
Decoding. Models were decoded using an in-house implementation of beam search decoder de- scribed in (Graves & Jaitly (014).
Training. To test the effectiveness of AWP, we train the models for several epochs, and then apply our framework, namely adding the AWP loss to the CTC loss as stated in Eq.
[] The epoch in which
we start to apply our framework denoted as "start epoch' in tables 2
3
The Wav2Vec2 model was finetuned using SpecAugment (Park et al. 2019) on LS-960 using the
Adam optimizer (Kingma & Ba 2014) with a flat learning rate (LR) scheduler [aavsski et al. (2020). The Conformer was trained on LS-960 with the same training scheme described in Gulati et al.
2020). The Stacked ResNet models were optimized with RAdam optimizer (Liu al. 2019) with a ReduceL_ROnnlltean schedulei on different data scales.
Evaluation Metrics. To test AWP, in the mWER setting we evaluated models with a standard
implementation of WER. In the low latency setting we evaluated DL and TL as defined in section
2.3
4
RESULTS
In this section, we present the results achieved by training using AWP in the low latency and mWER applications.
4.1 Low LATENCY
shows the results when training the Stacked ResNet model on small, medium and large scales of data. We can see a clear trend across all scales that the AWP training successfully decreases the DL. The DL for each model is computed in comparison to its relevant offline model. It can also be seen that some models achieve negative DL, meaning that the TL is reduced beyond its expected lower bound induced by the DCL. In most cases, achieving such low TL solely by reducing the architectural future context using another padding optimization would not have been possible. This trend also holds for various implementations of the property function,
be seen in Appendix
Table 2 also shows our implementation (or adjustment of public code) of selected prior work in the field of low latency in CTC training: BayesRisk CTC (Tian et al. 2002),, Peak First CTC (Tian al.) 2023) and TrimTail (Song et al.) 2023). It can be seen that AWP outperforms the other methods, both in terms of WER and latency. See Appendix D F for more details.
We can also see that as the scale of the data increases, the WER decreases. This statement holds independently for the offline models and for the online models, and remains valid also after adding the AWP loss. This shows that AWP does not affect the ability of the model to improve its basic transcription capabilities using larger scales of data, which aligns with previous observations on large scale training Baevski et al. 2020 Radford et al. 2023
In almost all the experiments, the WER increases with the latency reduction. This is a known trade- off between latency and accuracy as reported in prior work (Pratap et al.] 2020). The choice of the operating point in terms of the balance between latency and accuracy can be determined by the weight of the AWP loss, a, and the scheduling of when we add the AWP loss ('start epoch'), ), as can be seen in Fig. 6
The Conformer with AWP experiment demonstrates a DL reduction and a trade-off between latency and accuracy, thus affirming that AWP not limited to specific architecture. Given the unrestricted
1r__sheerr___heeee ler.
Reduccel.KomP wtteeleeennntaeeeeeeem html

---

Published as a conference paper at ICLR 2024
future context of the offline model, DCL matches the imput size, making TL measurement irrelevant. The online model, not trained in an online fashion (sce Appendix (A.2) is expected to lack a DL. Yet, AWP can reduce the DL to negative values, which in turn reduces the TL.
Table Low Latency model training with and without AWP on different data scales, and with other frameworks. 'Start Epoch' denotes the step that added AWP, and was chosen based on list of milestones WER of the online model. The different entries the table are reported based on the
best checkpoint terms of WER, for each model separately. Results are on Libri Test-Clean.
Model
Training Data
Start Epoch
DL (ms)
TL (ms)
WER
Stacked ResNet Offline Stacked ResNet Online
Internal-280K
0
3.2K
2.34
Internal-2800
249
679
2.6
+AWP
Internal-280K
5.7
50
480
2.71
Stacked ResNet Offline Stacked ResNet Online
LV-35K
0
3.2K
2.42
LV-35K
341
771
2.72
+AWP
LV-35K
0.1
-251
179
3.28
Stacked ResNet Offline
LS-960
0
3.2K
3.72
Stacked ResNet Online
LS-960
278
708
4.06
+AWP
LS-960
0.9
-79
351
4.38
+Peak First CTC
LS-960
186
616
4.41
+ +rimmTail
LS-960
-76
354
4.46
+Bayes Risk
LS-960
63
493
4.78
Conformer Offline
LS-960
0
3.7
Conformer Online
LS-960
2
432
3.75
+AWP
LS-960
12
- 172
263
3.74
Start epoch VS WER and drifting
Drfhing
AWP loss weight VS WER and dritting
7.0
200
200
100
100
6.0
-0
WER
8.5
1100
2.3
-000
220
4.
300 4.0
330
54
7.2
Start opoch
AWP loss weightt (o)
Figure 6: The effect of the 'start epoch' and a the word error rate (WER) and the drifting (DL). While the 'start epoch' has more effect on the DL, the WER only slightly changes (Left). To test the effect of the a, we fixed the 'start epoch' 2.7 and applied AWP with different weights. The selection of a has a significant impact on both the latency and the WER (Right).
4.2 MINIMUM WORD ERROR RATE
Table 3 shows a significant relative improvement of 4-4.5% in Word Error Rate (WER) when ap- plying AWP. It also shows that AWP yields similar results to our adaptation of MWER optimization (MFER.OF)), as suggested by Prabhavalkar et al. (2018) (originally designed for soft alignments models trained with CE loss). This emphasizes that AWP is competitive with application-specif methods while maintaining its general nature and simplicity. Improvement WER also gained with various implementations of the property function, can be seen in Appendix C.
Furthermore, proposed framework proves to be versatile, as it successfully operates both streaming (Stacked ResNet) and offline (Wav2Vec2) architectures. The ability our approach to adapt different architectures highlights its applicability across various ASR systems.
8

---

Published as a conference paper at ICLR 2024
Table 3: WER of baseline models and of models optimized for WER with AWP or MWER_OPT.
Model
Start Epoch
% WER Libri Test-Clean
% WER Libri Test-Other
(% Relative improvement)
(% Relative improvement)
Stacked ResNet
2.63
7.46
+AWP
4.3
2.57 (2.2)
7.16 (4)
+MWER OPT
4.3
2.54 (3.4)
7.31 (2)
Wav2Vec +AWP
2.38
5,82
2.3
2.33 (2.1)
5.56 (4.5)
5 DISCUSSION & FUTURE WORK
The results obtained from our study provide valuable insights regarding the potential for improve- ment in ASR models trained with the CTC criterion. Although not tested, this framework could be easily applied to other hard-alignment criteria such as Transducer (Graves) 2012). Furthermore, by adapting and extending the concepts from our framework, it may be possible enhance soft- alignment methods, even in domains beyond ASR.
In addition, intriguing aspect for future research is the formalization of the properties that can be enhanced using AWP. By establishing a formal framework, researchers can systematically identify, define, and prioritize the properties to be enhanced. This can lead to targeted improvements and deeper understanding of the impact of different properties on ASR performance. Finally, our study showcases the capability of enhancing single property at time. In some applications, multiple properties should be enhanced simultaneously. potentially leading to better performance. could be especially intriguing in scenarios where the distinct properties exhibit trade-off, like the low latency and WER properties. Utilizing AWP on both properties can provide a more nuanced control over their trade-off.
6 LIMITATION & BROADER IMPACT
Although the AWP framework relatively easy to use, its main limitation is that one needs to think carefully about the property function fprop- When formulated elegantly, the implementation is straight forward.
The proposed AWP framework enables one to enhance a desired property of an ASR model trained with CTC. As mentioned 5] this method can be applied adapted to domains other than ASR. On the choice of the property to enhance, especially in generative AI, one should be thoughtful not increase bias, malicious racist content of models.
7 CONCLUSIONS
The dichotomy between perfect and imperfect alignments in CTC highlights its limitation in cap- turing additional alignment properties, which key requirement in many real-world applications. To overcome this limitation, we introduce Align With Purpose, a general Plug-and-Play framework designed to enhance specific properties in models trained using the CTC criterion. Our experimental results demonstrate promising outcomes in two key aspects: latency and minimum Word Error Rate optimization. Importantly, these optimizations are independent of each other, highlighting the ver- satility of AWP. The reduced latency achieved by our approach indicates faster transcription while maintaining transcription quality even with significantly reduced drift. Furthermore, our improved WER emphasizes the importance in enabling differentiation between imperfect alignments for en- hancing the transcription quality of ASR systems. One of the strengths of AWP lies in its generality. It offers flexibility in selecting specific alignment properties, applies to large-scale training datasets, and is versatile to architectural choice. Our method does not require modifications to the CTC loss function and can be implemented using only few lines of code.

---

Published as a conference paper at ICLR 2024
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intel- ligence, 44(12):8717-872) 2018.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0; A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449 15s11449-12000 2020.
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv: 2303:2333300747, 2023.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4960 4964. IEEE, 2016.
Cisco. Vni complete forecast highlights, 2018.
TTtpLtpp://ww.ccccc...oo//////// m/ en wtt.a//wwwtttooeee...eeeeerraeaa
G Global htt._errro___erooot_aaaaaaaaaoop
Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Wav2letter: an end-to-end convnet-
based speech recognition system. arXiv preprint arXiv:1609.03193, 2016.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint
arXiv:1211.37112011 2012.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pp. 1764-1772, PMLR, 2014.
Alex Graves and Jürgen Schmidhuber. Offline handwriting recognition with multidimensional re-
current neural networks. Advances neural information processing systems, 21, 2008.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmennt transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 141.5554 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Chenyang Huang, Hao Zhou, Osmar R Zaiane, Lili Mou, and Lei Li. Non-autoregressivv translation with layer-wise prediction and deep supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10776-10784. 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax, arXiv preprint ar6iv:161 1.0/ 144, 2016
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Aleksandr Laptev, Vladimir Bataev, Igor Gitman, and Boris Ginsburg. Powerful and extensible wfst framework for rnn rnn-transee transducer losses. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEF, 2023.
10

---

Published as a conference paper at ICLR 2024
Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arxiv 2019. arXiv preprint arXiv:1908.03265, 2019.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203. 200::2203..8044 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acouslics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Chene Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 19XX:x:99440077700 2019.
Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Cheng Chiu, and Anjuli Kannan. Minimum word enror rate training for attention-baun sequence-to- to sequence models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pp. 4839-4843 IEEE, 2018.
Vineel Pratap, Qiantong Xu, Jacob Kahn, Gilad Avidov, Tatiana Likhomanenko, Awmi Hannun, Vi- taliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Scaling up online speech recognition using convnets. arXiv preprint arXiv:2001.000 /09927, 2020.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference on Ma- chine Learning, pp. 18492-28518.. PMLR, 2023.
Yusuke Shinohara and Shinji Watanabe. Minimum latency training of sequence transducers for streaming end-to-end speech recognition. In Proc. Interspeech 2022, pp. 2098-2102, 2022. doi:
10.21437/lntesseecch.o 20t...2022222000000
Xingchen Song, Di Wu, Zhiyong Wu, Binbin Zhang, Yuekai Zhang, Zhendong Peng, Wenpeng Li, Fuping Pan, and Changbao Zhu. Trimtail: Low-latency streaming ast with simple but effec- tive spectrogram-level length penalty. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jinchuan Tian, Brian Yan, Jianwei Yu, Chao Weng, Dong Yu, and Shinji Watanabe. Bayes risk ctc: Controllable ctc alignment sequence-to-sequence tasks. arXiv preprint arXiv:2210.07499, 2022.
Zhengkun Tian, Hongyu Xiang, Min Li, Feifei Lin, Ke Ding, and Guanglu Wan. Peak-first etc: Reducing the peak latency of ctc models by applying peak-first regularization. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00399 2021.
Yongqiang Wang. Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer-based acous- tic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6874-6878. IEEE, 2020.
Wei Yang, Yang Liu, and Chunjing Xiao. Deep metric learning for accurate protein secondary structure prediction. Knowledge-Bavel Systems, 242: 202:10 2022.

---

Published as a conference paper at ICLR 2024
Zengwei Yao, Wei Kang, Fangjun Kuang, Liyong Guo, Xiaoyu Yang. Yifan Yang, Long Lin, and Daniel Povey. Delay-pent Malay-peellized etc implemented based on finite state transducer. arXiv preprint arXiv:2305.11539, 2023.
Jiahui Yu, Chung-Cheng Chiu, Bo Li, Shuo-yiin Chang, Tara N Sainath, Yanzhang He, Arun Narayanan, Wei Han, Anmol Gulati, Yonghui Wu, et al. Fastemit: Low-latency streaming asr with sequence-level emission regularization. In ICASSP 2021-2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 6004-6008. IEEE, 2021.
Albert Zeyer, Ralf Schlüter, and Hermann Ney. Why does ctc result in peaky behavior? arXiv preprint arXiv:2105 2021.
A APPENDIX: ADDITIONAL EXPERIMENTAL SETTINGS
A.1 GENERAL EXPERIMENTAL SETTINGS
Datasets. For the small scale, we train models on the LibriSpeech dataset (Panayotov et al., 2015 which consists of 960 training hours (LS--60).. For the medium scale, we train models on a 35K hours curated subset of Libri (LV-35K), where samples with low confidence of reference model were filtered out. For the large scale, we train models on an internal dataset of 280K hours of audio-transcript pairs (Internal-280K), which, to the best of our knowledge, is the largest dataset that was used to train a low-latency model. We test our framework on the test splits of LibriSpeech. Audio is sampled at 16KHz, 16 bits/sample.
Architecture. We trained Stacked ResNet (He et al. 2016). Wav2Vec2 (Baevski et al. 2020). and Conformer (Gulati et al. 2020) models. The ResNet consists of 20 ResNet blocks (66M parameters). For the Wav2Vec2, used pre-trained version of the base Wav2Vec2 model (90M parameters) available on HuggingFace 6 The model was pre-trained for 30 epochs on the 100K hours from VoxPopuli dataset (Wang et al., 2021). Lastly, trained medium Conformer model (30.7M 7M parameters) implemented according to the setup described in (Gulati et al.) 2020). All models output 29 English lower-case characters, including upostrophes, spaces, and blank tokens.
Regarding the Stacked ResNet model, we extracted 80-channel Mel filter-banks features computed from 32ms window with a stride of 16ms. For each frame, we stacked the filter banks with a first and second derivative, resulting in 240-dimensiioal input vector. We down-sample the audio input from 16ms to 32ms by applying MaxPool layer within the first layer of the first Stacked ResNet block, then stacked 20 ResNet blocks (He et al., 2016) with a kernel size of 5. Skip connections are added every 4 ResNet blocks. The model consists of 66M parameters in total. This architecture induces 6.4 seconds of context in total. The results are shown using an exponential moving average (EMA) model, which is aggregated alongside the model.
Decoding. Models were decoded using in-house implementation of beam search decoder de- scribed in (Graves & Jaitly] 2014), using a beam size of 100, and two language models: an open- source 5-gram language mode (WordLM) trained the LibriSpeech LM corpus, and a character- level language model (CharLM) that we trained on the same corpus. The beam search picks tran- scriptions y which maximize the quantity L(y) defined by:
L(y P Pacouustic (y)x² + PPcharLM(V) + 7 PWonaLM (1)
where / = 0.8 and 7 = 0.8 are the CharLM and WordLM weights, respectively.
Text Normalization. We used an in-house implementation of text normalization to remain in a vocabulary of 29 English characters.
A.2 Low LATENCY EXPERIMENTAL SETTINGS
Architecture. Experiments detailed in this section are conducted with the Stacked ResNet and desscrcee in A.
Conformer architecturesstteeeeeeeeeeeeese described
httts://ww..pensslr.rrgg.//
httpss://ww..gesssr..rr////
12

---

Published as a conference paper at ICLR 2024
The ResNet architecture be implemented in streaming manner and can be highly optimized for edge devices. Therefore, it's a natural choice for an online system in a low- resource environment. Our offline version of it has 6.4 seconds of context in total, divided equally between past and future contexts. Although the model can be implemented in a streaming fashion, it has a large DCL of 3.2s. The online version has similar architecture and the same total context, but it has DCL of 430ms, achieved by asymmetric padding as suggested by Pratap et al. (2020). The small DCL of this model makes feasible to deploy an online ASR system.
For the Conformer, we only trained it in an offline fashion, with or without AWP. To transform the Conformer into an online model, at inference the DCL was to restricted 430ms and the left context was restricted 5.57s. The input was fed into the model chunk by chunk, similarly to TTian et al. 2022).
Training. For training the ResNet with AWP on LS-960, LV-35K, and Internal-280K, the hyper- parameters a and were set to 0.001, 0.001, 0.0005, and 0.01, 0, O, respectively.
RAdam optimizer (Liu al. 2019) with a = 0.9, B = 0.999 and weight decay of 0.0001 were used. We set the LR to 0.001, with ReduceLLRnnPltteen scheduler
To train the Conformer with AWP, scheduling for was required 0.1 at commencement and 1e-6 after 7K training steps. À remained constant and was set to 0.01.
For both models, we set = 5, the number of sampled alignments.
Measuring DL. Measuring the DI. of an online model is relative to the offline model of the same ar- chitecture that was trained the same data. To measure the DI., we force-align the target transcript (GT) of the offline and online models independently and take the difference between the index of the first appearance of each token in the two force-aligned texts. Then, we take the average difference between all tokens. We empirically verified that the DL of the offline models compared to the true emission time is negligible, and for some instances, it even appears to be negative. This behavior was also observed by (Tian et al. (023).
A.3 MINIMUM WORD ERROR RATE EXPERIMENTAL. SETTINGS
Architecture. In this setting, we applied AWP to a Stacked ResNet and a Wav2Vec2 models, as described in Subsection[A.I The Stacked ResNet model that was used here is the same as the offline model described in subsection A.2
Training. The baseline Stacked ResNet model was pre-trained on the -20KK dataset. Then we continue its training solely on LS-960 for 4.3 epochs before we apply AWP. The AWP hyper- parameters were a = 0.1, À = 0. The baseline and the model with AWP were trained for 4.2 additional epochs, reaching 8.5 epochs in total. We used the RAdam optimizer (Liu et al. 2019) with the same hyper parameters as in subsection A.
The Wav2Vec2 baseline model was finetuned with SpecAugment (Park et al. 2019) (with p=0.05 for time masking and p=0.0016 for channel masking) solely on LS-960 for 2.3 epochs before applied AWP, and both the baseline and the AWP models were trained for another 27.5 epochs. We used the Adam optimizer (Kingma & Ba 2014) for this training, well
flat LR scheduler
Baevski et al. (2020). AWP
were set to = 0.05 and X = 0.
While training all models with AWP, we used softmax temperature of 0.5 for the sampling of the N alignments. Additionally, we set N- 10 under these settings.
B SAMPLING METHOD
Throughout our experiments, used the standard torch library for sampling To verify that the results weren't I compromised by the lack of differentiability of the sampling process, we conducted
similar experiments with Gumbel Softmax (Jang et al. 2016). As can be seen in Fig. 7 the Gumbel Softmax had not effect on results.
RedduceluKomllateu..htt
www.hh/mmtoor.utttii. www.tttttaaaaa..a....oo...a..a.. data. Sampler
13

---

Published as a conference paper at ICLR 2024
WSR w/wo Cumble-
Drifting wiwo Gumble-
36
gumble soitmaa
gumbie softmax
gumble
96
250
200
04
W5R
Unitiin
1.0
93-
Train ng step
Training step
Figure 7: Stacked ResNet low lateney model, with AWP applied after 2.7 epochs. The sampling method was either the standard torch implementation Gumbel Softmax. It can be seen that the sampling method has minimal impact on the drifting (DL) and on the WSR, which the word success rate (100-WER).
C VARIANTS OF THE PROPERTY FUNCTION
For both applications, low latency and WER, we verified that AWP is robust to the choice of the property function. Table shows different implementations for the low latency function presented in section 2.3 Specifically, instead of shifting alignments by one token to the left, shift them by multiple tokens in random positions. It can be seen that gradual changes in alignments, iee fewer shifts, have positive effect improvement is far greater than a shift in a few tokens, as the model improves throughout the training process, and every training step it is optimized improve its (current) latency by a shift of few tokens.
the latency. Although the changes are gradual, the overall
Table Different property function implementations for the Low Latency application. The different implementations vary in the number of tokens to shift and are reported after the same number of training steps.
Tokens Shifted
DL (ms)
WER
-79
4.38
2
-68
4.46
4
21
4.39
shows shoows variation for the property function defined in section 2.4 when correcting words instead of
As for the WER application, table
Table 5: Different property function implementations for the WER application. The different imple- mentations vary in the number of words to correct and are reported after the same number of training steps.
Words Corrected
% WER Libri Test-Clean
% WER Libri Test-Other
1
2.57
7.16
2
2.54
7.29
14

---

Published as a conference paper at ICLR 2024
D DETAILS AND RESULTS OF PRIOR WORK
In this section, we present both the findings and the specific details of prior work that address latency reduction in CTC training. Since most of the prior work used different datasets and architectures, we implemented or adjusted existing implementations to work in our setting.
Methods. The different methods we tried are TrimTail (Song et al. 2023). Peak First CTC (PFR)
(Tinn 2023 and Bayes Risk CTC (BRCTC) TTan al. 2022)
Implementation. We implemented TrimTaal and PFR in accordance with the relevant paper. In the case of BRCTC, we utilized an open-source implementation from ESPnet library ID]
Experimental Settings And Results. We trained the models on LS-960, using the same online ResNet model as described
Decoding, training optimizer and scheduler are the same as in A.2
As for the hyper-parameters, the TrimTaai has a max trim value, PFR has a weight and BRCTC has risk factor value. These parameters responsible for controlling the method's effectiveness. For loss_CTC the PFR we used a dynamic weight w with a fixed ratio a, s.l. weloss PFR w.
We conducted extensive research in order to find the best hyper-parameters that fit our model and setup to achieve the best drift latency with minimum degradation in the WER results. The best parameters we found are max max.trim_vve valuee00 for TrimTail, a=0.001 for PFR, and risk factoor00000000 for Bayes Risk. We used this set of parameters for the results reported in table
Table 6 presents the full results of these models using various ranges of hyper-parameters.
Table 6: Other low latency methods results. DL was calculated using the Offline model trained on LS-960 dataset. Results are Libri Test-Clean.
Model
Hyper-parameter
DL (ms)
TL (ms)
WER
Peak First CTC
a=0.01
198
628
4.22
Peak First CTC
a=0.001
235
665
4.11
Peak First CTC
a=00000
186
616
4.41
Trim'Tai
max_trim=30
47
477
4.38
Trim
max_trim=50
-76
354
4.46
TrimTail
max max_trim=70
-126
304
4.79
Bayes Risk
risk factor==000
199
629
4.21
Bayes Risk
risk factor=200
63
493
4.78
Bayes Risk
250_
91
521
4.82
E FURTHER COMPARISON BETWEEN AWP AND PRIOR WORK
To enable additional comparison between AWP and prior work, table shows that for similar WER value, the latency varies across methods. This specific WER value was taken since this was the minimal WER that one of the other methods achieved. It can be seen that AWP outperforms in terms of latency.
15

---

Published as a conference paper at ICLR 2024
Table 7: AWP and other low lateney methods results, when the WER is similar across the different methods. The models were trained on LS-960 dataset and are the same models used in table 2 DL was calculated using the Stacked ResNet Offline model trained on LS-960 dataset. Results are on Libri Test-Clean.
Method
WER
DL (ms)
Peak First CTC
4.7
186
Bayes Risk
4.78
64
TrimTail
4.81
-59
AWP
4.75
-96
F DETAILED Low LATENCY EXPERIMENTAL RESULTS
Table Full results with and without AWP, with other frameworks and on different data scales. Results are on Libri Test-Clean.
Model
Training Data
Start Epoch
DL (ms)
TL (ms)
WER
Stacked ResNet Offline Stacked ResNet Online
Internal-2804
0
3.2K
2.34
Internal-280K
249
679
2.6
+AWP
Internal-280K
0.03
33
463
3.13
+AWP
Internal-280K
5.7
50
480
2.71
Stacked ResNet Offline
LV-35K
0
3.2K
2.42
Stacked ResNet Online
LV-35K
341
771
2.72
+AWP
LV-35K
0.1
-251
179
3.28
Stacked ResNet Offline
LS-960
0
3.2K
3.72
Stacked ResNet Online
LS-960
278
708
4.06
+AWP
LS-960
0.9
-79
351
4.38
+AWP
LS-960
2.7
-54
376
4.11
+AWP
LS-960
5.4
-18
412
4.13
+AWP
LS-960
7.2
-32
398
4.07
+AWP
LS-960
-24
406
3.92
+AWP
LS-960
10.8
53
483
4.06
+AWP
LS-960
16.7
136
566
3.92
+Peak First CTC
LS-960
186
616
4.41
TrimmTail
LS-960
354
4.46
+Bayes Risk
LS-960
63
493
4.78
Conformer Offline
LS-960
0
3.7
+AWP
LS-960
12
-172
3.74
Table 9: Low Latency model training w/ & w/o AWP, with different AWP loss weight. Results are on Libri Test-Clean
Model
Training Data
AWP loss weight
DL (ms)
TL (ms)
WER
Stacked ResNet Offline
LS-960
0
3.2K
3.72
Stacked ResNet Online
LS-960
278
708
4.06
+AWP
I.S-600
0.01
-402
28
6.52
+AWP
ISS960
0.005
-346
84
5.69
+AWP
LS-960
0.001
-54
376
4.11
+AWP
LS-960
0.0005
114
566
3.9
16


Reviews:
[{"review_id": "kXxPZpCKUv", "reviewer": "Reviewer_Nofc", "summary": "The paper proposes a framework to train a CTC model with a desired property by adding an auxiliary loss. It samples N alignments from a pre-trained CTC model, feeds them to a property-designed function to get N better alignments w.r.t. the property, then adds a hinge loss on each pair of alignments as an auxiliary loss to the original CTC loss, to increase the probability of the better alignments with the desired property. The proposed framework is experimentally tested in applications to optimize latency and WER respectively and has shown improvement of the designed property compared to the vanilla CTC training and some other existing approaches.", "questions": "- In Figure 6: What is WSR? It is not a standard metric and it is not defined anywhere.\n\nSee other questions above in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework is flexible and simple enough to generalize to different properties and provides a generic way to makes the CTC training more controllable.\n- For the low latency application, the proposed framework achieves better latency and quality tradeoff than a few existing approaches and be on par with another best approach (TrimTail).\n- For the minimum word error rate application, the proposed framework achieves some modest improvement over the MLE baseline.", "weaknesses": "- For the minimum word error rate application, there has been a number of work in optimizing it in a discriminative sequence training setting for ASR, e.g. MBR training, large margin training, etc. The proposed framework should be compared to those stronger baselines instead. Right now it is only compared to the weaker vanilla MLE baseline with some modest improvement. In particular, if the property function is to generate the ground truth alignment, instead of only allowing reducing 1 word error, then it should be closer to the traditional discriminative sequence training setup.\n- For the minimum word error rate application, Prabhavalkar et al. 2018 found that using the n-best beam search hypotheses is more effective than the sampling-based approach. This paper should do a similar comparison whether it should compute the property function in the n-best alignments instead of sampled alignments.\n- Latency and WER optimizations typically compete with each other. It would be great to utilize the proposed framework to optimize these two properties jointly with a single property function to see if they can be balanced better together, and also see how effective and generalizable the framework is.\n- How sensitive is the optimization to the specific choice of the property function? E.g. the currently designed latency function only allows one time step faster, and property function for the minimum WER application only allows one word error reduction. Are these choices made in order to stabilize the training, or actually they can be relaxed to allow more changes as well? The paper should compare more different property function choices for the same specific application.\n- For the latency application, another intuitive approach would be to sample an alignment corresponding to the ground truth label sequence, and then the property function would be to run a force aligner to get the more accurate time alignment for the label sequence. How would this compare to the current proposed approach?\n- Adding the latency optimization to an offline Conformer model doesn't make much sense, since the full-context model is not used in a streaming fashion and it has to wait for the entire sentence to come first, which by itself already has a much higher latency. Conformer can be implemented in a streaming manner as well by just using the left context, which can be optimized for edge devices as well. The latency experiment should be conducted on an online Conformer.\n- Using \"start epoch\" as a tunable hyperparameter to control the balance between latency and quality is a bit strange. How transferable is the optimal start epoch to different learning schedule, model architectures and data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zGqLgy5Ocj", "reviewer": "Reviewer_bLBJ", "summary": "For speech recognition task with CTC loss (and others) we assume equal weights between different alignments and we optimize the total probability of all correct alignments. Authors of the paper are concerned about the latter fact and push for considering different weights for different alignments. Authors propose an independent loss, plug and play, which will control what alignments are more preferable to inherit desired properties, like latency (emit tokens faster, without delay and drift) and minimum WER instead of loss. They claim that proposed method is simple compared to prior works and adds few lines of code with no need to change the main loss function (e.g. CTC) used in training. Results on low latency and minimum WER for different models (transformer, conv), data scale and tasks show validity of the proposed idea.", "questions": "There are many typos in the text, including missing dots in the end of sentences, usage of words with capitalisation for the first letter and some ambiguity in sentence formulation, style of citations in brackets or without brackets, dashes usage. Proof-read is needed for the final revision.\n\nComments / Questions / Suggestions\n- \"CTC posteriors tend to be peaky (Zeyer et al., 2021; Tian et al., 2022), and hence the posterior of one specific alignment is dominant over the others.\" I would smooth this formulation a bit, as likely several tokens are dominant for each time frame, and thus set of alignments (few) are dominant, not only one as discussed in a bit in Likhomanenko, T., Collobert, R., Jaitly, N., & Bengio, S. (2023, February). Continuous Soft Pseudo-Labeling in ASR. In Proceedings on (pp. 66-84). PMLR.\n- what will happen if we choose top-N alignments instead of sampling them? \n- what will happen if we use several tokens removal for the latency function instead of only 1 token? is it improving latency?\n- \"35K hours curated from LibriVox\" is it multilingual or English only? If multilingual, why not English only as then another confound factor is introduced?\n- I would suggest to report results with both greedy decoding and language model, also report both clean and noisy LibriSpeech as a lot of effects are not visible on clean anymore.\n- Throughout the text it is not clear where LM is used or not in the reported numbers.\n- Seems in Table 2 performance of conformer model is not so good as in the prior paper (3.7 vs 2.0), or check Squeezeformer baselines.\n- What is WSR abbreviation (I could not find this notation in the text)? Figure 6 is hard to parse in the current form.\n- Why are multilingual wav2vec used for experiments? why not English only?\n- I found it surprising that Gumbel softmax with temperature is similar to the standard sampling. Some discussion would be helpful on this topic in the main body, as seems we are very limited with potential improvements if we manipulate with alignments weights.\n- what is the relation between Table 2 and Table 4 for the prior methods? I see that Table 4 has better results in WER than in Table 2.\n- why are features computed with \"32ms window with a stride of 16ms\"? This is really very non-standard Mel filter banks extraction for ASR models.\n\n**Update after rebuttal**\n\nThanks again for clarifications, additional ablations and patience for my response. I had read again the whole discussion with all reviewers as well as had another pass over the final version of the paper.\n\nI think most of the main concerns are resolved now:\n- robustness - usage of the proposed loss from 80-90% WER model is a nice and simple empirical rule\n- implementation of the property function for min WER - looks good to me in the code\n- ablation that beam didn't help totally makes sense to me, though still interesting that sampling is enough (maybe due to peaky CTC distribution and thus it samples from top hypothesis still)\n- proper comparison for Table 2 - I think this is mainly resolved for the comparison with prior works, though still I find the baseline Resnet online vs Resnet online + AWP not directly comparable, but Table 8 with different hyper-parameters makes it then clear that AWP either improves both latency and WER or it improves latency for the same WER performance for all models considered.\n\nThe remaining concerns are\n- improvements for AWP in min WER task is marginal having std 0.03 for clean set (while for latency task AWP really has significant improvement). I still would like to see greedy decoding results for the final version of the paper, as maybe it improves more, but doesn't make significant improvement after beam-search decoding with LM (as LM knowledge is transferred into acoustic model). This can give more insights in future.\n- Results with more tokens shifted or more words correction: it doesn't really improve model - this either show that method is limited or that we incorporate all things over the course of the training (which is possible). In future, will be valuable to have some analysis on that to show if this is really limitation of the method or not.\n\nBased on the above, I am raising score from 3 to 6 (marginally above the acceptance threshold) and soundness and contribution from 2 to 3 both, as I believe results on latency improvement are solid enough, though results on min WER are not very supportive (maybe a weaker models are needed as at the limit of 50k-100k hours of labeled data we don't need anything to have very good models) and thus shows that method has limited impact.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- exploring idea on reweighing alignments depending on the task / desired properties, e.g. better latency\n- simplicity of the method, including plugin property instead of modifying CTC loss itself", "weaknesses": "- absence of comparison of proposed method with prior works and baselines (e.g. if we introduce reweighing of alignments directly in the CTC loss)\n- complexity of additional hyper-parameters choice (no robustness), e.g. when we start the additional proposed loss function optimization\n- missing details on how exactly the sampling of the word to be corrected is implemented, as it could be that no word is available for substitution. Why WER and not CER as language model could fix the errors? Is language model used in this process?\n- I believe that sampling alignments is not the right / optimal way: we could consider optimization of the top-N alignments instead, as otherwise we spend time on optimization low probability alignments. \n- A bunch of models in Table 2 (on latency empirical results) are not comparable: either latency should be fixed and WER is compared or vice versa. Right now it is hard to make any conclusions from Table 1 due to different values for latency and WER of different models.\n- Results in Table 3 are within std on Librispeech as 0.1 variation is normal between different models often for clean part of test set. Also it is not clear if improvement is consistent for both greedy and LM decoding or only for the latter one. Greedy decoding should be reported too to make full clear picture how the proposed method improves results.\n- I found it overall hard to formulate the proper reweighing between different alignments, rather than simple way of controlling the latency by restricting context or optimising directly mWER. It is not clear why proposed way of sampling alignments is sufficient or significantly beneficial. Overall, reported results are only marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oxaApHlpU3", "reviewer": "Reviewer_jMFR", "summary": "The paper proposes a plug-and-play framework to CTC loss so as to improve the performance of trained model on a specific perspective. The preference is achieved by a hinge loss calculated between an example and a better example. The experiments on ASR with different data scale show that the proposed method can help model to recognize text promptly or accurately.", "questions": "It seems that start epoch is a sensitive parameter, which is different in different model. So how is start epoch defined? By grid search?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The design of Align With Purpose is nice and interesting. The method makes use of a fact that in conventional CTC loss, all the perfect alignment are treated equally. In this case, the preference can be achieved by helping model compare to possible paths. The idea is clear and the paper is well-written. The experiments also verify the effectiveness of proposed method.", "weaknesses": "I do not witness obvious weakness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPvooKME9D", "reviewer": "Reviewer_4pZf", "summary": "This paper proposes a novel method for improving the desired properties (latency and accuracy) of CTC-based speech recognition models. The core idea of the proposed method, called AWP, is to distinguish different alignment paths by prioritizing the one exhibiting the better property using an additional loss term. To promote such properties, simple rule-based strategies are employed to modify the alignment. For example, one may shift 1 token to generate a ‘worse’ (delayed) alignment path. Experiments conducted on online/offline ASR models show that AWP can boost the desired properties compared to previous baselines.", "questions": "* What is WSR in Figures 6 and 7? Is it (100 – WER)?\n* It seems that AWP needs random sampling at each training step. How many alignments (N) do you sample for each step? How does AWP affect the overall training time/resource usage?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is very well-written and easy to understand. Especially, the introduction and related work sections are such a joy to read.\n* The comparative experiments against previous methods (including very recent ones) are conducted under the same condition. The result demonstrates the effectiveness of AWP (Table 2).\n* The proposed method appears to be a novel CTC modification utilizing a sampling-based hinge loss function. The approach is clearly different from previous methods pursuing similar objectives.", "weaknesses": "* In Table 2, only the ‘Stacked ResNet Online’ model case is compared with other methods. It would strengthen the claim if there exists more comparison for Conformer-Online + AWP, Peak First CTC, Trim-Tail, etc.\n* While the latency reduction part presents extensive experimental results (various training data sizes, comparison, ablation, ...), there are not many results on minimum WER training. Furthermore, the gain from minimum WER training is marginal.\n* There seems to be room for improvement; for example, how about increasing the number of shifted frames (tokens) instead of selecting just one? How about applying AWP together with Trim-Tail? I appreciate the simplicity of the proposed method, but I am also curious about the limitations of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)