W-CTC: a Connectionist Temporal Classification Loss with Wild Cards
Authors: Xingyu Cai
Jiahong Yuan
Yuchen Bian
Guangxu Xun
Jiaji Huang
Kenneth Church
Venue: ICLR 2022 Poster (2022)
Keywords: CTC
wild cards
dynamic programing
partial alignment

Abstract: Connectionist Temporal Classification (CTC) loss is commonly used in sequence learning applications. For example, in Automatic Speech Recognition (ASR) task, the training data consists of pairs of audio (input sequence) and text (output label),without temporal alignment information. Standard CTC computes a loss by aggregating over all possible alignment paths, that map the entire sequence to the entire label (full alignment). However, in practice, there are often cases where the label is incomplete. Specifically, we solve the partial alignment problem where the label only matches a middle part of the sequence. This paper proposes the wild-card CTC (W-CTC) to address this issue, by padding wild-cards at both ends of the labels. Consequently, the proposed W-CTC improves the standard CTC via aggregating  over even more alignment paths. Evaluations on a number of tasks in speech and vision domains, show that the proposed W-CTC consistently outperforms the standard CTC by a large margin when label is incomplete. The effectiveness of the proposed method is further confirmed in an ablation study.

Published as a conference paper at ICLR 2022
W-CTC: A CONNECTIONIST TEMPORAL CLASSIFICA- TION LOSS WITH WILD CARDS
Xingyu Cai, Jiahong Yuan, Yuchen Bian, Guangxu Xun, Jiaji Huang, Kenneth Church Baidu Research, 1 1195 Bordeaux Dr, Sunnyvale, CA 94089, USA xingyucai @baidu.com
ABSTRACT
Connectionist Temporal Classification (CTC) loss is commonly used in sequence learning applications. For example, in Automatic Speech Recognition (ASR) task, the training data consists of pairs of audio (input sequence) and text (output label), without temporal aligmment information. Standard CTC computes a loss by ag- gregating over all possible alignment paths, that map the entire sequence the entire label (full alignment). However, in practice, there are often cases where the label incomplete. Specifically, we solve the partial alignment problem where the label only matches a middle part of the sequence. This paper proposes the wild-card CTC (W-CTC) to address this issue, by padding wild-cards both ends of the labels. Consequently, the proposed W-CTC improves the standard CTC via aggregating over even more alignment paths. Evaluations on a number tasks in speech and vision domains, show that the proposed W-CTC consistently outperforms the standard CTC by a large margin when label is incomplete. The effectiveness of the proposed method is further confirmed in an ablation study.
INTRODUCTION
Connectionist Temporal Classification (CTC) was proposed in (Graves et al. 2006) to train end- to-end sequence learning models. has become the most popular loss function in many sequence learning applications, such as Automatic Speech Recognition (ASR) (Amodei et al. 20066] Batten- berg et al. 2017) and Optical Character Recognition (OCR) (Liu et al. 2015; Subramani et al. [2020] Chen et al. I021I. CTC has number of attractive properties. For example, it comes with clear interpretation that maximizes the posterior marginal probability summed over all possible alignment paths. In addition, CTC loss can be easily incorporated into neural nets because is fully differen- tiable. The gradients can be estimated efficiently by borrowing the well-known forward-backwer algorithm from Hidden Markov Models (HMMs) (Rabiner) 1989).
One limitation of standard CTC is that it assumes the label matches the whole input sequence. CTC performs full alignment between them. However, we often face the partial alignment in practice. For example. a speech transcription often omits the prefacing introduction of the speaker, a beginning of an audio book that tells who read the book, along with various other details, are typically not in the written version; sign language interpreter often ignore the irrelevant gestures in the beginning and ending stages; etc. The above motivating examples are also referred as the streaming problem (Sakurai et al.) 2007). In streaming speech or video, we only capture fraction of interested segments to form the labels. It is also common that the label is corrupted or noisy due to variable recording conditions. Partial aligmment is useful in semi-supervised or transfer learning as well, which enables us to use a coarse model (from a different domain) to automatically generate labels. We can discard the low-confident outputs and form the incomplete, but more accurate labels, then feed them into the target model to train. In time series domain, streaming Dynamic Time Warping (S-DTW) was proposed (Sakurai et al. 2007) for partial warping, and Cai et al. (2019) extended it to differentiable training in neural nets. Inspired by their work, we propose the wild-card enhanced CTC, dubbed as W-CTC, perform the end-to-end partial-alignment sequence learning. To the best of the authors' knowledge, this is the first study to apply CTC on incomplete labels.
1.1 PROBLEM FORMULATION
Let X = X1..X1..... X be the input sequence of length T. e.g., speech audio frames for an ASR system, or image segments for an OCR system. Let Y = [Y, Yi, Yx-1] be the corre- sponding targets (labels) of length N, e.g., the transcription for ASR, or the recognized string for

---

Published as a conference paper at ICLR 2022
OCR. Assume T 2 N. Though both X and Y are provided, we don't 't know how they are aligned.
Let the alignment function be vp : X + Y, which maps a frame x E X to a target y E Y. Define:
Definition 1 (Subsequence), A subsequence of X, denoted as X C X, is a vector of multiple consecutive elements in X, i.e. X = X
where T >.0. 0.
Definition (Full alignment). A full alignment maps the entire to the entire Y, L.e. y = (b)) E Y, for any E X, as well as Definition 3 (Alignment path). An alignment path, n. is a vector of the same length as X, i.e. [30 41 ................................ ................................ ............ V/- where each element yi (X,) E Y, for 0 Si <iTT T.
(v) E x, for any y E E.
Definition 4 (Valid alignment). An alignment path is valid, if it satisfies:
Monotonicity: if X; maps to Y,, Xi+t cannot map to Yj-k. any t > 0 and k: O. Surjection (many-to-one mapping): (b() is surjective function, such that subsequence
X C X, could maps to single Y3 E Y. But each X; can have only one label y E Y.
Definition f is a common assumption, because we often have a higher sampling rate in audios or images, for a better resolution. In consequence, T 2 N, thereafter multiple frames correspond to a single target. In practice, X could be the sequence of frame embedding vectors, i.c. X, E Rd, obtained by a neural network. Provided the targets Y, want to train the neural network by implicitly learning the alignment between X and Y. This is a full alignment problem and often handled by CTC loss (Graves al. 2006) attention mechanism (Bahdanau al. 2014).
In this paper, we extend the full alignment problem, to a more general partial alignment case, where the input could be incomplete. That is, the input sequence X, does not necessarily have an end-to- end mapping to the label Y (violate Definition 2 In particular, we solve the missing Y problem, where X is complete. For example, the annotated transcription is incomplete, or corrupted, such that it does not correspond to the full audio speech. It is sometimes referred as the streaming alignment problem, as X can be seen as a long stream signal, while Y only matches part of X.
Definition 5 (Partial Alignment Problem). A problem where the label is a subsequence Ý [Yx, Yx+1....., Y++L-1] C Y, that has length L < N. Both k and L are unknown.
Note that the missing portion are at the beginning and the end of Y. A commonly seen example in ASR system, is the silence/noise at the two sides of X. The transcription often does not include such silence/noise remarks, resulting in a mismatch at the two ends. This is already handled by CTC loss with an additional "blank" label (refer (Graves et al. 2006)). We tackle the more complex problem, that the missing part is meaningful rather than just noise/silence.
1.2 BRIEF INTRODUCTION OF CTC
To introduce CTC, let's first consider the full alignment problem. There many paths leading to the same label Y from input X. Let Z be the set of all valid paths. The standard CTC loss (Graves) et al. 2006) is defined as the negative likelioel of the labels given input sequences, i.e.
(cre(X, -og P( = log Σ (X(X) REZ
where P. XY is the probability of particular alignment path For any path, CTC assumes conditional independence amongst X,'s, O S i < T. So the probability of that path, is approx- imated by product of the path-determined label's probability at each frame, i.e. Px(Y[X) z Hlster (=[(4 X). Minimizing the CTC loss is equivalent to maximizing the likelihood of the observed label given the input sequence. CTC is fully differentiable, making it favorable to train an ASR or OCR system, where exact temporal alignment information is provided. CTC is calculated using Dynamic Programming (DP), which will be further analyzed in Section 2.1]
Our target in this paper is the partial alignment problem, which could reduce the full alignment problem, by trying do full alignment between the corrupted targets, Y, and all possible subse- quences of X. Let X = [X Xi-1..... X++r] be one subsequence, starting from i-th frame with a length of 7 + 1. The straight-forward way is to loop over all possible (i, r) pairs, obtain the CTC loss for each subsequence, and summarize them, i.e..
Cere = <T}
(2)
2

---

Published as a conference paper at ICLR 2022
where of) means summary of the set, e.g., summation
This forwad approach is too expensive to compute: For each ( y), DP is utilized, leading to O(T'N) cost. Unfortunately, DP cannot be completely parallelized. Looping through all (i, pairs yields total complexity of O(T A trick that when is fixed, we can reuse the DP trellis from fere(Xi,r; Y) to obtain Lere (xrr(1 Y), at a cost of ((N) instead of O(TN). Even with this trick, O(T²N) is still required as looping over i cannot be avoided. Considering T can be fairly large for audio frames, this approach is hardly deployable in practice.
2
PROPOSED WILD-CARD CTC
2.1 REVIEW OF CTC COMPUTATION
Before presenting W-CTC, we briefly review in detail the Dynamic Programming (DP) based for- ward algorithm to compute the CTC loss in Equation [T Taking an English ASR system an example, let X be sequence of embedding vectors of 7 speech frames, and Y be the correspond- ing transcription of length N. Since there could be silence frames in X, which does not correspond to any English letter, the first step is to augment Y by inserting "blank" label that could represent the silence. "blank" is inserted between any two characters, as well as the beginning and the end of the transcription. For instance, a transcription of "hello" will be augmented to "-h-e-l-l-o-", where "_" represents the "blank" label. The augmented transcription Y has length 2.N 1.
The second step to build trellis (DP matrix) M, which has 2NV rows and T columns, as illustrated Figure la Each node, Ms,j denotes a subproblem's 's likelihood, i.e. M.j = p( i] [ j]). Since the speech signal could end with the transcription's 's last English letter, with additional silence, the final CTC loss a combination of these two cases. More specifically, it the negative log of the sum the two final states, i.e. Lere(X, Y)- = Mog(MNN-113) + M2N,T-1).
These two final-state nodes are colored orange at the right-bottom corner of Figure la
The two green nodes at top-left corner are initial states for DP recursion. The two or- ange nodes bottom-right corner ending nodes for final loss caleulation. The black and blue arrows are allowed transitions based on Equation The red arrows are forbidden.
(a) Standard CTC:
(b) W-CTC: The first corresponds the prepended wild-card "*" symbol (the blue nodes). There are three initial states. The ending nodes are the entire last two rows, rather than only two right- bottom nodes. These changes enable Y to match only a fraction of X.
Figure 1: Illustration of Dynamic Programming based CTC and proposed W-CTC loss calculation.
To fill the trellis, we start from the top-left corner. The value of Mo.o is the probability of "blank" given the first frame, i.c. Mo.o P())X,)). Similarly, M1,0 P(Y (X,) is the probability of transcription's 's first letter. The speech could begin with either silence or the transcription's 's first English letter, so both Ma,o and M1,0 are valid initial states (the green nodes at top-left corner in Figure Lap. Note that the probability of a predicted symbol, i.c. Xo.i) is often obtained by projecting the embedding vector X, logit vector (length is the augmented vocabulary size), then
3

---

Published as a conference paper at ICLR 2022
applying softmax operation. As the prediction at any time step can be conditioned the entire sequence of X (e.g., using a bi-directional LSTM model), write P(Y, X) instead of P(Y,|Xoo)) to account for the more general cases. Referring to the illustration in fig. la the rules for updating the remaining values Ma are as follows:
X() (Mas-1+ + M1-15-1), if Y = 4. or Y
Mi = X (Maj-1 + 11-1-11 + M2 21),,11), otherwise
(3)
The line 1 in Equation does not have the term M,- 2, which means: (1) If Y; is "blank", the path cannot come from skip "blank" before (M,-2). (2) If two consecutive letters are the same, the second 'I" cannot directly have a path from the node corresponding to the first "I". In this case, the "blank" symbol between two 'I"s plays a separator role, enabling the decoder to know there are two "I"s instead of one. As a result, the red arrows in Figure la are forbidden. In other cases, the node Mi,j could have path from M;-2,j, so the blue arrows are allowed in Figure Ia
e.g., "-I-I-" in "-h-e-I-l-o-", the node corresponding
Following above DP recursion rules, the trellis M is filled and the values of the two ending nodes (orange nodes at bottom-right corner), are added to obtain the final CTC loss. To achieve numerical stability, the computation typically done in log scale. This is the forward calculation step. The backward calculation is similar and we omit it due to space limit. Interested readers can refer to (Graves al. 2006) for details.
2.2 THE PROPOSED W-CTC ALGORITHM
The label could be corrupted, e.g.. the speech utterance still corresponds "hello", but the anno- tated label is only "ell". The original CTC does not work here because it assumes full alignment between the speech and the transcription (both have to be complete). Inspired by (Sakurai et al.) 2007) and (Cai et al. 2019), we modify the original CTC loss computation procedure as follows:
Not only inserting a "blank" symbol between characters, we also prepend the augmented transcrip- tion with wild-card symbol "*". For example, "hello" converted to "s-h-e-l-l-o-". The wild- card symbol could represent any character in vocabulary including the "blank" symbol, therefore perfectly matches any speech frames. In other words, P(*[X) 1. As a result, the augmented Y now has a length 2.V + 2, and the trellis M has 2N rows.
Instead of two initial-state nodes (the green nodes), we also include the first node that corresponds to the wild-card "*", leading to a total of 3 starting nodes. This is illustrated in Figure [b The top-left 3 green nodes are valid initial states, where Mo,0 P(+ = 1.
Instead of only considering two final states M2N- MMN,T_11 we consider the entire last two rows (the bottom two rows are all colored orange Figure lb). That is, we collect set of (( = - log( (0ggMM22155 + M2N.j), for j = N - 1 to T 1, then we summarize them to obtain our W-CTC loss, i.c. LW-eTe = ofclic N - 1-15 < T}. We can safely ignore the case of j < N - 1 as those are not feasible alignments (the probability P[Y]X j)) = 0 if < N - 1), because at least N frames are needed to match N characters in the transcription. So the left nodes in the last two rows in Figure [b] are not colored orange (not ending nodes). Here the function a..) can be chosen from max, summation, or weighted sum.
The rest of the algorithm is the same as the standard CTC. The DP recursion is the same Equa- tion 3 Empirically, we found that choosing weighted sum for (.)), yields the best results:
Lw-cre- Σ wy Cerc where [WN-1..... WT-1l softmax (1-C(Nc (N-
-Cre -(fo"1) (4)
An ablation study on different strategies for ( J.J. is presented in Section 3.4 Note that since addi-
tional wild-card symbol with a constant probability is inserted for each frame, the total probability becomes unnormalized. However, empirically we found that there is clear difference whether to normalize the total probability. A detailed discussion is provided in Appendix Therefore, we use the unnormalized likelihood for simplicity.
Complexity: Since we don't t introduce additional cost except for adding a row in the trellis, the time complexity remains the same as the original CTC computation, i.e. O(TN). The DP updates
4

---

Published as a conference paper at ICLR 2022
the same. It can be speed up by computing the whole column in parallel, resulting in parallel time complexity of (TT) using ((V) cores. The space complexity is O(TN) as we need to store the graph for backpropogation.
2.3 THE KEY SUMMARY
Compared to the standard CTC, the key changes are: (1) Prepend "*" symbol on Y, resulting in an additional row on top of the trellis M (the blue nodes); (2) There are 3 green nodes to start DP recursion; (3) The orange ending nodes (final states) are the entire last two rows.
The change (prepend wild-card symbol), ensures that the transcription can start anywhere in the speech frames. Due to P(* X) 1, there will be punishment if going from Mo.j-1 to Mo.; (the probability remains the same). The 3rd change (consider last two rows), make the transcription end anywhere in the frames, rather than matching to the end of the speech. It is equivalent to implicitly adding a wild-card at the end of the transcription. As a result, the explicit front wild-card, and the implicit final wild-card, successfully handle the missing transcription in both ends.
Combining the two changes, we convert the standard CTC built only for full alignment, to a very flexible partial alignment usage, at negligible cost. The computation is still fully differentiable. Therefore, we can still obtain the gradient and learn the network parameters, even if the transcription only matches the middle part of the speech. These will be validated in the experiment section.
worth noting that, the wild-card symbol "*", NOT equivalent to the "blank" symbol. The "blank" symbol within the augmented vocabulary, which has a physical meaning of silence, back- ground noise, or an unknown character. The network still needs to predict the probability of "blank" symbol at each frame, given the speech. Usually, this probability P(-IX) is < 1. On the contrary, the wild-card "x" is not in the vocabulary. It matches any character in the vocabulary. The probability P(*[X) always 1. Only the wild-card symbol can let the alignment skip the beginning speech frames (Mo.0 + Mo.
Mo.2 - ..... .), without any punishment. Just putting the "blank" symbol in the first row won't 't be able to achieve this.
Limitations: There are a few limitations of the proposed method. (1) It only handles missing transcription (missing Y) problem, but not the case where speech frames are missing (missing X). Since one key assumption is the many-to-one mapping (v(-) is surgective), simple extension of prepending a wild-card in front of X, will not work. However, this can be addressed by recording longer audio/videos to have some redundancy at both sides, which covers all the labels. (2) The missing part in Y, needs to be at two sides, but not in the middle. Taking the "hello" example, it cannot solve the corrupted transcription "h???o", where the "?" symbol represents the missing part. But it can handle the "Tell?" case. (3) This approach requires the remaining fraction Y be continuous subsequence of Y, e.g., "?ell?" can be solved, but "?e?lo" cannot.
3 EXPERIMENTS
We conduct experiments speech, OCR and Sign Language tasks in $3.1 $3.2 respec- tively. An ablation study is discussed $3.4 We implement the forward computing scheme and leverage the auto-grad mechanism to achieve backpropogation. All the codes can be found at hhttps:/github.comm/ https//ggghbb.oommm/iiDDnnncc//t /wtt////aaaaneeee22--etttt
Experimental Settings: For each dataset, randomly mask out portion of each training label. For example, r = 0.5, all the training labels will have half of their text masked out. The model expected learn from the corrupted labels. The same training procedure is used for both the proposed W-CTC as well as the baseline (standard CTC). The mask-ratio r ranges from 0 (no corruption) to 0.9 (90 % is missing). After training. we evaluate on the test set. Note that while the labels in the training set are masked, the ground-truth labels in the test set for evaluation, are not corrupted. We perform runs, and plot the averages and error bars (standard deviations).
3.1 AUTOMATIC SPEECH RECOGNITION (ASR) AND PHONEME RECOGNITION (PR)
We use the TIMIT Garofolo 1993) dataset in this experiment. The TIMIT dataset includes audios and corresponding transcriptions, at word level and phoneme level. All of the materials are in English. Word transcriptions include 26 English letters plus 10 digits, and phoneme transcriptions
5

---

Published as a conference paper at ICLR 2022
use 61 phonemes. There are 630 speakers. Audio is recorded at sampling rate of 16 kHz. The transcriptions include timestamps specifying which words and phonemes were spoken when.
Our experiments consist of two tasks: (a) Automatic Speech Recognition (ASR) and (b) Phoneme Recognition (PR). The standard word-error-ree (WER) and phoneme-error-rate (PER) are used to evaluate ASR and PR, respectively. Both WER and PER compare hypothesis sequence reference sequence, where the hypothesis is the predicted sequence, and the reference the gold sequence (the labels). The comparison counts the number of substitutions, S, deletions, D, and in- S++ where N is the length of the reference (in words). sertions, /. Therefore, WER is defined as N PER is similar to WER but based on phonemes. Note that WER and PER can exceed 100%.
The backbone model is the pretrained Wav2vec-2.0-base (Bacvski et al. 2020). We do fine-tuning until convergence. See Appendix [A.I for more details on the training process.
See Appendix D.1 (as opposed to 61).
for PER results based on reduced phoneme inventory of 39 phoneme classes
These 39 classes are used by a number of other papers, e.g., (Lopes & Perdigao 2011 Lee & Hon
1989
ASR
PR
10
W-CTC
1.0
W-CTC CTC
CTC
0.8
0.7
0.6
0.5
0.5
0.4
0.4
0.3
0.2
0.2
0.1
0.1
0.0
0.1 0.2
0.3 0.4 0.5
0.6
0.7
0.8
0.9
0.0
0.1 0.2
0.3 0.4
0.5
0.6
0.7 0.8 0.9
mask-ratior
mask-ratio
Figure 2: WER / PER mask-ratio in ASR and PR tasks, TIMIT test set.
Figure 2 shows the results on the two tasks: ASR (left panel) and PR (right panel). Both panels plot error rates as function of mask-ratio r. From the figure, we can see: (1) On both tasks, the models trained using W-CTC achieve better results than standard CTC. The blue curves almost always stay lower than the orange curves. (2) The performance of W-CTC is very stable when small portion of label is masked. For example, when r 0.5 in ASR experiment, the model trained using W-CTC can still get below 30% WER, while the model using standard CTC has 80% WER (almost not usable in practice). (3) The performance of standard CTC deteriorate nearly linearly against mask- ratio, while W-CTC only deteriorates when more than 50% label is missing. (4) When mask-ratio is higher, e.g., r > 0.7, though W-CTC is still slightly better, but as expected, both models struggle and yield more than 90% WER. (5) The only exception is that when
(label is completely clean), the model using standard CTC is slightly better than W-CTC. The reason is the wild-card symbol introduced W-CTC, brings significantly more alignment paths, therefore increases uncertainty and hurt the performance when data is absolutely clean.
3.2 OPTICAL CHARACTER RECOGNITION (OCR)
The Optical Character Recognition (OCR) task inputs image containing text, and outputs the recognized text with their locations on the image. OCR systems typically consist of two major components: text detection and text recognition. Text detection creates bounding boxes segment candidate patches that contain text, and text recognition converts the patches into strings. In the training phase, it cuts the patches (containing text) into "frames", and applies either attention or CTC to do implicit alignment with the text labels. Recent survey papers, e.g., (Chen et al. 2021 Subramani al. 2020), describe the state-of-the OCR systems.
In this paper, we focus on the text recognition block as CTC is widely adopted here. We tested the proposed W-CTC using the well-known CTC-based OCR model, CRNN (Shi et al., 2016). Our implementation is based on (Sun et al. 2020). with backbone ResNet (He et al. 2016). Testing and training materials were obtained from (Back et al. 2019). Two standard collections were used for training: (a) MJSynth (MJ. 9 million images) (Jaderberg et al.) 2014) and (b) SynthText (ST, 800k images) (Gupta et al., 2016). Each image is 32 x 100 pixels. We trained for 150k iterations, with batch size of 500 images. After training. the evaluations
performed on 7 standard bench- marks: IIIT5K 3000 (Mishra et al., 2012). SVT (Wang et al.) 2011). IC03 867 (Lucas et al. 2005), IC13 1015 (Karatzas al. 2013). IC15 2077 (Karatzas al. 2015). SVTP (Phan et al. 2013) and

---

Published as a conference paper at ICLR 2022
CUTESO (Risnumawan et al. 2014). The evaluation metric is the recognition accuracy, ACC, de- fined as the number of correct images divided by total number of images. Images count as correct and only if the predicted text matches the ground truth exactly. The higher ACC, the better results.
HITSK_ 3000
SVT
IC08 867
IC13 1015
0.8
0.8
0.6
0.4
W-ETE
WACTC
58 0.44
CIC
CTC
0.4
WCCC
CTC
0.2
0.2
0.2
0.2
0,0
0.0
0.0
0.2
0.6
0.0
0,4
0.0
0.2
0.6
0.4
0.5
massoratioo
masbralloo
maskeratio
meskratio
IC15, 2077
SVTP
CUTEBO
average
0.6
0.4
W-CTC
ACC
W-CTC
0.A
WCCTC CTC
CTC
0.2
0.2
0.0
0.0
0.2
0.8
0.0
022
0.6
3.8
0.0
02
0.6
0.6
0.8
0.0
0.2
0.4
0.8
mask rade
masi
rot/o
Figure 3: Test accuracy on 7 standard test sets, as a function r (mask ratio). The last plot is the average. The proposed W-CTC has generally better accuracy than the standard CTC.
Evaluation Accuracy: Figure shows a consistent results over different testsets. For standard CTC (red curves), when r > 0.1, the performance drops significantly. On the contrary, the model trained using W-CTC (green curves), performs very stable even until 70% of label masked. This indicates that the model can successfully learn the image embeddings, as well as the mappings characters, from partial labels using W-CTC. Similarly, when data is nearly clean (r S 0.1), there is small gap between W-CTC and standard CTC, due the uncertainty introduced by the wild-card symbol.
Image with corrupted label "caut"
Image with corrupted label "eaut"
Image with corrupted label "eaut"
beautifull
beaut
beautifully
The DP matrix
The DP matrix M
The DP matrix M
(a) Model A, standard CTC.
(b) Model B, standard CTC.
(c) Model C, W-CTC.
Figure 4: The aligmment paths (in the trellis M) from CTC computation. Model A and C are trained on corrupted labels r = 0.7: model is trained on clean data r = O. Model A and B use standard CTC; model C uses W-CTC. Model A fails to produce correct alignment. Model B has correct alignment but with confusions. Model C provides clear path though trained on corrupted labels.
To better illustrate how the partial alignment is done using the W-CTC loss, we provide sample image with the corrupted label, and demonstrate how the DP trellis M are different between using standard CTC and W-CTC Recall that in Equation M provides a soft alignment path that generates the likelihood of X(Y[X).
Alignment Path:
The sample image contains 11 characters, "beautifully". However, 70% of the label is masked and only 4 characters "eaut" are left to form the corrupted label. We feed the image and the corrupted label into three models: (1) Model A, trained using standard CTC, on the training data with labels 70% masked (r 0.7). According Figure 3 when
0.7, model A would learn nearly nothing from the data. (2) Model B, trained using standard CTC, but on the clean training data (r = 0). This model successfully learn the representations and should be able predict the characters correctly. But it will face the difficulty dealing with the partial label here, (3) Model C, trained using W-CTC, on the training data with label 70% masked (r 0.7). Figure 3 shows that the model using W-CTC

---

Published as a conference paper at ICLR 2022
can still learn from the heavily corrupted (r 0.7) labels. In Figure 4 show the trellis M for all three models. Here model A and B are using standard CTC's trellis, while model C continues to use the W-CTC's. So the augmented label of model A and B, is "-e-a-u-t-" (*-" is blank), while model C has the label prepended by a wild-card, i.e. "x-e-a-u-t-".
In Figure 4 the lighter the color, the higher probability values. The nodes with higher values, form soft alignment path that maps the input image to the corresponding label. As expected, model A (in Figure 44) does not make reasonable predictions, therefore the path does not reflect the alignments between the partial label and the input image. In contrast, since model B is trained using clean data, is able to get reasonable path aligning 'eaut' the image (shown in Figure 4b).. Model B also maps the front of the image (containing letter "b", which is not in the corrupted label), to the "blank" symbol (the 1st and 3rd rows). However, it clearly struggles in the beginning (the lighter nodes in the left), and does not have a strong confidence (the path's 's color is darker). The final model C. although trained on the corrupted data, still generates a very reasonable alignment path in Figure [4c The partial label is well aligned with the middle of the image, where the first row corresponds to the wild-card symbol that has a constant probability I. Unlike model B, there are no confusions of other paths in Figure 4c] Considering that model C is trained purely heavily corrupted data and only see 30% of training labels, the proposed W-CTC is very effective.
3.3 CONTINUOUS SIGN LANGUAGE RECOGNITION (CSLR)
Sign Language is a visual language composed of hand shape, hand movements, facial expression, mouth, head movements, etc. It the most important communication medium for people having lis- tening disabilities. In computer vision domain, researchers focus recognizing sequence of sign glosses from continuous sign language videos. Sign glosses refer to the spoken language words that represent the signs' meanings. This is called the Continuous Sign Language Recognition (CSLR)
task, which has attracted lots of efforts, e.g., (Koller et al. 2019 Niu & Mak] (020). Typically, CSLR models use CTC loss because no temporal alignment provided in the data.
In this experiment, we leverage the framework proposed by Camgoz et al. (2020). and perform the CSLR (sign gloss) task, with standard CTC and the W-CTC loss during training. The model is the encoder of Sign Language Recognition Transformer (SLRT) (Afouras et al. 2018). The dataset is PHOENIXI4T (Camgoz al., (2018). which provides both spoken language translations and gloss level annotations for weather broadcasts. Word-error-rate (WER) is used to measure the recognition quality on the test set. The results are presented in Table
Table I: WER on CSLR task. W-CTC significantly outperforms CTC when label corrupted.
mask ratio r
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
CTC
0.286
0.422
0.537
0.663
0.735
0.814
0.896
0.945
0.989
0.981
W-CTC
0.297
0.328
0.346
0.395
0.418
0.392
0.440
0.568
0.926
0.916
From Table we can see similar results as before. Only when data is absolutely clean (r = 0), standard CTC is slightly better (0.286 0.297 WER). Starting from 0.1, the WER of the standard CTC model, deteriors quickly. On the contrary, W-CTC model can still achieve good WER up to 0.6. That means, even only trained on heavily corrupted labels, the model can still learn the representation of videos very well using the W-CTC loss.
3.4 ABLATION STUDY ON CHOICE OF O (cclcc)
In this section, we conduct ablation study on how to combine the last two rows in the trellis of W- CTC computation. In other words, the choice of of o(cl)) We the weighted sum, illustrated in Equation 4 in previous experiments. The other options could be taking the maximum or summing the probabilities. We formally define these three choices as:
sum-prob (log-sum-expp: -log Exel-cli)
log P(Y
o(c = max-prob (equivalent to min, ((i) miny L - log (max, P(Y [D)])) i))
weighted-sun (Equation 4 E, wy Clic. where wy = softmax -
(5)
8

---

Published as a conference paper at ICLR 2022
Note that the 'sum-prob' means summing (over j) the probabilities of alignments between subsequence x[0 j] and label Y, ie. P( j]). And 'max-prob' means selecting the j-th final node with the maximum probability among those alignments. So 'max-prob' also equivalent to taking the minimum of CTC. We perform runs for each choice of a { and present the average results on ASR, PR and CSLR tasks.
where
Ablation-ASR
Ablation-PR
Ablation-CSL
-0- Sutt-prot
Su/t-prob
Sumt-prot
max unighted-um
max prob
srob www.rtedduum
1.0
0.8
www.gnterbeu.mm
0.8
0.8
0.6
0.4
0.4
0.-
02
0.1
0.8
0.1 0.2
0.5
0.9
0.
mator
m2sk r
maskrattoo
Figure 5: Ablation study on choices of ot)). The 'max-prob' yields the worst performance in all three experiments. The 'weighted-sum' is slightly better than 'sum-prob" in general.
From Figure 5 we can tell that taking the max (orange curves) always yields worst performance (higher than other curves). Especially in the CSLR task, when mask-ratio < 0.5, 'max' leads to word-error-ras rate much higher than The Summppob'' and 'weighted-sum are similar but 'weighted- sum' wins by small margin. For example, in ASR task, 'weighted-sum' performs better when r < 0.7; In PR task, all three curves are very close but 'sum-prob' is better only when r = 0.5; In CSLR task, 'weightes-sum' generally performs better across different Therefore, choosing weighted-summ for (.) provides gains over other options.
4 RELATED WORK
Connectionist Temporal Classification (CTC) Graves et al. 2006 was proposed as a loss function
to train end-to-end sequence learning models. It is widely used in speech recognition, e.g., (Amodei)
al. 2016 Battenberg al.) 2017); scene text recognition, e.g., (Graves et al., 2008 Shi et al.
2016; video action labeling, e.g. (Huang et al. 2016 Lin et al. 2017: Rouast & Adam 2020) sign
language translation, e.g., (Camgoz et al. 2020 Cui et al. 2011 and many other tasks.
To improve CTC, variety of modified versions of CTC are proposed from different aspects. Auto Segmentation Criterion (ASG) is proposed in (Collobert et al., 2016) to simplify the CTC and re- move the conditional independence assumption in CTC. In (Higuchi al. 2020), the authors bor- rowed the idea from masked language model, and propose the Mask-CTC for non-autoorerressive end-to-end ASR training. The work in (Lee & Watanabe (221) used CTC for intermediate layers as a regularization to the CTC training. [Talnikar et al. (2021) proposed a joint training framework by taking advantages of both the unsupervised contrastive loss and the supervised CTC loss, for ASR. Entropy regularized CTC (Liu et al.] (0088) was proposed to mitigate a problem that CTC often produces a peaky output distribution (Miao et al. 2015). more recent work. Zeyer et al.) (2021), I), provides a deeper numerical analysis on the role of the "blank" label and its resulting peaky behavior. Although CTC has not been applied to the partial-label training problem, a few research (Sakurai et al., 2007 Cai et al.; (2099) in the time series domain, have studied similar partial alignment problem. This paper is inspired by these works.
CONCLUSIONS
This paper studies the partial alignment (streaming) problem in sequence learning, where the label only matches a middle portion of the input sequence. To address this missing Y problem, we pro- pose the W-CTC loss, which improves the standard CTC loss with wild-cards. The wild-cards are prepended and also appended (implicitly done by considering the entire last two rows in trellis), to the original labels. The W-CTC efficiently computed using the proposed scheme. Comprehensive experiments on a variety of speech and vision tasks, demonstrate that the W-CTC significantly out- performs standard CTC loss, when label has mismatches. Ablation study is conducted verify our approach. Future work can implement more efficient backpropogation to speed up the computation.

---

Published as a conference paper at ICLR 2022
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visude speech recognition. IEEE transactions on pattern analysis and machine intel- ligence, 2018.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anuhhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016.
Jeonghun Back. Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee. What wrong with scene text recognition model compar- isons? dataset and model analysis. In International Conference on Computer Vision (ICCV), 2019.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. arXiv preprint arXiv:2006. 11477, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1oX.:::40990475: 2014.
Eric Battenberg, Jitong Chen, Rewon Child, Adam Coates, Yashesh Gaur Yi Li, Hairong Liu, San- jeev Satheesh, Anuroop Sriram, and Zhenyao Zhu. Exploring neural transducers for end-to-end speech recognition. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 206-213. IEEE, 2017.
Xingyu Cai, Tingyang Xu, Jinfeng Yi, Junzhou Huang, and Sanguthevar Rajasekaran. Dtwnet: dynamic timewarping network. Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 32, 2019.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7784-7793, 2018
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language trans- formers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10033-1003,, 2020.
Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, and Tianwei Wang. Text recognition in the wild: A survey. ACM Computing Surveys 548) 54(2):1-33,
2021. 2021.
Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Wav2letter: end-to-end convnet- based speech recognition system. arXiv preprint arXiv:1609.03193 2016.
Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for continu- ous sign language recognition by staged optimization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7361-7369, 2017.
John S Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pp. 369-376, 2006.
Alex Graves, Mareus Liwicki, Santiago Fernández, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber. A novel connectionist system for unconstrained handwriting recognition. IEEE transactions on pattern analysis and machine intelligence, 31(5):855-865; 2008
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In IEEE Conference on Computer Vision and Puttern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 330-645. Springer, 2016.
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask ctc: Non-autoregressive end-to-end asr with ctc and mask predict. arXiv preprint N.Kv:2005.008700. 2020.
De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal modeling for weakly supervised action labeling. European Conference on Computer Vision, pp. 137-153. Springer, 2016.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. In Workshop on Deep Learning, NIPS, 2014.
Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Iedar 2013 robust reading competition. In 2013 12th International Conference on Document Analysis and Recognition, pp. 1484-1493. IEEE, 2013.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bag- danov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shi- jian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1156-11 1 116.. IEEE, 2015.
Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. Weakly supervised learn- ing with multi-stream mn-llstm-hmms to discover sequential parallelism sign language videos. IEEE transactions on pattern analysis and machine intelligence, 42(9):2306 (19):2300-2330,, 2019.
Jaesong Lee and Shinji Watanabe. Intermediate loss regularization for etc-based speech recognition. In ICASSP 202 2021-2021 IEEE International Conference on Acousties, Speech and Signal Process- ing (ICASSP), pp. 6224 6228. IEEE, 2021.
K-F Lee and H-W Hon. Speaker-independent phone recognition using hidden markov models. IEEE Transactions Acousties, Speech, and Signal Processing, 37(11): 164111184411 1989.
Mengxi Lin, Nakamasa Inoue, and Koichi Shinoda. Ctc network with statistical language modeling for action sequence recognition in videos. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 393-401, 2017.
Hu Liu, Sheng Jin, and Changshui Zhang. Connectionist temporal classification with maximum entropy regularization. Advances in Neural Information Processing Systems, 31:831-841, 2018.
Qi Liu, Lijuan Wang, and Qiang Huo. A study on effects of implicit and explicit language model information for dblstm-cte based handwriting recognition. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 461-465. IEEE, 2015.
Carla Lopes and Fernando Perdigao. Phone recognition on the timit database. Speech Technolo- gies/Book, 1:285-302, 2011.
Simon M Lucas, Alex Panaretos, Luis Sosa, Anthony Tang, Shirley Wong, Robert Young, Kazuki Ashida, Hiroki Nagai, Masayuki Okamoto, Hiroaki Yamamoto, al. Iedar 2003 robust reading competitions: entries, results, and future directions. International Journal of Document Analysis and Recognition (IJDAR), 7(2-3):105-122. 2005.
Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 167-174. IEFE, 2015.
A. Mishra, K. Alahari, and C. V. Jawahar, Scene text recognition using higher order language priors. In BMYC, 2012.
11

---

Published as a conference paper at ICLR 2022
Zhe Niu and Brian Mak. Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition. In European Conference on Computer Vision, pp. 172-186. Springer, 2020.
Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recogniz- ing text with perspective distortion in natural scenes. In Proceedings of the IEEE International Conference on Computer Vision, pp. 569-576, 2013.
Lawrence Rabiner. tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 7722)257-286, 1989.
Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, (18):8027-8044) 8048, 2014.
Philipp V Rouast and Mare TP Adam. Single-stage intake gesture detection using loss and extended prefix beam search. arXiv preprint arXiv:2008. 02009, 2020.
Yasushi Sakurai, Christos Faloutsos, and Masashi Yamamuro. Stream monitoring under the time warping distance. In 2007 IEEE 23rd International Conference on Data Engineering, pp. 1046- 1055. IEEE, 2007.
Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(1 22222-223444 2016.
Nishant Subramani, Alexandre Matton, Malcolm Greaves, and Adrian Lam. A survey of deep learning approaches for ocr and document understanding, arXiv preprint arXiv:2011.13534 2020.
Jun Sun, Hongxiang Cai, and Yichao Xiong. vedastr: A toolbox for scene text recognition, 2020.
URL htttps://githhii www/Meoiiiia Smart/vedasts
Chaitanya Talnikar, Tatiana Likhomanenko, Ronan Collobert, and Gabriel Synnaeve. Joint masked epe and etc training for asr. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3045-3049. IEEE, 2021.
Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In 2011 Inter- national Conference on Computer Vision, pp. 1457-1464. IEEE, 2011.
Albert Zeyer, Ralf Schlüter, and Hermann Ney. Why does etc result in peaky behavior? arXiv preprint arX.v:2105. 2021.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX: EXPERIMENTS DETAILS
A.1 CODEBASE AND HYPER PARAMETERS
Here we list the literatures and their code repositories that our implementation is based on, in Table We also list the key training hyper-parameteer parameters in Table 3
Table 2: List of code bases and their references that we use in the experiments.
W-CTC implementation
NA
ASR and PR experiments
Baevski et al. (2020
OCR experiment
Sun al. 20201
http://gghhb..oommmdddd--aaarrrn
CSLR experiment
Camgoz al. (2020)
https//ggihubb.co//eeaaamm//
Table List of key training hyper-parameters.
Batch-size
Optimizer
LR
Steps
ASR
32
A
lc-4
7k (50 epochs)
PR
32
AdamW
le-4
7k (50 epochs)
OCR
500
AdaDelta
150k
CSLR
32
Adam
le-3
Stop if no better for 800 steps
A.2 SEQUENCE LENGTHS (DURATION) AND THEIR EFFECTS
In this section, we collected the statistics of the duration for ASR and PR experiments (the TIMIT dataset), as well as for CSLR experiment (the PHONEX147 14T dataset). We draw the histogram of the lengths (the number of frames, denoted as T) in Figure 6 The mean, std, max and min values are provided in Table For OCR, we fix the image size as 30x 100, so the duration (proportional to the image length, which is 100) is fixed.
204
352
(a) TIMIT training set.
(b) PHONEXI4T dataset.
Figure 6: The histogram of duration for TIMIT (for ASR and PR), and PHONEXI4T (for CSLR) dataset.
Table 4: Duration (# of frames) for TIMIT training set (in ASR and PR experiments), and the PHONEXI4T dataset (in CSLR experiment).
mean
std
min
TIMIT PHONEX14T
153.60
43.05
46
389
108.67
45.25
16
395
From Figure 6 and Table we can see the varies a lot (e.g. 16 to 395). This covers a wide range of the duration. In theory, if 7' is not normalized out, it should hurt the performance when
13

---

Published as a conference paper at ICLR 2022
the proposed W-CTC without normalization. The longer sequence get higher probability (total probability is 2"), resulting in a smaller loss.
However, empirically found normalization has little impact shown in Appendix B in presence of such wide range of duration in the dataset. The W-CTC performs constantly well. Note that in a seq2seq generative model, without normalization, the model tends to generate short sentences, which unwanted. But here we are dealing with a different problem. The higher probability essentially reduces some weight longer sequences. The model encouraged to pay more attention to the correctness of the shorter sequences. However, this does not necessarily lead to poorer performance. Shorter audios could be either clear (easy to learn) or noisy (hard learn), the same for longer sequences. Therefore, it is like bootstraping, but in random fashion. It does not bring gains, neither hurts. Empirically we also didn't observe difference whether normalize or not. So we choose the unnormalized version due to simplicity. This is not theoretically proved, and requires tests on more datasets future work.
A.3 THE MASKING PROCEDURE
Let us use an example to illustrate the masking procedure. Suppose we have a label string "hel- loworld", 10 characters in total. We want mask 50%, which means 5 characters. We only mask front and end, randomly. So we generate a random starting position, uniformly from [0, 5]. Suppose the random generator yields value 3, then keep the characters from 3 to 7, and throw the rest. In this case, the remaining label for training will be "llowo".
Masking only applies in training, the goal is let the model learn from corrupted labels. It is hard to find a public dataset that contains incomplete labels (they are mostly very clean), so we apply masking
the training phase to mimic the corrupted label scenario. The inference is just to test whether the model has successfully learned the representations, so there is no masking involved inference time.
A.4 EXPERIMENTS ON RANDOM MASK-RATIO
We conduct the ASR experiment on TIMIT dataset, using random mask-ratio. This means, we do not fix the mask-ratio r, but for each utterance, we generate random r that is uniformly distributed between 0 and 1. Apparently, E[r] = 0.5. We compare with the case that has fixed mask-ratio 0.5, as well no-masking (r 0). Word-error-rate (WER) is obtained from runs for each case. We also include standard CTC. The results (mean and std) are listed in Table
Table 5: ASR experiments on random mask-ratio
W-CTC random
CTC
WER
0.5
= 0
0.5
0
mean
0.260
0.285
0.191
0.789
0.189
std
0.013
0.005
0.007
0.183
0.005
As expected, the random has similar results as r 0.5, because the expectation is 0.5. Both not far from = (clean label) case, which verifies the effectiveness of W-CTC. W-CTC are much better than standard CTC when >
B APPENDIX: NORMALIZATION
B. NORMALIZATION EFFECTS
Note that since an additional wild-card symbol "z", with a constant probability 1, is added in each frame, the total probability accumulating T frames is no longer normalized. Specifically, Z = 2n is the normalization factor. Therefore, to make P a proper likelihood, should normalized by Z. i.e. P(Y|X)/2 Otherwise, the longer imput sequence (larger T). the higher P(Y values can get. Due to the fact Lere = - log PXY the loss on longer sequence would be smaller than the shorter ones. In other words, it punishes more shorter X, regardless of the prediction
14

---

Published as a conference paper at ICLR 2022
quality. Clearly, this is not what want. A mitigation to this, is to normalize the probability and obtain the normalized W-CTC loss:
P(Y 2
moorm -log
Lw.cre + 0.37
From Equation 6, we can see that the longer sequence (larger T), the more punishment (log, (2") 2 0.37) is added the original W-CTC loss. However, we didn't observe clear advantage perform normalization this way. In the following Figure we compare the performance unnormalized and normalized version of W-CTC. This is obtained from the average of 3 runs. very hard to observe a clear trend that the normalized version outperforms the unnormalized version. We have more Eetailed-discussee in Appendix A.2 W-CTC for simplicity.
So in the experiments, we just use the unnormalized
Normelization-AA5
Normalizationn PR
NoomalizzationCSU
1.0
0.9
lzed
0/8
0.7
0.6
0.5
64
0.3
0.1
6.1
0.0
0.2
0.2
0.3
3-4
0.5
0.6
0.7
0.8 0.M
0.0 D.1
0.2
0.3 0-4 0.5
04
0.7
0.8
0.0
0.1
02
0.5
2.4 2.5 mask
04
0.7
0.8 0.
maskraafor
mask-rador
Figure 7: Compare unnormalized loss with normalized version. No significant difference is ob- served.
B.2 THE EMISSION PROBABILITY OF WILD-CARD
In this paper, we set the probability of the wild-card to be always 1, meaning that it matches any character the labels. However, making P(*[X) 1, and Σ. (s) = (where represents any possible character in vocabulary) at the same time, leads to total probability 2 instead of (unnormalized), for each frame. One way to mitigate this issue, is to normalize over (the total number of frames), which is studied in Appendix B Here provide another solution, that making P() = p. and all the characters get a total probability Σ, P(s = 1- The obvious advantage is that get normalized probability for each frame, thus the maximum likelihood estimation could be carried out without concerns. On the other hand, the disadvantage that we have select p hyper-parameter, which requires tuning for each experiment. Meanwhile, slightly alters the meaning of the wild-card. For each frame, we assign a probability less than 1, to the wild-card. This makes the wild-card being treated like regular symbol in the vocabulary.
Nevertheless, we conducted an additional experiment to study whether this scheme works in prac- tice. We range the value p from 0.1 0.9. For each p. we train the model under different mask ratio r E 1). 1). We perform the ASR experiment here. All the other hyper- parameters remain the same.
Figure show the results ASR experiment. The blue solid curve (labeled "proposed") is the proposed W-CTC, with constant probability for the wild-card (the unnormalized version). All the other dotted lines represent different jyyer-parameet and std of word-error-rate (WER), against the mask-ratio The lower, the better. It appears that P(+ 0.8 is slightly better compared with other cases, only when > 0.6. However, the difference is subtle, and all curves very similar. The unnormalized version (proposed) performs as good as others.
p. from 0.1 0.9. We plot the mean
Considering the hyper-paramenn needs to be tuned, and the meaning of wild-card is slightly changed, there is no clear advantage using this scheme to control the wild-card's emission prob- ability. least from empirical point of view. Therefore, combined with this study and the one in Appendix B we choose the unnormalized version due to its simplicity. However, for a better understanding. more tests wider range of dataset needs to be carried out. It is worth noticing
15

---

Published as a conference paper at ICLR 2022
1.2
propused
1.0
P]1]X = 0.2
R(1 0.4
Prtpo
RIN10 0.6
WEB
0.6
RIN/0 0.8
0.2
0.0
0.0
0.2
0.4
0.6
0.8
mask-ratio
Figure 8: The ASR experiment, when P(+[X) p and P(s - The plots word-error-rate vs mask-ratio.
that, this scheme could also brings the potential of dynamically adjusting p. Instead fixed p. could be learned by the network and adjusted for each individual frames. This will again bring more complexity but can potentially boost the performance, and avoid the hyper- seper-parameter tuning.
B.3 DISCUSSION ON WILD-CARD AND WEIGHTED-SUM
The proposed W-CTC, essentially enables wild-card at both sides. However, they are done in two different ways. For the beginning, we explicitly prepend the wild-card symbol (*) before the actual label. For the ending, let the trellis ends at arbitrary node in the last row. This is adopted from the SPRING algorithm proposed in (Sakurai et al., 2007). originally for DTW. We would like explain the rationale for the different choices.
Prepending a wild-card in front, allow the path to start in the middle of the frame. However, appending a wild-card in the back, is slightly different. Recall that each node in the trellis represents the probability of aggregated paths arriving at that node. If the the trellis' last row corresponds the wild-card, whenever a node in the last row is reached, will carry its probability to the next node (its neighbor in the right) without penalty. In the final node (the right-most node in the last row), it essentially sums all the possible iading-positions probability. This is conceptually equivalent to the "sum-prob" case in Equation (5). However, may want to choose the best possible ending point, maximize its probability. Therefore, using the wild-card loses the flexibility of hard-max soft-max, opposed to the weighted-sum scheme. Another issue comes with the unnormalized probability. If the ending wild-card node propogate to the next node (the neighbor in the right), the total probability increases by 1 for each step towards the right-most node. The paths probability is typically a small number, because there are a huge number of paths. In this unnormalized version, the paths' probability is overridden by the wild-cards' probability, resulting in a failure of training. We conducted experiments and observed that when having wide- cards both sides, the training does not converge. To mitigate this, using normalized version could be a solution, but brings more complexity as described in the paper. It is much simpler to use a weight-sum scheme in the end rather than the wild-card.
Why wild-card only in the beginning:
Why weighted-sum: Ideally we would like to choose the ending node with the maximum aggre- gated path probability. However, empirically we found that taking the hard maximum, leading to convergence issues. Therefore, a relaxed version, the soft-max (the weighted-sum) is adopted here. Our ablation study Section 3.4 also verifies this choice. Intuitively speaking, the hard choice (take the max one), will ignore the frames beyond that ending point. For example, if we have 10 frames, and the path ending at the 8th frame, got the maximum probability. Once we use max- sox-, will completely ignore the 9th and 10th frame, where the gradient will not back- propogate to these frames. The soft version (weighted-sumg mitigate this problem to some extent. A theoretical justification due to future work.
16

---

Published as a conference paper at ICLR 2022
Why weighted-sum only at the end: The weighted-sun scheme is not straight- forwardd to be ap- plied in front. This is to select the best path. There is no path in the beginning phase. we want to choose (a hard selection) starting position, still needs wait until the very end to get metric, e.g. the path's 's probability, in order to make the selection. Meanwhile, hard selection also brings con- vergence issues. A weighted combination of starting point has the same issue that it has wait until the path ends. In contrast, a wild-card is both conceptually simpler and easier for implementation, handle the front part.
C APPENDIX: CONVERGENCE BEHAVIOR
C.I TRAINING CURVES COMPARISON
Since W-CTC expand the number of possible alignment paths, it is worth investigating the con- vergence behavior of W-CTC, compared with the standard CTC. In this subsection, we draw the training curves (evaluation word-error-rate vs training steps) to show that, the proposed W-CTC converges as good as standard CTC, but achieves much better results when label is incomplete. The ASR and PR experiments are performed, The batch size and the learning rate are the same for both W-CTC and CTC. Both are running 7000 steps (50 epochs).
033
53
02
2002
4003
60:00
1102
1030
230
8800
1:000
1002
(a) Standard CTC
(b) W-CTC
Figure 9; ASR experiments. WER vs training steps.
2.8
20
2.8
66-
0.6
66
E4
02
1O06
2000
4000
2008
7003
1200
3000
40.00
5.000
7060
11823
1286
(a) Standard CTC
(b) W-CTC
Figure 10: PR experiments. WER vs training steps.
Figure g shows the convergence behavior for the ASR experiment, while Figure for the experiment. Each curve represents training dynamics for particular mask-ratio Clearly, in both experiments, whenever label is not full (r > 0), W-CTC not only yields a better final WER, but also converges faster than standard CTC. For example, when 0.6. in F the WER starts decreasing after around 1500 steps (the pink curve). However, in Figure 9 (b). it starts decreasing at around 500 steps, which is nearly 1000 steps earlier than the standard CTC. When label is clean (the blue curve), both standard CTC and W-CTC converges almost identically. The PR task is harder, but the same trends hold when comparing Figure 10 (a) and (b). Therefore, can conclude that,
17

---

Published as a conference paper at ICLR 2022
W-CTC converges good as standard CTC, when label is clean, and becomes much better when label incomplete.
C.2 TRAINING LOSS COMPARISON
unnormal mm)
400
100
Figure 11: The training loss comparison between the unnormalized version and the normalized version.
Since Appendix B and Appendix A.2 suggests, the normalized W-CTC loss will be the original pro- posed W-CTC loss, plus a 0.33 term, where 7 is the number of input frames. To make a comparison with the model trained using the normalized version (+0.37)), with the unnormalized version, be- sides the performance comparison done in Appendix B we also plot the training loss here to verify the additional term 0.37 will have significant impact over the training. We pick the case mask- ratio r = 0.1 and 0.5, the PR task (using TIMIT dataset).
Figure [T the solid lines (the two curves at the bottom) corresponds to the unnormalized version, where the dashed lines (the upper two curves) are normalized (+-037"). The blue curves are the case of = 0.1, where the green curves are = 0.5. We can see that in both cases, the curves between very similar except for vertical shift. The unnormalized version
nromalized and unnormalized, are higher due to the +0.3T term. Therefore, it shows that the normalization term here does not affect the training trend in general, but simply adding constant values top.
D APPENDIX: MORE PHONEME RECOGNITION EXPERIMENTS
D.1 PR ON REDUCED PHONEME SET
is common in literature, e.g., (Lopes & Perdigao 2011). Leee & Hon 1989) Baevski et al. 2020). to the reduced phoneme set maps the original 61 phonemes into 39 classes. The mapping table in (Lopes & Perdigao 201 provided below. The original phoneme "q" removed in the reduced
test the Phoneme Recognition performance. The reduced
Table 6: Mapping table from 61 phonemes to 39 phonemes to form the reduced set.
allax,ax
bi.by
ib.ix
Tel
m.mm
B.en.nx
ng.eng
sh.zh
kel, del.
08
ah
er
hb
th
m
n
ng
sh
uw
sil
We train the model using original 61 phonemes, and evaluate on the reduced 39 phonemes set, by following the mapping in Table 6 The average performance over 3 runs, is reported in Figure [2 below. The results are similar as the experiments on the full 61 phone set, which is shown in Figure 2 right panel. With standard CTC loss, the trained model's performance drops quickly as the mask-ratio increases. In contrast, the W-CTC helps the model learn from corrupted labels and yield consistent performance until 0.6.
18

---

Published as a conference paper at ICLR 2022
39 phonemes
W-CTC
0.9
CTC
0.8
0.7
0.6
0.4
0.3
0.2
0.1
mask-ratio
Figure 12: PER vs mask-ratio, on reduced phoneme set (39 phonemes) of TIMIT test set.
E PR EXPERIMENT USING CRDNN BACKBONE
In the ASR and PR experiments, we adopt the pretrained Wav2vec-2.0 model because is one of the state-of-the-aat model that yields best ASR and PR results on TIMTI dataset. One concern that, since the pretrained Wav2vec-2.0 has learned the representations during its unsupervised pretraining, the fine-tuning on TIMIT dataset may just learn a simple mapping rather than a representation of the acoustic signals. To address this concern, we conduct another experiment using a customized CRDNN (CNN + RNN + MLP) model, trained from scratch with the W-CTC loss, to demonstrate that our proposed W-CTC is model agnostic and effective in the learning of representations.
The model and codebase are from SpeechBrain | We use the same syper-parameters as shown in the repository's config file, except setting the learning rate 0.1, the patience to 10, and train 100 epochs. The reason is that making the learning rate to 1, sometimes cause unstable training and yields NaN loss. To speed up the training with 1r=0.1, in the scheduler, we increase patience to 10 such that the learning rate can stay constant for more epochs.
WCTTC
PER
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
mask-ratior
Figure 13: The PR experiment 39 phonemes in TIMIT dataset, using the CRDNN model.
Figure 13 shows the results obtained from standard CTC and proposed W-CTC training, under different mask-ratio r. Note that the reported standard CTC training results on the website (when r is 148, where we got 0. 150 (the left most dot on orange curve). The curves are very similar to the Wav2vec-2.0's. As r increases, the standard CTC loss's performance drops almost linearly. In contrast, the W-CTC can stay constant up to r 0.6. Overall, the W-CTC outperforms the standard CTC by large margin. This is consistent with the Wav2vec-2.0's results, as well as the other two
19

---

Published as a conference paper at ICLR 2022
experiments (the OCR and CSLR models are also trained from scratch). Therefore, conclude that W-CTC is able to enable representation learning, rather than a simple mapping.
F APPENDIX: MORE OCR EXAMPLES
This subsection provides more examples to illustrate the difference in the trellis, among W-CTC and the standard CTC. Following Section 3.2, we use the images from the OCR example. We randomly select a few images with long labels (length 10) for a better illustration. We still have three models. The first one is the standard CTC, trained on corrupted labels where mask-ratio r 0.7. We plot them in the left column. The second model is the standard CTC, but trained on clean labels. They are expected to learn the representations, but will face the difficulty in the corrupted label cases. We plot them in the middle column. The last model is the W-CTC, trained on corrupted labels r = 0.7. We plot them in the right column.
Image with corrupted label "espa"
Image with corrupted label "espa"
Image with corrupted label "espo"
Despoilment matrix
Despoilment matrix
Despoilment
The DP matrix
Image with corrupted label 'eapp"
Image with corrupted labe! 'eapp"
Image corrupted label "eapp"
reappraising
reappraising
reappraising
The DP matrix M
The DP matrix M
The DP matrix M
Figure 14: Left: Standard CTC trained on mask-ratio r = 0.7; Middle: Standard CTC trained on clean label; Right: W-CTC trained on 0.7. Here the standard CTC only make reasonable inference if it is trained on the clean label, where W-CTC generally output correct paths.
We group the images into 3 categories to show different aspects of the standard CTC W-CTC:
In Figure 14| we show two examples that have very clean images. We have the same results as shown in Section 3.2. The standard CTC trained incomplete labels, has no clue how the alignment is done in these examples. The standard CTC trained on clean labels, on the contrary, can figure out the correct path, although with very low confidence. The W-CTC can obtain the strong paths that correctly reflect the alignments, even though it is trained on heavy corrupted labels.
In Figure [5] the images becomes more complicated (has curves or less clear). The missing part in the label is pushed back towards the beginning of the labels. Now, even trained on the clean label, the standard CTC is no longer able to identify the correct paths (the middle column). However, the W-CTC is hardly affected, and still generate the main path that corresponds to the alignments. There are some tiny confusion paths, due to repeated characters in the labels.
we show the images with high confusions. The remaining labels has overlap to more than one places in the figure. For example, the "st" appears twice "mistrusting". These lead to second path with high confidence, for the W-CTC (the right column). However, before those confusion paths reach their ends, they got cut off due to mismatches. Therefore, the W-CTC can still figure out the most likely path that often corresponds to the true alignment. The standard CTC does not generate any reasonable paths these examples, no matter it's trained on incomplete labels (left column), or clean labels (middle column).
In Figure 16
20

---

Published as a conference paper at ICLR 2022
Image corrppted label Aficionados
Image comuuttd label "icio"
Image with corrupted label "icio"
Aficionados
Aficionados
The DP matrix
Image with carrupted label "rack"
Image with carrupted label "rack"
Image corrupted label "rack"
f firecrackers
f irecrackers
i irecrackers
The DP matrix M
The DP matrix M
The DP matrix
with corrupted label "edin"
Image with corrupted label "edin"
Image with corrupted label "edin"
Proceedings
Proceedings
Proceedings
The matrix
Figure 15: Left, middle and right are the same above. Here the standard CTC trained on clean label, cannot even make correct paths when facing such corrupted label. W-CTC is not affected.
Image comuppte
Image corrupted
Image with comupted label "stru"
mistrusting
mistrusting
mistrusting
The DP matrix M
The DP matrix
The DP matrix M
Image with corrupted label "ococ"
Image with corrupted label "ococ"
Image carrupted
Streptococcal
Streptococcal
Streptococcal
The DP matrix M
The DP matrix M
The DP matrix
Figure 16: Left, middle and right are the same as above. These are confusing examples. For example, the "st" appears twice in "mistrusting". the "oc" appears twice in "streptococcal" We can see clean paths starting from the incorrect starting positions, in the W-CTC figures (right plots). But those incorrect paths end before reaching the end. Standard CTC cannot handle them at all.
21

---

Published as a conference paper at ICLR 2022
In conclusion, labels. Due to the wild-cards, can match part of the input sequences (here are the images), and generate reasonable alignment paths. The standard CTC, even trained on clean labels, can only output reasonable path, the remained labels corresponds to the beginning of the sequence. the front part is missing, the standard CTC completely fails. These examples verify the effectiveness of the proposed W-CTC method.
clear advantage of the proposed W-CTC, when dealing with incomplete
22


Reviews:
[{"review_id": "rbKgMFBu1y9", "reviewer": "Reviewer_K22X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper describes the algorithm in a very clean way. It is very easy to understand the algorithm.\n\n2. The proposed method is evaluated on three different domains. This helps to support the claim of the paper better.\n\nWeakness:\n\n1. The experiments are conducted only with faked masked data. Although it is valuable to have a paper discussing this topic, it is not very clear the actual benefit in real scenarios. Given that the technique itself it not very novel, this might be important to claim the novelty.\n\n2. It uses a wild-card for the beginning while it uses unconstrained endpoints for the ending. There is no explanation about the choice. It would be great why the combination is chosen.\n\n3. The proposed method is slightly worse than the standard CTC at r=0.0. That means that it would not be a drop-in replacement of the standard CTC. It would be better if it was shown that the proposed algorithm was actually useful for some real application.", "summary_of_the_paper": "This paper proposes a method to utilize partially labeled data for the CTC loss. In order to handle untrascribed lables, it introduces a wild-card for the beginning of the sequence and uses unconstrained endpoints for the ending of the sequence. It also discusses how to summarize the paths at the ending. The proposed algorithm was evaluated with simulated masked data and it was shown that the proposed algorithm can handle incomplete labels more effectively than the naive CTC loss while the naive CTC loss is slightly better when the label is complete.\n\nDynamic time warping (or DP matching) with unconstrained endpoints itself is not a new idea and a classical topic for speech recognition (e.g. word spotting). The contribution of the paper is the formal introduction of the approach to CTC and to give experimental results to confirm the effectiveness.", "summary_of_the_review": "The paper is clean and it is easy to understand the content. The experiments are fair and the effectiveness of the approach is confirmed using simulated data. It is great that the results are given for three different domains. There are certain weak points in the paper, but the strengths slightly outweigh the weaknesses. I would recommend a weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gt_InYpcRPT", "reviewer": "Reviewer_pcpu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**\n- Authors propose simple and efficient modification in CTC algorithm to support partial labels (continuous)\n- Method is proved empirically to be applicable across different domains/tasks (OCR, ASR, CSLR) for a large percentage of label corruption with significance testing.\n\n**Weak points**\n- mostly the idea is based on SPRING (dynamic time warping, DTW) which is mentioned in the paper and applied for CTC instead of DTW.\n- Wav2vec is learning actually representation of phonemes/tokens and later fine-tuning is more about mapping each representation cluster to the proper label. That is why it could be that W-CTC could be enough to simply learn this mapping but not the necessary representations. \n- (Limitation) Algorithm doesn't work with label corruption in the middle or at random places, which is the more often case during human labeling process.\n\nThe paper is very well written and presented: all necessary intro information is given, there are nice figures to present comparison between CTC and W-CTC are given. Overall, it is very simple to read and follow the whole text and statements. I have only minor comments:\n- I suggest to mention also ASG loss https://arxiv.org/pdf/1609.03193.pdf as one of the directions on CTC modification \n- typo: \"The average performance over 3 runs, are reported\" -> \"The average performance over 3 runs, is reported\"\n- Could authors provide more details on how the masking is done: random val for the begging and (r - val) for the end? \n- To resolve concern pointed above on wav2vec: Could authors perform additional experiments (even without statistical significance) for ASR where model is trained from scratch and not pretrained wav2vec to demonstrate that W-CTC is able to learn proper representations too (i believe it will as it is doing well in OCR and CSLR tasks)?\n- Why in Eq. (7) it is $-0.3T$ and not $+0.3T$?\n- Does normalization from Eq. (6) not converge with all ways in Eq. (5)?\n- I don't get why in Eq. (6) there is $2^j$ and not $2^T$ as we assume that at the end we are staying in the wild-card state. Then actually Eq. (7) is the explicit form for normalized W-CTC.\n- With respect to results on the normalization: what is the average and std for the T in different tasks? Could it be that variation is very small so that normalization doesn't influence?\n", "summary_of_the_paper": "Mapping between two sequences of different length is frequently done via Connectionist Temporal Classification (CTC) loss when alignment is not available. CTC performs the full alignment between input and label sequences. However, there are applications when label is incomplete and only part of it is given. In the current paper authors consider the case when begin and end of label sequence are missed and extend CTC applicability to these partial labels. Based on the dynamic time warping SPRING algorithm for incomplete labels authors propose W-CTC: they similarly prepend and append label with \"*\" token which model missed begin and end of label sequence; then they aggregate all valid paths with the latter modification. W-CTC is empirically proved to significantly improve performance over CTC even if up to 40-70% label sequence is missed (overall performance similar to the complete label case) across different tasks, like speech, optical character, and continuous sign language recognition. W-CTC is simple and efficient.", "summary_of_the_review": "The paper is very well written with clear explanation of proposed method, well covered experimental results across domains and tasks. Proposed idea is mainly based on existing DTW algorithm (SPRING), however it is extended and applied to CTC demonstrating significantly improved results on incomplete labels. Thus, I would recommend this paper to be accepted. It could lead to a broader discussion on methods with incomplete labels and data - more real case scenario of data collection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RCEAHlXVvn_", "reviewer": "Reviewer_sfbX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n- The missing label issues often happen in the read training data. \n- I often faced this kind of issue when we try to use long recordings (e.g., youtube, ted corpus, or podcast data). In this case, the segmentation time stamp is not accurate and the missing labels or missing audios in the edge often happen. We usually use some force-alignment techniques to realign such data, but this method can be applied to such data directly. (Note that the missing audio issues can be solved by just using longer segmentation that would be supposed to cover the contents corresponding to the transcriptions).\n- CTC is now widely used as an alternative seq2seq model (originally only ASR/OCR, but now NMT and speech translation). This method would have a broad impact on the machine learning community.\n- The experimental effectiveness is valid. It shows robust learning behaviors. Also, it was shown in three different tasks.\n\nweaknesses\n- The experiments are not real. I recommend the authors try some long recording setup as I mentioned before.\n", "summary_of_the_paper": "This paper proposes an extension of CTC by considering the wild-card to adjust the label missing issues during training, which tends to happen in the onset/offset edges of the utterance. The paper elegantly formulates it as an extension of the CTC framework and applies it to two tasks (ASR and OCR). The method shows robust training behaviors compared with the original CTC training.\n\nOther comments:\n- Can we use it for RNN transducer?\n- I'm expecting that this method may work even if missing labels happen in the middle if we use the self-attention-based network. The self-attention network can handle some re-ordering (move the hidden vectors corresponding missing part to the edge).\n- Section 1.1, the definition parts. These parts are well written, but the discussion about <blank> is missing.\n- I mentioned below, but this problem can be applied to the missing audio (missing X) case in the long recording scenario by just using a longer segment to cover all labels. So, for my major applications of this method, I don't think that this becomes a limitation.\n- does the wild-card symbol contain the <blank> symbol? It may not be a matter, but I'm curious.\n- It would be better if the paper has more discussions of why 'max-prob' does not work and 'weighted-sum\" works the best.\n- Is it possible to correctly find the wild card region with the Viterbi algorithm? I think this is a very good option. I also want to verify that the wild card region is correctly identified (I think Figure 4 shows it to some extend, but I want to know more analysis/examples).\n\n\n", "summary_of_the_review": "This paper would have a broad impact on the machine learning community to tackle noisy data training based on CTC. The formulation is straightforward and the experimental effectiveness is valid. The paper does not show the experimental impact of this method on the real problem, and this addition makes the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BJwKRd8PFoi", "reviewer": "Reviewer_VmYt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem addressed by this paper is interesting and relevant. The paper is well written overall, easy to understand with clear definitions and examples. The equations look ok and the paper explains well the complexity and how to improve the loss in the presence of partial transcripts. \n\nThe \"Key summary\" section is quite helpful to summarize and clarify the contributions and helps a lot to understand the whole paper. In particular, it clarifies some parts that may not be so clear for the reader in the abstract (e.g. \"partial\" in this work only refers to missing labels at the beginning or end, or that the proposed \"wildcards\" are only used for the beginning part). \n\nThe experiments on three tasks look sound and good, the method is well detailed. The ablation study on the weighting of the losses for different sub-segments is interesting.\n\nThe general definitions about alignments in the introduction and the different mentions of attention-based methods in the paper do not bring much to the paper and could be at least reduced. Attention-based methods are not addressed at all by the paper anyway. Reducing these parts would leave some room for further analysis in the experiments section.\n\nRegarding the mask ratio in the experiments:\n  - it is not clear how the masking is done for training, whether the mask is evenly distributed or not between beginning and end\n  - in the figures it is not clear whether the mask ratio is the one used for training, if there is masking at inference too, and in that case, whether the mask ratio is the same for training and inference.\n  - in general it would be helpful to see the results when the mask ratio is random and a comparison with CTC trained on full vs masked vs partial transcript corresponding to knowing the mask\n\nRegarding the wildcards, why not either use wildcards at the beginning AND at the end, or apply the same trick for the beginning as the one applied at the end (i.e. allow the alignment to begin anywhere). As the authors state themselves, the fact that $p(\\*|X)$ is always $1$ does not allow to formulate the whole loss as a probability. A \"correct\" formulation of the problem from a mathematical standpoint would have been much more interesting. For example, how about $p(\\*|X) = r$ and use $(1 - p(\\*|X))$ to \"enter\" the main graph? How about doing something similar for the end?  \n\nIt is not completely clear either whether the wild-cards are used at inference as well or not. \n\nThe rationale behind weithing scheme is not completely clear. The summarized probabilities correspond to products of probabilities with a different number of factors: how to make sure that shorter segments do not have a lower loss just because less probabilities are multiplied. It would make sense to see the duration of the segment appear somewhere. Moreover, one would expect to treat the missing beginnings as the missing ends in the weighting scheme. As mentioned earlier, $\\mathcal{L}^{(j)}_{CTC}$ does not correspond to a log-probability.\n\nThe CTC loss also sometimes need time to start converging, since the underlying network has to learn to align. The proposed method probably makes this issue bigger, and it would have been nice to at least see convergence curves, or an analysis of the training dynamics.\n\nFinally, in the semi-supervised part about helping to generate actual labels: it is not exactly clear what or how samples would be discarded. Moreover, the proposed method could be used to simply do a forced alignment of the training set with a pretrained model to \"clean\" the dataset. One would expect to see i the paper the advantages of the proposed integrated training method vs this simple approach.\n\nEdit after rebuttal:\nThe authors addressed the doubts and questions raised in the review.", "summary_of_the_paper": "This paper presents a modification of the CTC training loss to cope with incomplete transcripts in the training set, where the actual transcription of the beginning or of the end is missing. The authors propose to minimize the loss over all possible sub-segments of the input to automatically align the one that matches the available transcript. The main contribution is that it allows to directly train the neural network with CTC without having to clean or align the data first when some transcript are only partial.", "summary_of_the_review": "The problem is interesting and the paper is clear and easy to understand. However, the contribution of how to use dynamic programming to compute the CTC loss over all sub-segments is marginal. The paper could benefit from a deeper analysis and a more mathematically sound definition.\n\nEdit:\nThe additional experiments and details are helpful and indeed benefit the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension of CTC by considering the wild-card to adjust the label missing issues during training. The authors propose to minimize the loss over all possible sub-segments of the input to automatically align the one that matches the available transcript. It is empirically proved to significantly improve performance over CTC even if up to 40-70% label sequence is missed (overall performance similar to the complete label case) across different tasks. 

As agreed by the reviewers, the paper is well presented and the problem is interesting to a broad community. Dynamic time warping with unconstrained endpoints itself is not a new idea and a classical topic for speech recognition (e.g. word spotting). The contribution of the paper is the formal introduction of the approach to CTC and to give experimental results to confirm the effectiveness. Also the use of simulated data weakens the paper a bit.

The decision is mainly based on the clear presentation and fair experimental justification.
Decision: Accept (Poster)