W-CTC: a Connectionist Temporal Classification Loss with Wild Cards

1 INTRODUCTION . Connectionist Temporal Classification ( CTC ) was proposed in ( Graves et al. , 2006 ) to train endto-end sequence learning models . It has become the most popular loss function in many sequence learning applications , such as Automatic Speech Recognition ( ASR ) ( Amodei et al. , 2016 ; Battenberg et al. , 2017 ) and Optical Character Recognition ( OCR ) ( Liu et al. , 2015 ; Subramani et al. , 2020 ; Chen et al. , 2021 ) . CTC has a number of attractive properties . For example , it comes with a clear interpretation that maximizes the posterior marginal probability summed over all possible alignment paths . In addition , CTC loss can be easily incorporated into neural nets because it is fully differentiable . The gradients can be estimated efficiently by borrowing the well-known forward-backward algorithm from Hidden Markov Models ( HMMs ) ( Rabiner , 1989 ) . One limitation of standard CTC is that it assumes the label matches the whole input sequence . CTC performs a full alignment between them . However , we often face the partial alignment in practice . For example , a speech transcription often omits the prefacing introduction of the speaker ; a beginning of an audio book that tells who read the book , along with various other details , are typically not in the written version ; a sign language interpreter often ignore the irrelevant gestures in the beginning and ending stages ; etc . The above motivating examples are also referred as the streaming problem ( Sakurai et al. , 2007 ) . In a streaming speech or video , we only capture a fraction of interested segments to form the labels . It is also common that the label is corrupted or noisy due to variable recording conditions . Partial alignment is useful in semi-supervised or transfer learning as well , which enables us to use a coarse model ( from a different domain ) to automatically generate labels . We can discard the low-confident outputs and form the incomplete , but more accurate labels , then feed them into the target model to train . In time series domain , streaming Dynamic Time Warping ( s-DTW ) was proposed ( Sakurai et al. , 2007 ) for partial warping , and Cai et al . ( 2019 ) extended it to differentiable training in neural nets . Inspired by their work , we propose the wild-card enhanced CTC , dubbed as W-CTC , to perform the end-to-end partial-alignment sequence learning . To the best of the authors ’ knowledge , this is the first study to apply CTC on incomplete labels . 1.1 PROBLEM FORMULATION . Let X = [ X0 , X1 , . . . , XT−1 ] be the input sequence of length T , e.g. , speech audio frames for an ASR system , or image segments for an OCR system . Let Y = [ Y0 , Y1 , . . . , YN−1 ] be the corresponding targets ( labels ) of length N , e.g. , the transcription for ASR , or the recognized string for OCR . Assume T ≥ N . Though both X and Y are provided , we don ’ t know how they are aligned . Let the alignment function be ψ : X → Y , which maps a frame x ∈ X to a target y ∈ Y . Define : Definition 1 ( Subsequence ) . A subsequence of X , denoted as X̂ ⊂ X , is a vector of multiple consecutive elements in X , i.e . X̂ = [ Xi , Xi+1 , . . . , Xi+τ ] , where τ > 0 . Definition 2 ( Full alignment ) . A full alignment maps the entire X to the entire Y , i.e . y = ψ ( x ) ∈ Y , for any x ∈ X , as well as x = ψ−1 ( y ) ∈ X , for any y ∈ Y . Definition 3 ( Alignment path ) . An alignment path , π , is a vector of the same length as X , i.e . π = [ y0 , y1 , . . . , yT−1 ] , where each element yi = ψ ( Xi ) ∈ Y , for 0 ≤ i < T . Definition 4 ( Valid alignment ) . An alignment path is valid , if it satisfies : • Monotonicity : if Xi maps to Yj , Xi+t can not map to Yj−k , for any t > 0 and k > 0 . • Surjection ( many-to-one mapping ) : ψ ( · ) is a surjective function , such that a subsequence X̂ ⊂ X , could maps to a single Yj ∈ Y . But each Xi can have only one label y ∈ Y . Definition 4 is a common assumption , because we often have a higher sampling rate in audios or images , for a better resolution . In consequence , T ≥ N , thereafter multiple frames correspond to a single target . In practice , X could be the sequence of frame embedding vectors , i.e . Xi ∈ Rd , obtained by a neural network . Provided the targets Y , we want to train the neural network by implicitly learning the alignment between X and Y . This is a full alignment problem and often handled by CTC loss ( Graves et al. , 2006 ) or attention mechanism ( Bahdanau et al. , 2014 ) . In this paper , we extend the full alignment problem , to a more general partial alignment case , where the input could be incomplete . That is , the input sequence X , does not necessarily have an end-toend mapping to the label Y ( violate Definition 2 ) . In particular , we solve the missing Y problem , whereX is complete . For example , the annotated transcription is incomplete , or corrupted , such that it does not correspond to the full audio speech . It is sometimes referred as the streaming alignment problem , as X can be seen as a long stream signal , while Y only matches part of X . Definition 5 ( Partial Alignment Problem ) . A problem where the label is a subsequence Ŷ = [ Yk , Yk+1 , . . . , Yk+L−1 ] ⊂ Y , that has a length L < N . Both k and L are unknown . Note that the missing portion are at the beginning and the end of Y . A commonly seen example in ASR system , is the silence/noise at the two sides of X . The transcription often does not include such silence/noise remarks , resulting in a mismatch at the two ends . This is already handled by CTC loss with an additional “ blank ” label ( refer to ( Graves et al. , 2006 ) ) . We tackle the more complex problem , that the missing part is meaningful rather than just noise/silence . 1.2 BRIEF INTRODUCTION OF CTC . To introduce CTC , let ’ s first consider the full alignment problem . There are many paths leading to the same label Y from input X . Let Z be the set of all valid paths . The standard CTC loss ( Graves et al. , 2006 ) is defined as the negative log-likelihood of the labels given input sequences , i.e . LCTC ( X , Y ) = − logP ( Y |X ) = − log ∑ π∈Z Pπ ( Y |X ) ( 1 ) where Pπ ( Y |X ) is the probability of a particular alignment path π . For any path , CTC assumes conditional independence amongst Xi ’ s , 0 ≤ i < T . So the probability of that path , is approximated by a product of the path-determined label ’ s probability at each frame , i.e . Pπ ( Y |X ) ≈∏ 0≤t < T Pt ( π [ t ] | X ) . Minimizing the CTC loss is equivalent to maximizing the likelihood of the observed label given the input sequence . CTC is fully differentiable , making it favorable to train an ASR or OCR system , where no exact temporal alignment information is provided . CTC is calculated using Dynamic Programming ( DP ) , which will be further analyzed in Section 2.1 . Our target in this paper is the partial alignment problem , which could reduce to the full alignment problem , by trying to do full alignment between the corrupted targets , Ŷ , and all possible subsequences of X . Let X̂i , τ = [ Xi , Xi+1 , . . . , Xi+τ ] be one subsequence , starting from i-th frame with a length of τ + 1 . The straight-forward way is to loop over all possible ( i , τ ) pairs , obtain the CTC loss for each subsequence , and summarize them , i.e. , L′CTC ( X , Ŷ ) = σ { LCTC ( X̂i , τ , Ŷ ) | i+ τ < T } ( 2 ) where σ { · } means a summary of the set , e.g. , summation or max . This straight-forward approach is too expensive to compute : For each LCTC ( X̂i , τ , Ŷ ) , a DP is utilized , leading to O ( TN ) cost . Unfortunately , DP can not be completely parallelized . Looping through all ( i , τ ) pairs yields a total complexity of O ( T 3N ) . A trick is that when i is fixed , we can reuse the DP trellis from LCTC ( X̂i , τ , Ŷ ) to obtain LCTC ( X̂i , τ+1 , Ŷ ) , at a cost of O ( N ) instead of O ( TN ) . Even with this trick , O ( T 2N ) is still required as looping over i can not be avoided . Considering T can be fairly large for audio frames , this approach is hardly deployable in practice . 2 PROPOSED WILD-CARD CTC . 2.1 REVIEW OF CTC COMPUTATION . Before presenting W-CTC , we briefly review in detail the Dynamic Programming ( DP ) based forward algorithm to compute the CTC loss in Equation 1 . Taking an English ASR system as an example , let X be a sequence of embedding vectors of T speech frames , and Y be the corresponding transcription of length N . Since there could be silence frames in X , which does not correspond to any English letter , the first step is to augment Y by inserting a “ blank ” label that could represent the silence . “ blank ” is inserted between any two characters , as well as the beginning and the end of the transcription . For instance , a transcription of “ hello ” will be augmented to “ -h-e-l-l-o- ” , where “ - ” represents the “ blank ” label . The augmented transcription Ỹ has a length 2N + 1 . The second step is to build the trellis ( DP matrix ) M , which has 2N + 1 rows and T columns , as illustrated in Figure 1a . Each node , Mi , j , denotes a subproblem ’ s likelihood , i.e . Mi , j = P ( Ỹ [ 0 : i ] | X [ 0 : j ] ) . Since the speech signal could end with the transcription ’ s last English letter , or with additional silence , the final CTC loss is a combination of these two cases . More specifically , it is the negative log of the sum of the two final states , i.e . LCTC ( X , Ỹ ) = − log ( M2N−1 , T−1 +M2N , T−1 ) . These two final-state nodes are colored orange at the right-bottom corner of Figure 1a . To fill the trellis , we start from the top-left corner . The value of M0,0 is the probability of “ blank ” given the first frame , i.e . M0,0 = P ( -|X0 ) . Similarly , M1,0 = P ( Y0|X0 ) is the probability of transcription ’ s first letter . The speech could begin with either silence or the transcription ’ s first English letter , so both M0,0 and M1,0 are valid initial states ( the green nodes at top-left corner in Figure 1a ) . Note that the probability of a predicted symbol , i.e . P ( Ỹi|X0 : j ) , is often obtained by projecting the embedding vectorXj to a logit vector ( length is the augmented vocabulary size ) , then applying softmax operation . As the prediction at any time step can be conditioned on the entire sequence of X ( e.g. , using a bi-directional LSTM model ) , we write P ( Ỹi|X ) instead of P ( Ỹi|X0 : j ) to account for the more general cases . Referring to the illustration in fig . 1a , the rules for updating the remaining values Mi , j are as follows : Mi , j = { P ( Ỹi|X ) · ( Mi , j−1 +Mi−1 , j−1 ) , if Ỹi = “ - ” or Ỹi = Ỹi−2 P ( Ỹi|X ) · ( Mi , j−1 +Mi−1 , j−1 +Mi−2 , j−1 ) , otherwise ( 3 ) The line 1 in Equation 3 does not have the term Mi−2 , j−1 , which means : ( 1 ) If Ỹi is “ blank ” , the path can not come from a skip “ blank ” before ( Mi−2 ) . ( 2 ) If two consecutive letters are the same , e.g. , “ -l-l- ” in “ -h-e-l-l-o- ” , the node corresponding to the second “ l ” can not directly have a path from the node corresponding to the first “ l ” . In this case , the “ blank ” symbol between two “ l ” s plays a separator role , enabling the decoder to know there are two “ l ” s instead of one . As a result , the red arrows in Figure 1a are forbidden . In other cases , the node Mi , j could have a path from Mi−2 , j , so the blue arrows are allowed in Figure 1a . Following above DP recursion rules , the trellis M is filled and the values of the two ending nodes ( orange nodes at bottom-right corner ) , are added to obtain the final CTC loss . To achieve numerical stability , the computation is typically done in log scale . This is the forward calculation step . The backward calculation is similar and we omit it due to space limit . Interested readers can refer to ( Graves et al. , 2006 ) for details .

Summary: This paper proposes an extension of CTC by considering the wild-card to adjust the label missing issues during training, which tends to happen in the onset/offset edges of the utterance. The paper elegantly formulates it as an extension of the CTC framework and applies it to two tasks (ASR and OCR). The method shows robust training behaviors compared with the original CTC training.  Other comments: - Can we use it for RNN transducer? - I'm expecting that this method may work even if missing labels happen in the middle if we use the self-attention-based network. The self-attention network can handle some re-ordering (move the hidden vectors corresponding missing part to the edge). - Section 1.1, the definition parts. These parts are well written, but the discussion about <blank> is missing. - I mentioned below, but this problem can be applied to the missing audio (missing X) case in the long recording scenario by just using a longer segment to cover all labels. So, for my major applications of this method, I don't think that this becomes a limitation. - does the wild-card symbol contain the <blank> symbol? It may not be a matter, but I'm curious. - It would be better if the paper has more discussions of why 'max-prob' does not work and 'weighted-sum" works the best. - Is it possible to correctly find the wild card region with the Viterbi algorithm? I think this is a very good option. I also want to verify that the wild card region is correctly identified (I think Figure 4 shows it to some extend, but I want to know more analysis/examples).