Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an extension of CTC by considering the wild-card to adjust the label missing issues during training, which tends to happen in the onset/offset edges of the utterance. The paper elegantly formulates it as an extension of the CTC framework and applies it to two tasks (ASR and OCR). The method shows robust training behaviors compared with the original CTC training.

Other comments:
- Can we use it for RNN transducer?
- I'm expecting that this method may work even if missing labels happen in the middle if we use the self-attention-based network. The self-attention network can handle some re-ordering (move the hidden vectors corresponding missing part to the edge).
- Section 1.1, the definition parts. These parts are well written, but the discussion about <blank> is missing.
- I mentioned below, but this problem can be applied to the missing audio (missing X) case in the long recording scenario by just using a longer segment to cover all labels. So, for my major applications of this method, I don't think that this becomes a limitation.
- does the wild-card symbol contain the <blank> symbol? It may not be a matter, but I'm curious.
- It would be better if the paper has more discussions of why 'max-prob' does not work and 'weighted-sum" works the best.
- Is it possible to correctly find the wild card region with the Viterbi algorithm? I think this is a very good option. I also want to verify that the wild card region is correctly identified (I think Figure 4 shows it to some extend, but I want to know more analysis/examples).



main_review: strengths:
- The missing label issues often happen in the read training data. 
- I often faced this kind of issue when we try to use long recordings (e.g., youtube, ted corpus, or podcast data). In this case, the segmentation time stamp is not accurate and the missing labels or missing audios in the edge often happen. We usually use some force-alignment techniques to realign such data, but this method can be applied to such data directly. (Note that the missing audio issues can be solved by just using longer segmentation that would be supposed to cover the contents corresponding to the transcriptions).
- CTC is now widely used as an alternative seq2seq model (originally only ASR/OCR, but now NMT and speech translation). This method would have a broad impact on the machine learning community.
- The experimental effectiveness is valid. It shows robust learning behaviors. Also, it was shown in three different tasks.

weaknesses
- The experiments are not real. I recommend the authors try some long recording setup as I mentioned before.

summary_of_the_review: This paper would have a broad impact on the machine learning community to tackle noisy data training based on CTC. The formulation is straightforward and the experimental effectiveness is valid. The paper does not show the experimental impact of this method on the real problem, and this addition makes the paper stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a modification of the CTC training loss to cope with incomplete transcripts in the training set, where the actual transcription of the beginning or of the end is missing. The authors propose to minimize the loss over all possible sub-segments of the input to automatically align the one that matches the available transcript. The main contribution is that it allows to directly train the neural network with CTC without having to clean or align the data first when some transcript are only partial.
main_review: The problem addressed by this paper is interesting and relevant. The paper is well written overall, easy to understand with clear definitions and examples. The equations look ok and the paper explains well the complexity and how to improve the loss in the presence of partial transcripts. 

The "Key summary" section is quite helpful to summarize and clarify the contributions and helps a lot to understand the whole paper. In particular, it clarifies some parts that may not be so clear for the reader in the abstract (e.g. "partial" in this work only refers to missing labels at the beginning or end, or that the proposed "wildcards" are only used for the beginning part). 

The experiments on three tasks look sound and good, the method is well detailed. The ablation study on the weighting of the losses for different sub-segments is interesting.

The general definitions about alignments in the introduction and the different mentions of attention-based methods in the paper do not bring much to the paper and could be at least reduced. Attention-based methods are not addressed at all by the paper anyway. Reducing these parts would leave some room for further analysis in the experiments section.

Regarding the mask ratio in the experiments:
  - it is not clear how the masking is done for training, whether the mask is evenly distributed or not between beginning and end
  - in the figures it is not clear whether the mask ratio is the one used for training, if there is masking at inference too, and in that case, whether the mask ratio is the same for training and inference.
  - in general it would be helpful to see the results when the mask ratio is random and a comparison with CTC trained on full vs masked vs partial transcript corresponding to knowing the mask

Regarding the wildcards, why not either use wildcards at the beginning AND at the end, or apply the same trick for the beginning as the one applied at the end (i.e. allow the alignment to begin anywhere). As the authors state themselves, the fact that $p(\*|X)$ is always $1$ does not allow to formulate the whole loss as a probability. A "correct" formulation of the problem from a mathematical standpoint would have been much more interesting. For example, how about $p(\*|X) = r$ and use $(1 - p(\*|X))$ to "enter" the main graph? How about doing something similar for the end?  

It is not completely clear either whether the wild-cards are used at inference as well or not. 

The rationale behind weithing scheme is not completely clear. The summarized probabilities correspond to products of probabilities with a different number of factors: how to make sure that shorter segments do not have a lower loss just because less probabilities are multiplied. It would make sense to see the duration of the segment appear somewhere. Moreover, one would expect to treat the missing beginnings as the missing ends in the weighting scheme. As mentioned earlier, $\mathcal{L}^{(j)}_{CTC}$ does not correspond to a log-probability.

The CTC loss also sometimes need time to start converging, since the underlying network has to learn to align. The proposed method probably makes this issue bigger, and it would have been nice to at least see convergence curves, or an analysis of the training dynamics.

Finally, in the semi-supervised part about helping to generate actual labels: it is not exactly clear what or how samples would be discarded. Moreover, the proposed method could be used to simply do a forced alignment of the training set with a pretrained model to "clean" the dataset. One would expect to see i the paper the advantages of the proposed integrated training method vs this simple approach.

Edit after rebuttal:
The authors addressed the doubts and questions raised in the review.
summary_of_the_review: The problem is interesting and the paper is clear and easy to understand. However, the contribution of how to use dynamic programming to compute the CTC loss over all sub-segments is marginal. The paper could benefit from a deeper analysis and a more mathematically sound definition.

Edit:
The additional experiments and details are helpful and indeed benefit the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Mapping between two sequences of different length is frequently done via Connectionist Temporal Classification (CTC) loss when alignment is not available. CTC performs the full alignment between input and label sequences. However, there are applications when label is incomplete and only part of it is given. In the current paper authors consider the case when begin and end of label sequence are missed and extend CTC applicability to these partial labels. Based on the dynamic time warping SPRING algorithm for incomplete labels authors propose W-CTC: they similarly prepend and append label with "*" token which model missed begin and end of label sequence; then they aggregate all valid paths with the latter modification. W-CTC is empirically proved to significantly improve performance over CTC even if up to 40-70% label sequence is missed (overall performance similar to the complete label case) across different tasks, like speech, optical character, and continuous sign language recognition. W-CTC is simple and efficient.
main_review: **Strong points**
- Authors propose simple and efficient modification in CTC algorithm to support partial labels (continuous)
- Method is proved empirically to be applicable across different domains/tasks (OCR, ASR, CSLR) for a large percentage of label corruption with significance testing.

**Weak points**
- mostly the idea is based on SPRING (dynamic time warping, DTW) which is mentioned in the paper and applied for CTC instead of DTW.
- Wav2vec is learning actually representation of phonemes/tokens and later fine-tuning is more about mapping each representation cluster to the proper label. That is why it could be that W-CTC could be enough to simply learn this mapping but not the necessary representations. 
- (Limitation) Algorithm doesn't work with label corruption in the middle or at random places, which is the more often case during human labeling process.

The paper is very well written and presented: all necessary intro information is given, there are nice figures to present comparison between CTC and W-CTC are given. Overall, it is very simple to read and follow the whole text and statements. I have only minor comments:
- I suggest to mention also ASG loss https://arxiv.org/pdf/1609.03193.pdf as one of the directions on CTC modification 
- typo: "The average performance over 3 runs, are reported" -> "The average performance over 3 runs, is reported"
- Could authors provide more details on how the masking is done: random val for the begging and (r - val) for the end? 
- To resolve concern pointed above on wav2vec: Could authors perform additional experiments (even without statistical significance) for ASR where model is trained from scratch and not pretrained wav2vec to demonstrate that W-CTC is able to learn proper representations too (i believe it will as it is doing well in OCR and CSLR tasks)?
- Why in Eq. (7) it is $-0.3T$ and not $+0.3T$?
- Does normalization from Eq. (6) not converge with all ways in Eq. (5)?
- I don't get why in Eq. (6) there is $2^j$ and not $2^T$ as we assume that at the end we are staying in the wild-card state. Then actually Eq. (7) is the explicit form for normalized W-CTC.
- With respect to results on the normalization: what is the average and std for the T in different tasks? Could it be that variation is very small so that normalization doesn't influence?

summary_of_the_review: The paper is very well written with clear explanation of proposed method, well covered experimental results across domains and tasks. Proposed idea is mainly based on existing DTW algorithm (SPRING), however it is extended and applied to CTC demonstrating significantly improved results on incomplete labels. Thus, I would recommend this paper to be accepted. It could lead to a broader discussion on methods with incomplete labels and data - more real case scenario of data collection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to utilize partially labeled data for the CTC loss. In order to handle untrascribed lables, it introduces a wild-card for the beginning of the sequence and uses unconstrained endpoints for the ending of the sequence. It also discusses how to summarize the paths at the ending. The proposed algorithm was evaluated with simulated masked data and it was shown that the proposed algorithm can handle incomplete labels more effectively than the naive CTC loss while the naive CTC loss is slightly better when the label is complete.

Dynamic time warping (or DP matching) with unconstrained endpoints itself is not a new idea and a classical topic for speech recognition (e.g. word spotting). The contribution of the paper is the formal introduction of the approach to CTC and to give experimental results to confirm the effectiveness.
main_review: Strengths:

1. This paper describes the algorithm in a very clean way. It is very easy to understand the algorithm.

2. The proposed method is evaluated on three different domains. This helps to support the claim of the paper better.

Weakness:

1. The experiments are conducted only with faked masked data. Although it is valuable to have a paper discussing this topic, it is not very clear the actual benefit in real scenarios. Given that the technique itself it not very novel, this might be important to claim the novelty.

2. It uses a wild-card for the beginning while it uses unconstrained endpoints for the ending. There is no explanation about the choice. It would be great why the combination is chosen.

3. The proposed method is slightly worse than the standard CTC at r=0.0. That means that it would not be a drop-in replacement of the standard CTC. It would be better if it was shown that the proposed algorithm was actually useful for some real application.
summary_of_the_review: The paper is clean and it is easy to understand the content. The experiments are fair and the effectiveness of the approach is confirmed using simulated data. It is great that the results are given for three different domains. There are certain weak points in the paper, but the strengths slightly outweigh the weaknesses. I would recommend a weak accept.
review_title: null

Meta-review: This paper proposes an extension of CTC by considering the wild-card to adjust the label missing issues during training. The authors propose to minimize the loss over all possible sub-segments of the input to automatically align the one that matches the available transcript. It is empirically proved to significantly improve performance over CTC even if up to 40-70% label sequence is missed (overall performance similar to the complete label case) across different tasks. 

As agreed by the reviewers, the paper is well presented and the problem is interesting to a broad community. Dynamic time warping with unconstrained endpoints itself is not a new idea and a classical topic for speech recognition (e.g. word spotting). The contribution of the paper is the formal introduction of the approach to CTC and to give experimental results to confirm the effectiveness. Also the use of simulated data weakens the paper a bit.

The decision is mainly based on the clear presentation and fair experimental justification.
Decision: Accept (Poster)