Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes star temporal classification (STC), an extension of CTC particularly designed when only partial labeled supervision is available.
strengths_and_weaknesses: Strength:
- Modeling when partial label is available is an interesting problem and can have several applications including the ones explained in introduction. A very relevant example can be recovering complete text for ancient literature when only partial text is recovered (references 2 and 3 of paper).

Weakness:
- One major issue with this paper is clarity of text and definitions. Examples:
1. Use of 
 as star token: this symbol is widely used as start-of-sentence in the literature
2. Equation 3, $P(
|x) = \sum_{y\in\mathcal{A}} P(y|x)$, I think there should be some notion of time in this equation, otherwise I am not sure if rest of model makes sense. Second, if this is time dependent posterior, is it simply 1.0 - P(<b> | x, t)? 
3. text issues:page 5, "is is useful think ..."
4. Many issues in Table1:
4.a row starting with "TRANSFORMER[36]": there is some number under LM (2.5/5.9), what does this mean?
4.b one to the last row, there is 0.1/5.9, does this model perform 0.1 on clean or is it a typo?
5. The definition of the randomly split the words in bottom of page 6 is not clear to me.
6. The notation pDrop, how about p_drop ?

- Another major concern I have is the significance of modeling presented in this paper.
I think the change is not very far from original CTC, it only changes state transition of CTC, this can be done for any other model, like RNNT. While important, I am not sure if it is significant.

- Finally, there is a major concern about baseline comparisons:
1. All the experiments presented are on simulated dataset created by authors, no previous numbers are reported on these partially labeled dataset. The main comparison is with fully labeled data. I don't think this sufficiently evaluate STC. Why not reporting some number on some other datasets, like the ones presented in reference 2 or 3.
2. STC is not compared against other methods for recovering partially labeled data. For example one baseline can be creating pseudo label from CTC + LM for missing labeled and train with partial+pseudo labels. Maybe this will do as good as STC?

questions: I would like to hear authors opinion about my points in weakness.
limitations: I did not see any discussion on limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents Star Temporal Classification (STC) an approach to train sequential models where there is an expectation that some labels are missing.  The key machinery to accomplish this is composition with a WFST constructed, and appropriately weighted to train with sequences of the (regular expression) form * A * B * C * rather than A B C.
strengths_and_weaknesses: Strengths 
* STC is a clearly written extension to CTC, leveraging the ability to construct a decoding topology with a WFST.  
* The basic approach taken opens itself up to modifications of the WFST beyond the STC formulation.
* Demonstrates that STC can substantially improve performance on ASR and handwriting recognition, even with relatively high rates of label deletion.

Weakness
* The practical motivation is somewhat strained. While there are plenty of instances where labels may be missing (occlusion etc.), however, in most (many?) cases including those offered in the introduction the "missing" label's position is known.  For example for semi-supervised learning to mask low-confidence tokens, while the identity of the tokens may not be trusted, their position usually can.  Similarly in the handwriting recognition example, while there may be ambiguity in which characters are present, the number of characters is less confusable.
questions: * The introduction and abstract experimental performance is described imprecisely. "recover most of the performance" "low word error rates".  It might be more impactful to be more exact in these descriptions -- how close to fully observed labeling exactly? what does "low" mean in these contexts.

* STC is not evaluated with pDrop=0.0. This data point would provide confidence that this approach can be used without losing performance if labels are present (or if they are not).  Even the randomly split samples always assume at least 10% of labels are dropped.

* What is the impact of modifying the collapse function to only remove blanks instead of also removing duplicates?

* How is the token insertion penalty tuned? How big an impact does correctly setting this hyperparameter have?

* What is the rationale for word based targets followed by letter based refinement?  The 1k word pieces seems to be a better target for librispeech asr, but this isn't used in any CTC or STC experiments. why not?

* re baselines: Why not use a Conformer model in these experiments?  (the rationale for using CTC rather than RNNT is clear though.)

* The best performing approach for STC requires a two-pass training routine using Word targets to recover missing labels and then Letter targets for final training. The word targets are described as an efficiency bottleneck in section 5.3 as well.  Why can't STC work on letter targets directly?  Is this because the sequence length is too long with letter targets?  Is there any benefit to this style training for the CTC baseline?  While the performance improvements are clear, this seems unrelated to the central STC idea.

* A couple of possible typos in table 1.  The second to last line reports errors of ".1/5.9".  The third row in the LM column includes "2.5/5.9"
limitations: * Without an evaluation at pDrop=0.0 it is hard to have confidence that this is a reliable drop-in replacement for CTC training.  If it's not this limits the impact of this work to only those instances where a user has a reasonable understanding that the available training data has been partially labeled.

* STC requires additional hyperparameter tuning and these vary by pDrop.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes Star Temporal Classification (STC) to solve the problem of missing labels for weakly supervised sequence labeling tasks (such as ASR, HWR). STC uses a special star token to allow alignments which include all possible tokens whenever a token could be missing. Technically, STC uses differentiable WFSTs to implement the loss function. Experiments are conducted for speech recognition over Librispeech and handwriting recognition over IAM, which demonstrate the effectiveness of STC.

strengths_and_weaknesses: -Strengths

The paper is generally well-written. The problem of missing labels is clearly formulated and solved in the framework of differentiable WFSTs. The techniques of introducing the star token and token insertion penalty are new. Moreover, the implementation of STC, especially to integrate CPU-based WFST algorithm and GPU-based neural network gradient calculation, is non-trivial, which requires engineering efforts. The experiments seem to be good.

-Weaknesses

Concerns about the experiments:

1) The proposed method is designed to work for weakly supervised setup. But in practice, it is not known whether the labels are missing and the proportion of missing is also unknown. So the performance of applying STC in the fully-labeled setting is an important indicator. If such performance becomes worse due to the use of STC, then STC may not be practically useful.

2) In real-world tasks, the labels are missing for many reasons. An important scenario is that the labels are not really missing, but are discarded after some filtering such as by confidence measures. This scenario is described in Line 21-23 in the Introduction, but not fully evaluated in the paper.

The proposed method is only experimented in the simple scenario, where some positions in the label sequence are really missing (not labeled). This is a simple scenario, which may not be so interesting in real-world tasks. The baseline of running CTC over the incomplete labels is too weak. 

I suggest that the authors can take experiments and show improvement in more realistic scenarios (e.g., semi-supervised learning or transfer learning), as described in the Introduction.
questions: see above
limitations: see above
review_title: null

Meta-review: metareview: This paper is right on the border. I'm going to mark this as accept as the only reviewer marking reject is due to limited evaluation, but i believe the evaluation is ok (as do the other two reviewers). The idea is interesting and novel, and the paper is well written.
confidence: Less certain
Decision: Accept