Title: Incremental Sequence Classification with Temporal Consistency
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequence classification

temporal-difference learning
Abstract: We address the problem of incremental sequence classification, where predictions are updated as new elements in the sequence are revealed. Drawing on temporal-difference learning from reinforcement learning, we identify a temporal-consistency condition that successive predictions should satisfy. We leverage this condition to develop a novel loss function for training incremental sequence classifiers. Through a concrete example, we demonstrate that optimizing this loss can offer substantial gains in data efficiency. We apply our method to text classification tasks and show that it improves predictive accuracy over competing approaches on several benchmark datasets. We further evaluate our approach on the task of verifying large language model generations for correctness in grade-school math problems. Our results show that models trained with our method are better able to distinguish promising generations from unpromising ones after observing only a few tokens.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of incremental sequence classification, where the goal is to classify partial sequences at every step that elements of the sequence are revealed. Drawing parallels to TD-learning, which considers a loss to the value at the next time step, as opposed to Monte-Carlo based methods that use full sequence reward, the authors introduce a cross-entropy-like method that minimizes incremental divergences. The authors theoretically analyze their work using a Markov-chain perspective, and empirically compare against standard cross entropy methods in several text classification tasks with decoder-only transformers, demonstrating improvement over the standard cross entropy baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nOverall, I think this paper is strong, which is why this review is relatively short.\n\nOverall idea makes sense and is definitely worth exploring (I was actually surprised that this hadn't been done before; I searched online for about 30 minutes and couldn't find it; doesn't mean it hasn't been done before, but I trust that the authors did a much more comprehensive search than I did). Theory makes sense, provides insights, and seems correct to me (though I only carefully checked a few things).\n\nEvaluations are very strong. Comprehensive baselines and ablations are considered, including things I thought of and more. Confidence intervals are included (although unfortunately not on all results). A range of datasets and models are tested. Results appear strong and convincing, supporting the value of the proposed method.\n\nPaper is well written overall and reasonably clear. I would have liked more explanation in many areas, but I understand that space is limited so sometimes there's no getting around the issue of having to write in a terse way.\n\nWeaknesses:\n\nLine 18 has a typo - x8 appears twice.\n\nLine 169 - what are these mild assumptions? Should be stated somewhere (or a reference given).\n\nAs the authors acknowledge, applicability of the theory to practical settings may be limited.\n\nWould have liked to see confidence intervals on all results (including e.g., Figure 3 left and Figure 5 left). \n\nFor originality, I'm only giving a \"fair\" rating because I think the main idea is slightly marginal. That said, lots of published conference works are quite marginal, so I don't think this should prevent this paper from being accepted; I definitely think this paper meets the standards for publication.\n\nQUESTIONS:\nHow does your work compare to Nested Variational Inference https://arxiv.org/abs/2106.11302? Intuitively it feels to me like there's similarity in terms of minimizing divergences to the next-step distribution.\n\nWhat kinds of confidence intervals did you use?\n\nFigure 5 left - how come 1 token TC outperforms? How is it different from DCE if there's only 1 token? Shouldn't we expect consistency to make more of a difference with more generated tokens? I'm probably missing something here. Is it that you still generate the whole set of tokens, but only evaluate on a cut off set using labels from the whole set?\n\nI think my score is unlikely to change (more clarity would be nice, but I already gave a pretty high score, and I don't see this paper having \"groundbreaking impact\"). I suppose my score could decrease if discussion reveals weaknesses.\n\nLIMITATIONS:\nThe authors acknowledge the limitations of their theoretical results in not being directly applicable to the language models they investigate. This honesty is good, but it does make me wonder how much value the theory provides. More discussion about how different they expect things to be, along with some further empirical investigation/comparison with the tabular setting or small-scale language modelling settings, might be useful.\n\nRegarding the potential negative societal impact, at least the authors are honest in saying that they don't address this, and they do at least acknowledge that \"downstream applications of such models may carry societal implications\". And while I do see the reasoning behind downstream application mattering more than the fundamental algorithm, I still think more could have been done (at least in the Appendix and/or checklist) to discuss possible negative impacts (even if unlikely), as fundamental algorithm advancements also enable more downstream applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of sequence classification of partial sentences. The method is related to temporal difference learning in reinforcement learning. The paper proves the convergence, consistency and data efficiency of the proposed approach. Over a number of datasets, the authors show classification accuracy improvement over baselines such as DCE, Last Token only baseline etc.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Significance: the research problem is timely and important for LLMs that incrementally generate sequences by nature.\n\n- Quality: The theoretical analysis of the consistency and data efficiency is solid. The analysis in Sec. 3.1 makes it convincing why the proposed TC outperforms DCE. The baselines are well-chosen, and the performance improvement is consistent over different model sizes and families.  The experiment section further shows consistent accuracy improvements under differently short partial sentences and the efficiency benefit of the approach. The two scenarios studied in the experiment section, text classification and LLM verification, are representative of the potential real-world application of the approach.\n\n- Clarity: The paper is well organized and the writing is clear.\n\n- Novelty: The novelty seems to lie on (1) applying the idea of TD learning to text classification over discrete label space (2) replacement of the squared loss to the cross entropy in the temporal consistency condition, which is a technically sound and natural choice. This novelty is acceptable for me.\n\n**Weakness**\n\n- Methodologically, the authors measures discrepancies with the cross entropy but conventional TD learning uses squared loss between the value function outputs. Adapting the TD learning idea to incremental sequence classification is non-trivial and interesting but I wonder whether the analysis like convergence of the approach is re-iteration (or minor adaptations) of known properties of TD learning. If so, citations are appreciated.\n\n- The cited Mudgal et al. paper exemplifies a setup where the value function for continuous rewards, which perform verification of the generated text in a setup similar to Sec 4.2 of the paper. What is the advantage of the proposed approach compared to Mudgal et al. for verification of LLM generation verification? Will using the cross entropy loss perform better than the squared loss for LLM generation verification, for example, on the GSM8k studied?\n\n- The experiment part can be strengthened - most of the points also mentioned in the limitations section by the author, like experiments in larger scale models (>1.3B parameters)\n\nQUESTIONS:\n- See the weakness part\n\n- Why does the difference between the partial sentence classification accuracy (like 16 tokens) and all-token accuracy in Table 1 vary significantly across the datasets? Does the dataset affect the choices of hyperparameters like lambda?\n\n- In Line 331: \"scaling beyond than 1.3 B parameters remains an open question\", this sentence is a bit confusing to me. Do you expect the proposed approach to face scalability issues under larger models (which I don't think so), or do you just mean you did not perform larger scale experiments due to computational constraints? Please enhance the clarity of the sentence.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a condition to express consistency between predictions in incremental sequence classification tasks and proposes a corresponding loss function to enforce this property during the training of a parametric model.\n\nSTRENGTHS AND WEAKNESSES:\nThe manuscript is extremely well written. The problems are clearly introduced and motivated, and all assumptions are adequately discussed.\n\nThe proposed method is analyzed both theoretically—demonstrating convergence and consistency—and experimentally, through evaluations on standard text classification benchmarks and a specific model verification task for LLMs. The experiments are well designed.\n\nIf I understand correctly, the theoretical analysis in Section 3 relies heavily on the assumption of working with tabular data, which does not hold for the experiments presented in Section 4. While one might hope that some desirable properties carry over to the parametric setting, the manuscript does not provide strong reasons or intuitions for why this should be the case.\n\nIn terms of novelty, the idea of enforcing temporal consistency across successive prefixes in a sequence classification task is a meaningful extension of related principles from reinforcement learning into a multiclass classification setting. This, to the best of my knowledge, is an original contribution.\n\nThis work advances the field from a theoretical point of view  by introducing a new temporal-consistency loss. The method is also proven consistent in a tabular setting and shown empirically to improve predictive performance.\n\nQUESTIONS:\n1. Equation (5) resembles, to me, the typical form of a cost functional in optimal control problems, where the first term corresponds to a terminal cost and the second term resembles the integral of a running cost. However, in line 122, you remark that this equation can be seen as a form of the Bellman equation, suggesting a connection to value functions. Could you please elaborate on this?\n\n2. You mention that the computational overhead introduced by the proposed method is negligible, as training time is nearly indistinguishable from the baseline. However, it would be helpful to quantify this claim more precisely, perhaps by providing a theoretical estimate or reporting empirical runtime differences in a controlled setting.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper propose a new loss function, TC-\lambda, for training incremental sequence classifiers. The authors is inspired by TD learning and propose a method that enforces consistency between prediction distribution between different timestamps. The paper provides some theoretical analysis as well as empirical results on text classification and large language model (LLM) verification tasks.

STRENGTHS AND WEAKNESSES:
Strength:
- The method is well motivated. The connection between TC-lambda and TD learning is straightforward and easy to intuitive.
- The authors provide comprehensive empirical evaluation on text classification and LLM verification.

Weaknesses:
- Given that the model can do verification task, it should be able to serve as a good value function. It would be great if the authors actually use the value function to finetune a LLM.
- My understanding is the authors originally want to learn an incremental value function for training LLM using RL. So how is TC-lambda different from simply train a value function using TD learning? One might simply use the original TD learning to train such an incremental sequence classification task, but there is no comparison on that in the paper.

QUESTIONS:
- Is there attempt to anneal the \lambda throughout the training process?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-13