Title: From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring
Venue: NeurIPS 2025
Accepted: yes
Keywords: harmful content detection

streaming content moderation

LLM safety
Abstract: Though safety alignment has been applied to most large language models (LLMs), LLM service providers generally deploy a subsequent moderation as the external safety guardrail in real-world products. Existing moderators mainly practice a conventional full detection, which determines the harmfulness based on the complete LLM output, causing high service latency. Recent works pay more attention to partial detection where moderators oversee the generation midway and early stop the output if harmfulness is detected, but they directly apply moderators trained with the full detection paradigm to incomplete outputs, introducing a training-inference gap that lowers the performance. In this paper, we explore how to form a data-and-model solution that natively supports partial detection. For the data, we construct **FineHarm**, a dataset consisting of 29K prompt-response pairs with fine-grained token-level annotations to provide reasonable supervision for token-level training. Then, we propose the **Streaming Content Monitor (SCM)**, which is trained with dual supervision of response- and token-level labels and can follow the output stream of LLM to make a timely judgment of harmfulness. Experiments show that SCM gains 0.95+ in macro F1 score that is comparable to full-detection, by only seeing the first 18% of tokens in responses on average. Moreover, the SCM can serve as a pseudo-harmfulness annotator for improving safety alignment and lead to a higher harmlessness score than DPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo achieve the early termination of harmful outputs from large language models (LLMs), this paper proposes a data-and-model method to monitor the output streams and detect the harmfulness of the stream without knowing the complete output. More specifically, it first constructs a dataset called FineHarm for harmfulness detection with both response-level and token-level labels. Then it introduces a streaming content monitor (SCM) and trains it over the FineHarm dataset with a customized loss function. Experimental results show that the trained SCM can achieve comparable performance with the baselines using the full output for detection, while using only partial tokens (18% tokens on average).\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of the paper are listed as follows.   \n1. The paper is well-motivated. The accurate detection of harmful responses from LLM is important for its safety and reducing the latency incurred by the detection in online applications is of great importance for user experience.\n2. The authors provide open-source codes and data from this paper, making it easy to follow up and reproduce.\n3. The paper is well-written and easy to follow. Figure 1, 2, and 3 clearly illustrate the technical novelty, the necessity of SCM training and the SCM overview of the proposed method, respectively.\n4. It is great to see that the proposed method can further improve the implicit safety of LLM by applying the token-level prediction from SCM to the typical safety alignment techniques like DPO. It may improve the impact of the proposed method.   \n    \nThe weaknesses of the paper are listed as follows.     \n1. The details of the test data for evaluation are unclear. What is its relationship with the training data (FineHarm)? Is there any distribution shift here? It would be better to see the performance of the SCM on the out-of-distribution test data, since the construction of the training data (FineHarm) is not cheap. The poor performance on the out-of-distribution may limit the usage of the proposed data-and-model pipeline.\n2. It would be better if the false alarm rate could be shown to see the accuracy of SDM on benign responses.\n3. For the evaluation of Table 2, for the results where full detection is better, it would be better to see whether there exists a k value such that the performance is better than full detection. If the k exists, then it can be claimed that the proposed method can be at least as good as the full detection with appropriate k.\n4. More justification of baselines should be provided. I think it is important to compare the performance of the proposed method with the existing method mentioned in the related work about LLM Harmful Output Mitigation such as [31, 29, 15, 37] and the ones in Fine-grained Harmful Content Detection. If the related work is not involved in the evaluation, it would be better if the authors could provide explicit justification about the reason to not including them.\n\nQUESTIONS:\nThe questions are listed as follows.   \n1. For the token-level labeling, does the proposed heuristic annotation approach use the delete method in Appendix B.1?\n2. What is the relationship between the holistic scorer and the token scorer? Do they share the model architecture or weights?\n3. For Equation (4), are there any typos in step 2 and 3? For example, in step 2, should it be $\\vee P(f_{hol}(h_{n})) \\cdot P( (f_{tok} (h_{i}) )_{(i=1)}^{n})$?  \n4. What is the stopping rule to get the results in Figure 4? Is delay-k equal to 1?\n\nLIMITATIONS:\nNo. The main concern is on the generalization of SDM. Providing an evaluation on the out-of-distribution test set is helpful to show the generalization.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a critical problem: existing moderation methods rely heavily on full-response evaluation, making them unsuitable for streaming generation scenarios. The main contributions are two-fold: (1) the construction of FineHarm, a token-level fine-grained harmfulness annotation dataset; and (2) the proposal of SCM, a streaming content moderator that natively supports partial detection. By leveraging fine-grained data and a hierarchical training strategy, SCM achieves comparable performance to full-response detection by inspecting only the first 18% of the response, significantly enhancing the real-time safety of LLM systems.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Leveraging partial detection for safety moderation in streaming LLM generation is a highly valuable research direction.\n2. The token-level fine-grained annotation dataset proposed in this work facilitates the development of partial detection techniques for LLM safety.\n3. The proposed Streaming Content Monitor (SCM) is simple yet highly effective, as demonstrated by experimental results.\n4. SCM can also be used to construct token-level preference datasets, and experiments show that token-level DPO outperforms response-level DPO in enhancing safety.\n\n##Weaknesses\n\n1. The proposed SCM method shares similarities with Speculative Decoding [1,2], which relies on generating draft tokens followed by parallel verification. Therefore, it is important to clearly explain the key differences between SCM and Speculative Decoding in the related work section.\n2. The main methodological innovation of SCM lies in its Hierarchical Consistency-Aware Learning, which jointly optimizes three loss functions. However, the experimental section only includes an ablation study on the logical consistency loss. A comprehensive ablation analysis of the holistic loss design would strengthen the empirical validation.\n3. The main evaluation dataset used in the experiments appears to be under-specified. It remains unclear whether it shares the same source or distribution as the training data. In the context of safety detection, out-of-distribution generalization is crucial, as jailbreak attacks are highly diverse and unpredictable. Clarifying the train/test dataset properties and evaluating generalization would greatly enhance the credibility of the results.\n4. The baselines are limited. Comparing a task-specific fine-tuned model with general-purpose LLMs is not entirely fair. Key safety models like LlamaGuard are also missing from the evaluation.\n\n[1] Fast Inference from Transformers via Speculative Decoding\n\n[2] AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability\n\nQUESTIONS:\nAlthough SCM runs in parallel with LLMs in streaming settings, several practical concerns remain: (1) Can SCM keep up with the LLM, i.e., is the token-level prediction completed before the next token is generated? (2) If SCM is slower than the LLM's generation speed, its per-token inference cost may outweigh the efficiency gains—since full-response detection requires only a single pass.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent LLM moderation paradigms are primarily executed in a post-hoc manner, i.e., they determine the harmfulness of LLM responses after receiving the complete output. This paper introduces  FineHarm, a dataset consisting of 29K prompt-response pairs with fine-grained token-level annotations, that facilitates work on partial detection in LLM output safety assessment. Consequently, the paper proposes the \"Streaming Content Monitor\" - a method capable of partial detection of harmfulness in the LLM output. SCM fetches the latest token at each step of the autoregressive LLM output generation and provides a harmfulness judgment. When the number of tokens that are judged as harmful reaches a preset threshold, the LLM output is considered harmful and terminated midway.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nS1: Important problem and good motivation. This work practically addresses a timely safety issue.\n\nS2: Dataset contribution. This work contributes FineHarm - a dataset of LLM outputs with fine-grained, token-level harmfulness labels, which constitutes a valuable resource for the community.\n\nS3: Strong empirical results.  SCM achieves impressive results. While seeing only the beginning of outputs in many cases, SCM reaches comparable performance to a full-sequence approach.\n\n### Weaknesses\n\nW1: Reliance on heuristic annotations. My concern is the quality of the token-level labels in the  FineHarm dataset. According to my understanding, the token labels are generated by algorithmic means, and no human annotation was involved. Thus, this approach may miss subtleties, e.g. some benign tokens might be labelled as harmful just because they are in a sentence flagged as harmful. I note that the authors themselves acknowledge this limitation.\n\nW2: Somewhat incremental method.  The idea of early-stop moderation is not entirely new. Solutions mentioned in the paper, e.g. ProtectAI or GuardrailsAI, have already considered scanning the LLM output stream as it is generated. The main technical contribution of the SCM method is adapting a classifier to operate on partial token sequences. This approach is essentially a form of multi-task or hierarchical learning enabled by training on a dataset with fine-grained labels.\n\nW3: The computational overhead introduced by this safety moderation method is not fully discussed.\n\nW4: In Figure 1: \"Existing ones [harmful content detectors] take the complete output of LLMs as input, which leaks harmful\noutputs to users before the withdrawal (if outputting immediately) or causes additional service delay (if outputting\nafter detection).\". This seems like an overstatement since the paper itself mentions existing solutions that allow for midway safety assessment.\n\nW4: My understanding is that the goal of Table 2 is to provide a comparison of partial and full detection, showcasing the benefits of the approach leveraged by SMC. However, it seems that there is no comparison with other existing dedicated safety assessment models (even not intended for partial detection). \n\nW5: It seems that there is no evaluation on how SMC generalises to other datasets beyond FineHarm. An evaluation (and comparison) of an SMC trained on FineHarm on another dataset, e.g. ToxiChat, would strengthen the work.\n\nQUESTIONS:\nQ1: The authors state that no prior partial detectors exist. Thus, as baselines, they apply full-detection models in a stepwise fashion. Expectedly, such baselines underperform, due to the training–inference gap problem (partial vs. full context). I wonder if other reasonable baselines or ablations can be explored. For example, training a token-level safety moderator without the consistency loss on a dataset where each token is naively labelled as harmful/not based on whether the full response is harmful. This would quantify the effect that fine-grained (and not naive) token labels have on the method's performance.\n\nQ2: Could you please elaborate on the computational overhead introduced by this method of safety assessment?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the detection of harmful content during the streaming output of LLMs without knowing the full response. It first introduces FineHarm, a dataset with token-level safety annotations for each response, then proposes the streaming content monitor to detect harmful content during streaming output. Specifically, the training of SCM applies the token loss and the holistic loss to optimize the safety prediction of tokens and response, respectively, and the logic loss to constrain the logical consistency. The paper then introduces delay-k partial detection, which stops streaming outputs when k harmful tokens are detected. Experimental results demonstrate that SCM surpasses the baseline in partial detection. Meanwhile, it shows additional applications in the safety alignment of LLMs with token-level DPO.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strength**\n1. The paper addresses an important and timely problem: detecting harmful content during LLM streaming outputs as early as possible. \n2. The paper proposes a novel data-and-model solution including both the data annotation and the consistency-aware learning, which could provide insight for the harmful content detection.\n3. The experiments illustrate strong results on the partial detection of harmful content, which shows its potential in real-world applications.\n### **Weakness**\n1. The baseline could be improved. While there could be a few existing works exactly addressing partial detection, it is still suggested to compare its performance with other harmful content detection methods, such as LlamaGuard, for both full detection and partial detection. \n2. The generalizability of the proposed method is not explored. It seems that the test data of FineHarm is totally sampled from the uncensored Llama-3.1 with WildJailbreak, considering harmful content from various LLMs and benchmarks in the evaluation could help to better understand the practicability of the method.\n\nQUESTIONS:\nIt seems that only the sensitivity of $\\alpha$ in Figure 7. Does $\\beta$ have a significant impact on the performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}