JudgeRail: Harnessing Open-Source LLMs for Fast Harmful Text Detection with Judicial Prompting and Logit Rectification

Authors: Zhongjie Ba, Hongye Fu, Yiqi Yang, Hui Chen, Qinglong Wang, Peng Cheng, Zhan Qin, Kui Ren
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Model, Harmful Text Detection, Toxic Speech Detection, Content Moderation

Abstract:
Large language models (LLMs) simultaneously facilitate the generation and detection of harmful text. Leading LLM developers, such as OpenAI, Meta, and Google, are driving a paradigm shift in the detection of harmful text, moving from conventional detectors to fine-tuned LLMs. However, these newly released models, which require substantial computational and data resources, have not yet been thoroughly investigated for their effectiveness in this new paradigm. In this work, we propose JudgeRail, a novel and generic framework that guides open-source LLMs to adhere to judicial principles during text moderation. Additionally, we introduce a new logit rectification method that accurately interprets an LLM's classification intent, rigorously controls its output format, and significantly accelerates detection. By integrating several top-performing open-source LLMs into JudgeRail without any fine-tuning and evaluating them against OpenAI Moderation API, LlamaGuard3, ShieldGemma, and other conventional moderation solutions across various datasets, including those specifically designed for jailbreaking LLMs, we demonstrate that JudgeRail can adapt these LLMs to be competitive with fine-tuned moderation models and significantly outperform conventional solutions. Moreover, we evaluate all models for detection latency, a critical yet rarely examined practical aspect, and show that LLMs with JudgeRail require only 46% to 55% of the time needed by LlamaGuard3 and ShieldGemma. The generic nature and competitive performance of JudgeRail highlight its potential for promoting the practicality of LLM-based harmful text detectors.

Decision: reject
Reviewer scores: 5, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730641408789

thread_type:
official_review

<thread_object>
creator: reviewer_p6gn

<content>
summary:
The paper introduces JudgeRail, a framework designed to enhance harmful text detection using open-source large language models (LLMs) without requiring extensive fine-tuning. By leveraging "judicial prompting" and a novel "logit rectification" technique, JudgeRail ensures accurate text classification and significantly reduces detection latency. Evaluated against established tools like OpenAI’s Moderation API and specialized models like LlamaGuard3 and ShieldGemma, JudgeRail-equipped LLMs demonstrated competitive performance while achieving faster processing times (only 46-55% of the time required by other advanced models).

strengths:
1) Introducing label systems, novel logit rectification method and calibration is really interesting and found to be helpful.

2) The evaluation and baselines includes the state of the art models.

3) The latency seems be a strength of the proposed model.

weaknesses:
1) No limitations section

2) No comparison with prompt detection techniques. Prompt detectors are simple and easy to integrate as well as latency is quite low.

3) Lack of enough baselines. Readers and researchers will be interested in comparison with prompting techniques which are much simpler to execute like baselines with only chain of thought prompting and other advanced prompting techniques.

4) The datsets implemented looks irrelevant hateful prompts differ from hate speech and hatexplain. please refer[1],[2] and it would be great if you can implement those.

5) The novel method cannot be implemented on the GPT models which is the model used by most. This makes the model cannot be utilised by many who are dependent on GPT models.

[1] Realtoxicityprompts: Evaluating neural toxic degeneration in language models
[2] Efficient Detection of Toxic Prompts in Large Language Models

questions:
1) Can you implement the proposed model on the GPT models.

2) Can this be applicable for vison models as well like text to image generation. please write a section in appendix.

3) Is the proposed method much faster than prompt detection techniques?

4) Why there is no evaluation comparison with prompting techniques? Though they might look simple readers wouldlike to their evaluation metrics as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720529502

thread_type:
official_review

<thread_object>
creator: reviewer_ycfh

<content>
summary:
This paper presents JudgeRail, a framework designed to adapt LLMs for detecting harmful content. The authors conduct extensive experiments comparing the performance of open-source LLMs with JudgeRail, against traditional harmful content detection models and commercial APIs. They introduce a logit rectification method to refine LLM outputs, ensuring more valid classifications and reducing latency. Results show that open-source LLMs equipped with JudgeRail perform comparably to commercial APIs and outperform conventional detection methods.

strengths:
- The research question is well-motivated, emphasizing the importance of detecting harmful content and preventing prompt jailbreaking in publicly deployed models. This paper effectively explores how to adapt open-source LLMs for better harmful content detection. In a landscape dominated by commercial APIs, it is crucial to investigate scalable and effective methods for open-source models. The results show that while open-source LLMs may not surpass commercial solutions in all respects, they can perform comparably, highlighting their value in content moderation efforts.

- They conduct extensive experiments to provide a robust comparison between open-source LLMs and both traditional and commercial detection models. Also, they explore the validity of their design choices very meticulously.

weaknesses:
- The novel logit rectification method has shown effectiveness on a limited set of examples. However, it is difficult to assess its overall impact on the framework's performance. The paper is missingcomparisons using simple prompts on the LLMs and ablation studies that evaluate performance with and without the logit rectification method, as these analyses could provide clearer insights into its contribution.

- While the paper is generally easy to understand, the experiments section is densely written, making it challenging to follow all the observations. For example, Section 4.3 would benefit from the inclusion of small headings or bolded paragraphs headings to better organize and group the observations, which would significantly enhance readability.

Minor comment:
- You use "a LLM" in many places (lines 183, 192 etc.), but I think it should be "an LLM". Please check with a native speaker and make correction if required

questions:
- Do you have any insights on how these open-source models operate without the JudgeRail framework? Are the LLMs capable enough with simple prompts, and does JudgeRail generally enhance their performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798185250

thread_type:
official_review

<thread_object>
creator: reviewer_4vqg

<content>
summary:
The paper presents a moderation framework for harmful text detection using open-source LLMs. First, the framework  uses prompting based on jurisdictal principles (assigning a judge role, using chain-of-thought prompting) and secondly, restrictis the logit output to a predetermined few-shot learned set of labels that are based on PerpepeciveAI, the OpenAI Moderation API and LlamaGuard3. The authors claim this is more efficient while performing almost on par with the closed models’ APIs.

strengths:
The paper introduces a simple and low-cost approach using prompting as a content moderation technique, along with controlled decoding. By adapting the prompting strategy of open LLMs models to the schema of the baseline APIs also including LlamaGuard3 and ShieldGemma, along with controlled decoding the authors achieve competitive performance without extra fine-tuning. This is interesting in spite of its simplicity. They provide a multi-faceted evaluation of false-positive classification.

The paper is generally well-written.

weaknesses:
While multiple aspect of the false-positive ratio are evaluated, I have several concerns regarding the evaluation.

1. The reated work section mentions two recent approaches, SplineLLM and RigorLLM, which JudgeLLM is not compared to (only what the authors call conventional models based on BERT (martin-ha, toxberta, s-nlp, all bert-based); I find this comparison important. How does this approach differ from those two?

2. Label set and constrained decoding:
- Lines 305-310: E.,g, “For Perspective API, we converted its multi-label detection results into binary classification ”: how does it perform using multi-label output?
- How much does the approach depend on the decoding vocabulary? 
    - I find a more detailed evaluation of the logit distribution important:
	 - different decoding vocabulary sizes
	 - a comparison to the performance using the overall logit distribution, without restrictions 
- The authors used only 100 samples to create the decoding vocabulary (logit distribution): how much does performance depend on the sample size and does it change with more/less samples?
	
Writing:
The phrasing in some parts is unneccessarily strong, e.g.: Lines 20-22: "accurately interprets an LLM’s classification intent, rigorously controls its output format, and significantly accelerates detection"

questions:
A more general question:
Interestingly, this simple approach performs on par with closed models' APIs and also fine-tuned ones such as  LlamaGuard3, while also being much more cost-efficient. What is your intuition on that? E.g, how much is it depending much on the task/datsets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730944125466

thread_type:
official_review

<thread_object>
creator: reviewer_mwtd

<content>
summary:
This paper introduces JudgeRail, a framework designed to guide open-source language models in detecting harmful text through judicial prompting and a novel logit rectification technique. The study compares JudgeRail’s effectiveness and efficiency against existing moderation tools, showing that JudgeRail enhances detection accuracy and latency without requiring fine-tuning.

strengths:
1. Significance: The paper tackles a critical issue, harmful text detection, which is increasingly urgent as LLMs are deployed widely in real-world applications.

2. Comprehensive Evaluation: The paper includes comparisons with state-of-the-art models and examines latency, a rarely explored aspect in text moderation studies, providing practical insights for real-world applications.

weaknesses:
1. Insufficient Justification: The paper does not adequately justify the specific harmful categories used in the label system, which could limit the generalizability of its results. What are P1, P2, etc., and S1, S2, etc? 

2. Lack of Novelty in Core Concept: Judge framework in content moderation is not new. There are some missed literature already explored this concept and implemented. Such as:

Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 115, 1–19. https://doi.org/10.1145/3491102.3502004
 
It would be important for the authors to justify their approach's novelty and difference with previous relevant work.

questions:
It is not clear why the paper claims that "These findings suggest that text moderation tasks have lower requirements for high-precision computing compared to text generation tasks." Low precision is the results, how could it explain the requirement? 
However, achieving acceptable results with lower precision does not inherently justify that text moderation has lower precision requirements. 
Instead, it simply indicates that this particular framework, JudgeRail, was effective in the given tests. The paper would benefit from clarifying the distinction between observed outcomes and actual task requirements, and ideally, providing evidence or reasoning that explains why moderation tasks inherently need less precision compared to generation tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732360225395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (Part 1)

comment:
We appreciate all reviewers for their valuable comments, suggestions, and questions. We have revised our manuscript accordingly to address these points. Additionally, a diff file is included in the supplementary material for the reviewers' convenience.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

The authors propose a LLM-based content moderation framework (JudgeRail) that prompts LLMs to assume the persona of a judge in detecting harmful text. This approach removes the need for additional fine-tuning, which the authors show leads to an efficient and effective system compared against other well-known content moderation solutions like LlamaGuard and ShieldGemma.  

**Strengths:**

- The paper offers a solution to an important, timely problem (automated content moderation detection) 

- The discussion of detection latency and 4-bit comparison in the paper is interesting, and I believe the authors have a valid point about inference-time efficiency being an overlooked factor that impacts practicality of moderation solutions. Their results indicate the JudgeRail approach has improved latency over baselines (46% to 55% less time than LlamaGuard and ShieldGemma).  

**Weaknesses:**

- It seems like the results are actually quite mixed for JudgeRail (table 2), and while there is lower latency, in practice I don't believe the difference would make a significant enough difference to offset the lower performance. Accuracy is more of a concern. For HateXplain, JudgeRail does better but the difference is marginal and doesn't appear to be statistically significant. 

- The experimentation lacks rigor. There were several baselines highlighted in the paper, but these were absent from comparison until the authors' rebuttal. Given the limited time for experimentation, the validity of these results is questionable. The paper would benefit from statistical significance testing. It also seems like the proposed approach would be highly brittle to the number and selection of in-context examples. While the GPT-4 labeling comparison is interesting, I'm not sure why the authors chose to denote the models' agreement with GPT-4 as the "fixed" results. Are the authors convinced these were errors in the original human-annotated dataset? If so, this needs better analysis than re-labeling with GPT-4, which has its own biases and may miss nuances that human annotators observe like dialectical variation or sarcasm. I do not find it very surprising that open LLMs would produce more similar results to GPT-4 than human annotators. 

- I do not agree with the authors that computational restrictions on training content moderation models is a major problem in content moderation. Training a 7b parameter model with the corpora sizes mentioned is feasible on smaller GPUs than an A100, and while this will be more time-consuming, even in industry there is rarely the need for continuous retraining. Additionally, there are many cloud computing resources available to researchers, sometimes at no cost. I think financial/psychological cost of label verification / human annotation for expanding existing corpora is a stronger argument to avoid retraining. 

Overall, I am not entirely convinced by either the results or motivation of the paper, and cannot recommend acceptance yet. However, I do think there are many positive aspects of the paper and the JudgeRail framework is a very solid, interesting idea. Any NLP/AI venue would be suitable for the paper, so I suggest the authors take a bit more time to refine it and resubmit.