Qwen3.5-SafeProbe-2B

A lightweight, probe-based input safety classifier for production LLM systems.

Qwen3.5-SafeProbe-2B is a classification model designed to detect harmful requests, jailbreak attempts, and prompt injection before they reach a downstream language model.

Rather than generating a moderation response, SafeProbe extracts intermediate representations from a frozen Qwen3.5 backbone and applies lightweight classification heads to estimate input risk in a single forward pass.

The model is designed for low-latency safety screening, including deployment as a first-stage guard in cascaded LLM safety architectures.

Key results

Metric	Result
Average F1 across 6 benchmarks	0.813
Average AUPRC across 6 benchmarks	0.904
Aegis 2.0 F1	0.845
WildGuardTest F1	0.864
MultiJail-vi recall	76%
p50 latency, short prompts	23 ms
Long-context p50 latency	69 ms

Latency measured with batch size 1 on an NVIDIA H200 using Hugging Face Transformers and BF16 inference.

1. Overview

Traditional LLM-based moderation systems often rely on prompting another language model to evaluate whether an input is harmful.

Although effective, this approach introduces additional autoregressive generation, inference cost, and latency.

SafeProbe explores a different approach: using intermediate representations from a pretrained language model as features for specialized safety classification.

The architecture combines:

• A frozen Qwen3.5-2B backbone.
• A prompt-level classification head for overall input safety.
• A token-level classification head for detecting localized harmful instructions in long inputs.
• A lightweight inference path that requires no autoregressive text generation.

The intended use is input screening in front of larger LLMs, rather than replacing comprehensive safety systems.

2. Highlights

Low-latency classification

SafeProbe performs safety classification in a single forward pass without generating moderation text.

• 23 ms p50 latency on short prompts.
• 69 ms p50 latency on inputs containing approximately 500–1,750 tokens.
• Batch size 1.

Specialized safety detection

The model is trained to distinguish harmful requests from sensitive but legitimate content.

Hard-negative examples include:

• Security research and vulnerability analysis.
• Historical and news-related discussions.
• Benign technical instructions.
• Process management and programming operations.
• Sensitive terminology used in harmless contexts.

The objective is to reduce unnecessary blocking while retaining sensitivity to genuinely harmful requests.

Long-context detection

A dedicated token-level classification head helps detect harmful requests embedded within otherwise benign documents.

This is particularly relevant for applications that process:

• Business documents.
• Emails and exported conversations.
• Retrieved documents.
• Long user-provided context.
• Untrusted external content.

Multilingual support

The model supports English and Vietnamese inputs.

On MultiJail-vi, SafeProbe achieves 76% harmful-request recall under the reported evaluation configuration.

3. Quick Start

Installation

pip install -U torch transformers accelerate

Load the model

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "beyoru/Qwen3.5-SafeProbe-2B"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()

Classify a text prompt

result = model.classify(
    tokenizer,
    "How can I improve the security of my server?"
)

print(result)

Classify OpenAI-style messages

SafeProbe also accepts structured conversation messages.

messages = [
    {
        "role": "system",
        "content": "You are a helpful accounting assistant."
    },
    {
        "role": "user",
        "content": "Please summarize this invoice."
    }
]

result = model.classify(tokenizer, messages)

print(result)

Output format

The classifier returns:

{
    "flagged": False,
    "p_unsafe": 0.03,
    "score": -2.1
}

Field	Description
flagged	Binary classification decision
p_unsafe	Estimated unsafe probability
score	Classification margin

The default decision threshold is 0.5 for p_unsafe, corresponding to a zero classification margin.

The latest user turn is evaluated, while preceding messages and associated context can be used as supporting information.

Input constraints

• Supported inputs: strings and OpenAI-style message lists.
• Maximum context length: 4,096 tokens.
• Truncation: left truncation.
• Recommended batch size: 1 for the reported latency configuration.

See the repository implementation for the exact output schema and supported input structures.

4. Evaluation

SafeProbe was evaluated against six public safety benchmarks using their official labels.

All evaluated models were tested on the same prompts under the reported evaluation setup.

For prompted LLM baselines, a shared moderation prompt based on an MLCommons-style hazard taxonomy was used.

F1 scores use the reported fixed evaluation thresholds. Training data was decontaminated against the evaluated test sets.

4.1 Prompt-level classification

F1 score for the unsafe class:

Model	WildGuardTest	ToxicChat	OpenAI Mod	Aegis 2.0	JBB	XSTest	Average
SafeProbe-2B	0.864	0.718	0.745	0.845	0.821	0.886	0.813
Qwen3.5-122B	0.883	0.737	0.819	0.822	0.857	0.951	0.845
Gemma4-31B	0.863	0.749	0.828	0.802	0.905	0.949	0.850
gpt-oss-120b	0.833	0.593	0.701	0.813	0.797	0.841	0.763

The results show that SafeProbe achieves competitive performance against substantially larger prompted LLM-based moderation baselines, while outperforming the listed baselines on Aegis 2.0.

These comparisons should be interpreted in the context of different model architectures and inference strategies.

4.2 Area Under the Precision-Recall Curve

Model	WildGuardTest	ToxicChat	OpenAI Mod	Aegis 2.0	JBB	XSTest	Average
SafeProbe-2B	0.941	0.800	0.848	0.926	0.952	0.960	0.904
Qwen3.5-122B	0.953	0.772	0.899	0.914	0.970	0.990	0.916
Gemma4-31B	0.929	0.820	0.884	0.870	0.985	0.983	0.912
gpt-oss-120b	0.822	0.592	0.635	0.715	1.000	0.728	0.749

AUPRC measures ranking quality across classification thresholds, complementing the fixed-threshold F1 evaluation.

4.3 False-positive behavior

Additional evaluation metrics:

Metric	SafeProbe-2B	Qwen3.5-122B	Gemma4-31B	gpt-oss-120b
SST recall	96%	98%	99%	100%
MultiJail-vi recall	76%	79%	74%	88%
XSTest over-block rate	9.6%	6.4%	3.2%	29.6%

These results illustrate the trade-off between harmful-content detection and false-positive behavior.

SafeProbe aims to provide an efficient first-stage classifier, not maximize recall at the expense of blocking legitimate content.

4.4 Long-context robustness

We evaluated whether the model could detect harmful instructions embedded within otherwise benign business text.

Metric	Result
Harmful embedded request detected	72%
Benign inserted question flagged	23%
Benign haystack flagged without insertion	0%

This experiment evaluates a specific long-context scenario with documents containing approximately 400–3,000 tokens.

The results are preliminary and should not be interpreted as a general guarantee against long-context prompt injection.

4.5 Inference latency

Model	Parameters	p50 latency
SafeProbe-2B	2B-class	23 ms
Gemma4-31B	31B	93 ms
Qwen3.5-122B	122B	110 ms
gpt-oss-120b	120B	147 ms

SafeProbe's short-prompt latency is approximately 4–6 times lower than the listed prompted LLM baselines under the reported setup.

Measurement configuration

• Framework: Hugging Face Transformers.
• Precision: BF16.
• Batch size: 1.
• Hardware: NVIDIA H200.
• SafeProbe: local inference with unfused kernels.
• Prompted LLM baselines: measured through an inference gateway.

The different execution paths should be considered when interpreting latency comparisons.

5. Architecture

SafeProbe uses intermediate representations from a frozen pretrained language model as input features for specialized classification heads.

                 Input messages
                       |
                       v
              Qwen3.5-2B Backbone
                  (Frozen)
                       |
              Intermediate Hidden States
                 Layers 6, 12, 18
                       |
                +------+------+
                |             |
                v             v
          Prompt Head     Token Head
                |             |
         Pooled features   Per-token scores
                |             |
                |          EMA + Max
                |             |
                +------+------+
                       |
                       v
              Safety Classification
                       |
              +--------+--------+
              |                 |
              v                 v
           flagged           p_unsafe

Prompt-level head

The prompt-level head aggregates representations associated with the latest user turn and predicts an overall safety score.

It is responsible for the primary input-level classification decision.

Token-level head

The token-level head produces localized token scores and applies an EMA-based aggregation followed by max pooling.

It is activated for long inputs (at least 384 tokens in the latest user turn).

This additional pathway helps preserve sensitivity to harmful instructions embedded within longer benign contexts.

Inference characteristics

• Single forward pass.
• No autoregressive text generation.
• Frozen backbone.
• Lightweight classification heads.
• Prompt-level and token-level safety signals.

6. Training

SafeProbe is trained using lightweight classification heads on top of a frozen Qwen3.5-2B backbone.

Base model: Qwen/Qwen3.5-2B

Training data consists of a mixture of public safety datasets and proprietary data, re-annotated against a written safety constitution and audited.

The training process emphasizes:

• Harmful request detection.
• Jailbreak and prompt injection detection.
• Benign-versus-harmful discrimination.
• Hard-negative examples.
• Long-context harmful instruction detection.
• English and Vietnamese safety classification.

The backbone remains frozen during the reported training process.

7. Intended Use

SafeProbe is intended for developers and researchers building safety layers around LLM applications.

Potential applications include:

• Input moderation for conversational AI.
• First-stage jailbreak screening.
• Prompt injection detection.
• Safety screening for RAG pipelines.
• Pre-inference request filtering.
• Lightweight safety classification in agent systems.
• Cascaded moderation architectures.

Recommended deployment architecture

User Request
     |
     v
SafeProbe-2B
     |
     +---- Safe ----------------> Main LLM
     |
     +---- Flagged --------------> Secondary Safety Check
                                      |
                                      v
                                Policy Decision

A cascaded architecture allows the inexpensive classifier to screen incoming traffic before invoking a more expensive moderation system when needed.

The decision threshold should be calibrated against the target application's risk tolerance and validation data.

8. Limitations

SafeProbe is an input-level safety classifier and should not be treated as a complete safety solution.

Known limitations include:

• Input-only classification: It does not independently evaluate generated model responses or tool execution results.
• Long-context limitations: Inputs exceeding approximately 3,000 tokens and complex multi-turn attacks require additional validation.
• Adversarial robustness: The model has not yet undergone comprehensive red-teaming against attacks specifically designed to evade the probe.
• Threshold sensitivity: Classification performance depends on the operating threshold and the distribution of deployment traffic.
• Multilingual coverage: English and Vietnamese are the primary supported languages; performance on other languages is not established.
• False positives and false negatives: No classifier can guarantee perfect detection across all safety categories and deployment scenarios.

For production deployments, combine SafeProbe with output moderation, tool-use validation, policy enforcement, and human review where appropriate.

9. License

This model is released under the Apache 2.0 license.

The model is based on Qwen3.5-2B. Please review the original base model license and applicable dataset terms when redistributing or using the model commercially.

10. Citation

If you use SafeProbe in your research or applications, please cite:

@misc{qwen35safeprobe2026,
  title  = {Qwen3.5-SafeProbe-2B: A Lightweight Probe-Based Input Safety Classifier},
  author = {beyoru},
  year   = {2026},
  url    = {https://huggingface.co/beyoru/Qwen3.5-SafeProbe-2B}
}

Acknowledgements

SafeProbe builds upon the Qwen3.5 model family and publicly available AI safety research and datasets.

Thanks to the researchers and open-source communities working on LLM safety, jailbreak detection, prompt injection defense, and model evaluation.

---

Developed by beyoru

Open-source AI research and engineering.