Advancing LLM Safe Alignment with Safety Representation Ranking

## Abstract

Abstract The rapid advancement of large language models (LLMs) has demonstrated milestone success in a variety of tasks, yet their potential for generating harmful content has raised significant safety concerns. Existing safety evaluation approaches typically operate directly on textual responses, overlooking the rich information embedded in the model’s internal representations. In this paper, we propose Safety Representation Ranking (SRR), a listwise ranking framework that selects safe responses using hidden states from the LLM itself. SRR encodes both instructions and candidate completions using intermediate transformer representations and ranks candidates via a lightweight similarity-based scorer. Our approach directly leverages internal model states and supervision at the list level to capture subtle safety signals. Experiments across multiple benchmarks show that SRR significantly improves robustness to adversarial prompts. Our code will be available upon publication.

## 1 Introduction

Recent large language models (LLMs) have achieved remarkable capabilities across a wide range of tasks. However, this power comes with serious safety and alignment concerns [32, 15, 1]. By default, LLMs have the potential to generate biased, toxic, or harmful content, and adversarial jailbreak prompts can coax an LLM into violating its own content guidelines [20, 33, 50]. These vulnerabilities persist despite extensive alignment efforts during pre-training and post-training phases [2, 7, 17]. In practice, the potential for harmful outputs and the ability to bypass built-in safeguards raise significant concerns for deploying LLMs in real-world applications.

To mitigate these safety risks, prior work has explored a variety of defense mechanisms. A common strategy is decoding-time intervention, which redirects the decoding logic of the LLM during inference, through token distributions [38, 3] or safe prompts [37, 35, 47]. For example, SafeDecoding [38] adjusts the token distribution toward safe response distributions during decoding, while in-context defense [35] aligns the generation distributions to safe contexts with demonstrations. Such interventions can introduce a trade-off between safety and fluency: altering the decoding process may degrade the model’s natural performance on benign inputs or increase inference cost. Meanwhile, post-processing-based defenses apply judging LLMs to inspect the harmfulness of LLMs [14, 21]. Unfortunately, recent studies have shown that LLM-based safety judges are often overcautious: they flag many benign prompts as unsafe (so-called over-refusal) [22, 36]. This unreliability, i.e., high false-positive rates, limits their practical use, as it can render the model unhelpful even on innocuous tasks.

In this work, we propose an alternative paradigm (which we call Safety Representation Ranking, SRR) for LLM safety that avoids alteration of the base model’s generation logic and unreliable external judges. Our key idea is to generate multiple candidate responses to a given prompt and then rank them by safety using the model’s internal representations. This approach is similar to using a learned reward model to select outputs [10, 4, 42], but there exists an important twist: Traditional reward models are trained on the final generated text, often focusing on general measures of quality or alignment. In contrast, our proposed SRR explicitly targets safety by learning directly from the LLM’s latent features. Existing external reward models may miss fine-grained safety cues embedded in the LLM’s state vectors. Moreover, relying solely on an LLM to judge its own outputs can be unreliable and costly. By delving into the model’s internal representation space, SRR can successfully detect subtle safety-critical representations [34, 49, 48] that an output-only classifier might overlook, and do so with a lightweight ranking step at inference time.

The SRR framework works in two phases. First, we identify safety-sensitive representations through contrastive training. We construct safety contrastive groups: for each prompt, we use examples of both safe and harmful responses. We feed these paired responses through the LLM and extract their internal representations. Because the groups are semantically related but differ in safety, we can train a lightweight model (a single-layer Transformer) to distinguish *safe* vectors from *unsafe* ones. Through this process, SRR learns which features of the LLM’s latent space correlate with safe content. Then, at inference time, we use the learned safety signals to rank candidate responses. In effect, SRR filters among the model’s own outputs without changing how they were produced. Because it operates on the outputs after generation, SRR imposes almost no modification to the LLM’s decoding logic. Its only overhead is the additional cost of scoring a few extra responses with a small model, which is negligible compared to full decoding.

We conduct comprehensive experiments to validate the effectiveness of the SRR model in identifying the safety responses across multiple datasets. Not only can SRR achieve a sufficiently high accuracy in unseen harmful prompts, but it can also generalize well across different safety evaluation datasets, demonstrating its prominent generalization ability in terms of safety ranking. Additionally, we extend our analysis in terms of other alignment perspectives like privacy and fairness, which validates the potential of SRR for diverse alignment considerations and broadens the applications of SRR.

Grounded by these empirical analyses, we characterize the practicality of SRR for serving as a safeguard module in real-world deployments. First, we incorporate SRR into LLM generation to study how it strengthens their robustness against jailbreak attacks. Additionally, we compare the natural performance of SRR with vanilla generation and other defense paradigms. Because SRR only ranks among natural outputs, the quality and correctness on benign queries remain essentially unchanged. Overall, our empirical results suggest that SRR is both a practical and effective module for LLM alignment.

Our contributions in this paper can be summarized as follows:

- 1.
We introduce a novel paradigm, Safety Representation Ranking (SRR), which uses LLM internal representations to rank candidate responses by safety, without altering the model’s decoding logic.
- 2.
We demonstrate that SRR accurately selects safe outputs across diverse safety benchmarks, generalizes to novel prompts, and can be adapted to other alignment perspectives like privacy and fairness.
- 3.
We show that integrating SRR into LLM inference significantly reduces harmful outputs under attack, with negligible impact on normal task performance.

## 2 Related Work

LLM Safe Alignment.
The issue of ensuring safe alignment in LLMs has become a longstanding challenge critical to their trustworthy deployment [1, 15, 41]. Specifically, LLMs have shown a tendency to generate harmful responses when confronted with malicious requests. While current alignment techniques have improved at mitigating these risks to some extent, they still tend to be superficial and inadequate [24]. Additionally, inference-time defenses can reduce the success rate of these attacks, but they often struggle with a significant drawback of rejecting benign inputs, leading to over-refusal issues. The underlying mechanism of such issues is that these distribution-based or prompt-based defenses commonly change the decoding strategies of LLMs, making their generation distributions favor refusals. Thus, ensuring safe alignment whilst maintaining the generation distribution stands for a viable solution for these risks.

Safety Representations of LLMs. Building on the representation engineering techniques of LLMs [49, 44], which examine LLM dynamics through the lens of hidden space with perspective-specific data, recent research has revealed the existence of safety representations within these models [34, 47]. Specifically, low-dimensional and structured representations emerge in the hidden states of LLMs, which indicate their safety status. When these representations are activated in specific directions, the LLM can successfully recognize and refuse harmful prompts that go against its ethical guidelines. Conversely, when the activations move in the opposite directions, the LLMs fail to reject harmful inputs and display jailbreak behavior. This interesting property has attracted significant research interest aimed at locating and interpreting these representations [6, 46]. Nonetheless, effective methods for leveraging them to enhance the safety of LLMs remain underexplored.

Ranking-based LLM generation.
A variety of rule-based generation methods have been proposed to improve language model performance, including top-$k$ sampling [8, 13], temperature-based sampling [9], and nucleus sampling [12]. Beyond these, more refined algorithms have been developed to focus on specific tasks. For example, [30, 31] leverage majority voting to improve Chain-of-Thought (CoT) reasoning. [39, 18, 45] employ carefully designed decoding methods to generate responses that better align with specific requirements in constrained scenarios.
Recent studies [26, 29, 43, 28] train additional reward models, scaled even equivalently to the base models, to perform reranking for specific tasks.
However, these approaches are either rule-based, task-specific, or impose significant computational overhead, inherently limiting their performance potential and application scope. To overcome these limitations, we propose a more general and lightweight ranker to optimize inference-time computation and extend its applicability across diverse tasks.

## 3 Methodology

In this section, we propose Safety Representation Ranking (SRR), a listwise learning-to-rank framework for scoring LLM responses by safety. Given an instruction, SRR generates a set of candidate completions and ranks them such that safe responses receive higher scores than unsafe ones. The core idea is to extract internal representations from a frozen base LLM and train a lightweight transformer ranker to assess instruction-response compatibility. Below, we describe the key components of SRR: candidate response generation, ranker architecture, and optimization with a listwise ranking objective.

### 3.1 Candidate Response Generation

To construct candidate lists for training, we sample the base LLM multiple times using stochastic decoding with moderate temperature. This yields a diverse set of $m$ plausible responses $\{{resp}_{1},\dots,{resp}_{m}\}$. for each instruction. We remove duplicates and include both benign and adversarial candidates by injecting jailbreak prompts [35, 50]. This helps ensure that the candidate pool contains both safe answers and hard negatives (unsafe answers) for training. Each response is labeled with a binary safety tag $y_{i}\in\{0,1\}$, where $y_{i}=1$ indicates a safe response. For training, we construct tuples of the form $(inst,\{resp_{i},y_{i}\}_{i=1}^{m})$, where each list includes at least one safe and one unsafe response.

### 3.2 Ranker Model Architecture

The core of SRR is a neural ranker that computes a compatibility score between an instruction and each candidate response. We build this ranker as follows:

- •
Step 1. Representation extraction: We use the base LLM as a fixed feature extractor. For each textual input (instruction or response), we run it through the LLM and take the hidden-state vector at a selected layer as its representation. Concretely, let $\mathbf{h}_{\text{inst}}\in\mathbb{R}^{d}$ be the hidden vector for the instruction (the state of the last token in the sequence) at the chosen layer, and let $\mathbf{h}_{\text{resp},i}\in\mathbb{R}^{d}$ be the hidden vector for the $i$-th response. Since the backbone is trained for next-token prediction, the final layers tend to overfit to this specific task. In contrast, intermediate layers typically provide more comprehensive representations of the preceding context, making them better suited for capturing the overall features required for ranking [27]. Therefore, we adopt intermeidiate layers to capture high-quality semantic content.
- •
Step 2. Transformer encoder: We map each high-dimensional LLM vector (typically $d=4096$) to a lower-dimensional space using a shared learned linear projection. This makes the downstream transformer encoder more lightweight and efficient. We concatenate the projected vectors into a sequence:
$[\mathbf{h}_{\text{inst}},\mathbf{h}_{\text{resp},1},\dots,\mathbf{h}_{\text{
resp},m}].$
(1)
This sequence is then passed through a Transformer encoder (single-layer in our implementation). The Transformer’s self-attention layers let the instruction embedding interact with each response embedding. After passing through the encoder, we obtain output vectors $\mathbf{o}_{\text{inst}}$ and $\mathbf{o}_{\text{resp},i}$ corresponding to the instruction and each response, respectively. Intuitively, $\mathbf{o}_{\text{inst}}$ is the contextualized instruction representation (having attended to all responses) and $\mathbf{o}_{\text{resp},i}$ is the $i$th response representation attended to the instruction.
- •
Step 3. Similarity computation: From these encoder outputs we compute a similarity score $s_{i}$ for each response. We use cosine similarity:
$s_{i}=\cos(\mathbf{o}_{\text{inst}},\mathbf{o}_{\text{resp},i})=\dfrac{\mathbf
{o}_{\text{inst}}^{\top}\mathbf{o}_{\text{resp},i}}{\|\mathbf{o}_{\text{inst}}
\|\|\mathbf{o}_{\text{resp},i}\|}.$
(2)
These scores $s_{i}\in[-1,1]$ measure the alignment between instruction and responses in the embedding space, which are used as unnormalized logits for ranking, with a temperature scaling parameter $\tau$ applied before softmax to control sharpness.

### 3.3 Training Objectives and Pipeline

We train the ranker end-to-end (keeping the base LLM frozen) using a listwise ranking loss. For safe/unsafe training, we interpret the similarity scores ${s_{i}}$ for a list of $m$ candidates as unnormalized logit scores. We compute a softmax probability for each response:

$$ $\hat{p}_{i}=\dfrac{\exp(s_{i}/\tau)}{\sum_{j=1}^{m}\exp(s_{j}/\tau)}.$ (3) $$

We also define a ground-truth probability distribution $p^{*}$ over the list, which places all mass on the safe responses. For instance, if there are $k$ safe responses among the $m$, we set $p^{*}_{i}=1/k$ for each safe response with $y_{i}=1$ and 0 0 for unsafe ones with $y_{i}=0$. Then we minimize the Kullback–Leibler divergence:

$$ $\mathbb{D}_{\mathrm{KL}}\left(p^{*}\,\|\,\hat{p_{i}}\right)=\sum_{i=1}^{m}p^{* }_{i}\log\dfrac{p^{*}_{i}}{\hat{p}_{i}}$ (4) $$

This loss, as a standard choice [23, 19], encourages the model to assign high probability to safe candidates. In effect, the ranker is trained so that the instruction and safe responses have higher cosine similarity than instruction-unsafe pairs.

### 3.4 Brief Summary

In conclusion, SRR learns to map instructions and responses into a joint embedding space where safety alignment is captured by similarity. The use of a transformer encoder allows each response to be scored in the context of the instruction and the other candidates. In the inference stage, for any new instruction and its candidate outputs, the ranker can compute similarity scores and produce a safety-based ranking, without further supervision. A pseudo-algorithm is provided in Algorithm [1](https://arxiv.org/html/2505.15710v1#alg1). First, SRR generates diverse candidate responses for each instruction during the training phase (line 2). It then extracts features from the LLM and uses a transformer-based ranker to compute similarity scores between instructions and responses (line 3-6). These scores are normalized via softmax and compared to ground-truth probabilities to compute a listwise loss, which is used to update the ranker (line 7-10). During inference, the algorithm repeats the feature extraction and similarity computation steps to rank responses based on safety.

Figure: Algorithm 1 Safety Representation Ranking (SRR)

## 4 Evaluation

In this section, we conduct comprehensive evaluations to show the effectiveness of SRR, including safety, privacy and bias, starting with the overall setup. To further demonstrate the generality of our approach, we also evaluate its ability on other datasets. We also state that the natural performance in math and coding does not deteriorate after attaching the ranker to the model.

### 4.1 Experiment Set-up

Models and datasets.
In our experiment, we apply three popular LLMs, including (1) Qwen2.5-7b-Instruct [40]
(2) Mistral-7-v0.3 [16], and (3) Vicuna-7b-v1.5 [48].

For datasets, we apply Harmbench [21], SorryBench [36], and JailbreakBench [5].
The HarmBench dataset here refers to the standard section of the Harmbench dataset, which includes 200 different harmful prompts in various areas. The SorryBench and the JailbreakBench have similar contents. For each dataset, we extract 50 of them as the training dataset, and the rest is used as the testing dataset. For each prompt in a dataset, we sample answers from the base model using In-context Attack and In-context Defense, each 20 times. After sampling, we check whether the answers are valid by examining the keywords in them. For safe answers, we examine "Sorry", "unable", "illegal", and "understand". For harmful answers, we examine "sure", "certainly". We then filter the answers using the above criterion to get high-quality data.

Metrics. In the experiments in this section, we request the model to choose between the safe answer and the harmful answer, and verify its correctness, referring to its label.

Ranker Settings. In all experiments, the rankers are implemented using a single Transformer block. The trainable parameters of the ranker model is less than 5M. They operate on features extracted from approximately the bottom 25% of the base model’s layers. During training and evaluation, every data group includes two candidate answers. The ranker is trained to distinguish the answers as safe and harmful. The hyperparameters are set as follows: learning rate is set to 0.001, weight decay is set to 0.0001, dropout is set to 0.1, and momentum is set to 1.0.

Baseline. The baseline of the experiment adopts a reward model to rate answers generated by the base model. A pretrained GPT2 [25] is used as the reward model in the experiment. Small as it seems, a GPT2 model is still 20 times larger than the ranker model.

### 4.2 Overall evaluation

**Table 1: Ranking accuracy of SRR in distinguishing safe and harmful prompts.**
|  |  | Model |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Source Dataset | Method | Qwen | Mistral | Vicuna | Average |
| Harmbench | Baseline | 41.18 | 35.21 | 57.60 | 44.66 |
| Ours | 82.35 | 91.55 | 90.40 | 88.10 |  |
| SorryBench | Baseline | 56.72 | 52.82 | 55.26 | 54.93 |
| Ours | 85.57 | 90.15 | 87.98 | 87.90 |  |
| JailbreakBench | Baseline | 70.00 | 67.39 | 50.00 | 62.46 |
| Ours | 80.00 | 95.65 | 95.24 | 90.30 |  |

We use the transformer-architectured ranker to improve the safety of different models on different datasets. As depicted in the [1](https://arxiv.org/html/2505.15710v1#S4.T1), our method greatly outperforms the reward model in all base models and all datasets. The accuracy of many experiments reach 90%. Our lightweight method significantly out performs the reward model (gpt2), despite being far smaller in scale. Specifically, when Qwen is used as the base model, the ranker reaches 82.35%, 91.55%, 90.40% respectively on three datasets. Similarly, the results are 85.57%, 90.15%, 87.98% when the base model is Mistral. Finally the performance is 80.00%, 95.65%, 95.24% when the base model is Vicuna.
This implies that rankers can adapt to even larger models.

### 4.3 Cross dataset validation

To further evaluate the generalization capability of our SRR framework across different safety benchmarks, we conduct cross-dataset validation experiments. We apply the ranker trained on one dataset to other unseen datasets. This experimental setup helps us demonstrate whether the model can effectively identify and prioritize safe responses regardless of the dataset’s specific characteristics or the types of adversarial prompts it contains.

**Table 2: Cross-dataset ranking accuracy of SRR in distinguishing safe and harmful prompts.**
|  |  | Model |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Source Dataset | Evaluation Dataset | Qwen | Mistral | Vicuna | Average |
| Harmbench | SorryBench | 76.96 | 88.06 | 66.04 | 77.02 |
| JailbreakBench | 80.00 | 93.48 | 85.71 | 86.40 |  |
| SorryBench | Harmbench | 76.47 | 90.14 | 80.00 | 82.20 |
| JailbreakBench | 77.78 | 89.13 | 76.19 | 81.03 |  |
| JailbreakBench | HarmBench | 79.41 | 89.44 | 90.40 | 86.42 |
| SorryBench | 72.41 | 87.16 | 78.59 | 79.39 |  |

The results in Table [2](https://arxiv.org/html/2505.15710v1#S4.T2) show that our SRR framework achieves consistently strong cross-dataset performance across all three LLMs (Qwen, Mistral, and Vicuna). When trained on one dataset and evaluated on another, SRR maintains a high level of accuracy in distinguishing safe from harmful responses. For instance, a ranker trained on Harmbench achieves 77.02% average accuracy on SorryBench and 86.40% on JailbreakBench. Similarly, a ranker trained on SorryBench achieves 82.20% on Harmbench and 81.03% on JailbreakBench. This cross-dataset effectiveness demonstrates that SRR’s safety signal is not overly specialized to any particular dataset but instead captures generalizable features of safety within the LLM’s internal representations.

This ability to generalize across different safety benchmarks is crucial for real-world deployment. In practical applications, LLMs may encounter a wide variety of adversarial prompts that differ significantly from those seen during training. The strong cross-dataset performance of SRR suggests that it can serve as a robust safeguard module, effectively filtering out harmful responses even when the specific types of attacks vary. This provides evidence that SRR’s approach of leveraging internal model representations for safety ranking is both versatile and adaptable to diverse safety challenges.

### 4.4 Extension to other alignment perspectives

In this part, we also extend the application of our Safety Representation Ranking (SRR) framework to other critical alignment perspectives beyond general safety, namely privacy and fairness. These dimensions are essential for ensuring that LLMs not only avoid harmful content but also respect user privacy and produce unbiased, equitable responses. Evaluating SRR’s effectiveness in these areas helps to demonstrate its versatility and potential for broader alignment applications.

Privacy. To evaluate the potential of SRR in addressing privacy concerns, we conducted experiments on the Harmcopy dataset, which contains prompts related to privacy infringement. The results are presented in Table [3](https://arxiv.org/html/2505.15710v1#S4.T3).

**Table 3: Ranking accuracy of SRR in distinguishing infringement and benign prompts.**
|  | Model |  |  |  |
| --- | --- | --- | --- | --- |
| Dataset | Qwen | Mistral | Vicuna | Average |
| Harmcopy | 98.08 | 95.83 | 89.74 | 94.28 |

The results show that SRR achieves a high accuracy rate in distinguishing between privacy-infringing and benign prompts across all models. In particular, Qwen demonstrates the highest accuracy of 98.08%, followed by Mistral with 95.83% and Vicuna with 89.74%. The average accuracy across all models is 94.28%, indicating that SRR is effective in identifying privacy-related safety concerns.

This strong performance in the privacy context further validates the generalizability of our approach. The ability to adapt to privacy-specific prompts shows that SRR can capture fine-grained safety signals related to different alignment perspectives beyond just general harmful content. By leveraging the internal representations of LLMs, SRR can effectively identify privacy risks without requiring extensive retraining or modification of the underlying model architecture. This makes it a versatile and efficient solution for enhancing the privacy safeguards in LLM applications.

Fairness.
To assess the effectiveness of SRR in ensuring fairness, we conducted experiments on the BBQ dataset. This dataset is designed to evaluate the model’s ability to avoid generating responses that may contain biases or unfair content. The results are presented in Table [4](https://arxiv.org/html/2505.15710v1#S4.T4).

**Table 4: Ranking accuracy of SRR in distinguishing safe and harmful prompts.**
|  | Model |  |  |  |
| --- | --- | --- | --- | --- |
| Dataset | Qwen | Mistral | Vicuna | Average |
| Biasedbenchmark for QA | 54.82 | 52.09 | 50.64 | 52.52 |

The results indicate that SRR achieves moderate accuracy in identifying and mitigating biased or unfair responses. The average accuracy across all models is 52.52%, which is relatively lower compared to the results obtained in privacy and safety evaluations. This suggests that while SRR demonstrates some capability in detecting fairness-related issues, there is still room for improvement in this area.
This performance highlights the complexity of fairness as a multifaceted alignment perspective. Fairness evaluation involves nuanced considerations of various social contexts, cultural factors, and potential biases that may not be as directly reflected in the internal representations of LLMs as other forms of harmful content. Despite this, SRR shows a foundational ability to distinguish between more and less fair responses, indicating that it can serve as a starting point for more specialized fairness enhancements in LLM applications.

Overall, the results demonstrate the initial potential of SRR in addressing privacy and fairness concerns for future advancement in this critical area of LLM alignment.

### 4.5 Brief Summary

In this section, we have comprehensively evaluated the effectiveness of our proposed SRR framework across various dimensions of LLM safety and alignment. Our experiments demonstrate that SRR achieves significant improvements in identifying and prioritizing safe responses over harmful ones, with high accuracy across multiple safety benchmarks. The cross-dataset validation further confirms the generalizability of SRR, showing its ability to adapt to different types of adversarial prompts and datasets without being overly specialized. Additionally, our extension to privacy-related prompts reveals SRR’s potential in mitigating privacy-infringing outputs, achieving a strong accuracy rate. Even in the context of fairness, where the task is more nuanced, SRR shows a foundational capability to distinguish between biased and unbiased responses, though with moderate accuracy that suggests room for further enhancement. Overall, these results highlight SRR’s versatility and effectiveness as a safeguard module that can be integrated into LLM inference to significantly reduce harmful outputs under attacks.

## 5 Discussion

This section further discusses the considerations for SRR in practical deployment. We focus on two fundamental problems:

- 1.
To what extent can SRR mitigate safety alignment issues?
- 2.
How does SRR impact the natural performance of LLMs?

### 5.1 Real-world application

Recall that we mainly apply the classification accuracy as the main metric to evaluate the precision of SRR in ranking the safety of multiple responses. In this part, we further explore how SRR can improve the safety alignment of LLMs, since aligned LLMs have already exhibited certain robustness against harmful prompts. To this end, we incorporate SRR during real-time inference of the protected LLMs, rather than classifying simulated harmful or safe responses. We also consider practical jailbreak attacks to demonstrate the robustness of SRR. The baseline in this experiment is "first accuracy", which means choosing the answer with the highest possibility generated by the base model. The results shown in Table [5](https://arxiv.org/html/2505.15710v1#S5.T5), [6](https://arxiv.org/html/2505.15710v1#S5.T6), and [7](https://arxiv.org/html/2505.15710v1#S5.T7) demonstrate that SRR significantly enhances the safety alignment of LLMs in real-world applications. When integrated into the inference process of protected LLMs, SRR demonstrates robust performance against practical jailbreak attacks. This indicates that SRR can effectively improve the safety mechanisms of LLMs, reducing their vulnerability to adversarial prompts. By leveraging the model’s internal representations, SRR provides an efficient and effective safeguard without compromising the natural performance of the LLMs. Overall, these findings support the practical utility of SRR as a valuable tool for improving the safety and reliability of LLMs in real-world scenarios.

**Table 5: Real-world ranking accuracy of SRR in distinguishing safe and harmful prompts in HarmBench**
|  | Model |  |  |
| --- | --- | --- | --- |
| Method | Qwen | Mistral | Average |
| First | 82.52 | 54.43 | 68.48 |
| Ranker | 83.22 | 63.29 | 73.26 |

**Table 6: Real-world ranking accuracy of SRR in distinguishing safe and harmful prompts in JailbreakingBench**
|  | Model |  |  |
| --- | --- | --- | --- |
| Method | Qwen | Mistral | Average |
| First | 16.25 | 32.91 | 24.58 |
| Ranker | 38.75 | 39.24 | 39.00 |

**Table 7: Real-world ranking accuracy of SRR in distinguishing safe and harmful prompts in SorryBench**
|  | Model |  |  |
| --- | --- | --- | --- |
| Method | Qwen | Mistral | Average |
| First | 84.28 | 46.22 | 65.25 |
| Ranker | 86.16 | 67.23 | 76.70 |

### 5.2 Natural performance

As discussed in earlier sections, a key advantage of SRR is that it does not intervene in the decoding process of the base language model. This allows SRR to be seamlessly applied at inference time without modifying generation behavior, thereby preserving the model’s natural task performance. In this section, we empirically validate this claim using a mathematical reasoning benchmark. We evaluate SRR using the MATH dataset [11], which contains 12,500 competition-level math problems spanning seven topics and five difficulty levels. To assess performance, we extract the final answer from each model-generated response and compare it against the ground-truth answer.

We use Qwen2.5-7B-Instruct as the base model. For each instruction, we sample 10 completions and apply the SRR ranker, which is trained solely on safety datasets, to rank them by their predicted safety. The top-ranked response is selected as the final answer. We then compare the answer accuracy of the ranked responses against the accuracy obtained by the base model’s default outputs.

The results are shown in Table [8](https://arxiv.org/html/2505.15710v1#S5.T8). Across all settings, the accuracy of the SRR-ranked completions remains nearly identical to the base model’s natural accuracy (68.7%). In fact, slight fluctuations ($\pm$0.2%) are observed depending on which safety dataset the ranker was trained on, but these differences fall within the margin of noise and do not indicate degradation in performance. Notably, this result holds despite the SRR ranker being trained exclusively on safety supervision signals, without any exposure to mathematical reasoning data. This demonstrates that the SRR scoring mechanism does not introduce unintended bias toward specific task domains or alter the correctness of model outputs in benign settings.

**Table 8: Accuracy (%) on the MATH dataset when responses are ranked using SRR trained on different safety datasets.**
| Source Dataset | Natural | HarmBench | SorryBench | JailbreakBench |
| --- | --- | --- | --- | --- |
| Accuracy | 68.7 | 69.1 | 68.5 | 68.6 |

## 6 Limitations

Although SRR performs excellently in enhancing the safety of LLMs, there are still a few minor limitations. SRR might need task-specific fine-tuning for optimal performance in certain situations, although the training cost is low. While it generalizes well across multiple safety benchmark datasets, its adaptability to special-domain safety scenarios requires further testing. Also, SRR’s effectiveness partly relies on the LLM generating diverse candidate responses; if the responses lack diversity, SRR’s performance may be somewhat affected. Despite these minor limitations, SRR remains a robust and practical solution for boosting LLM safety and reliability in various real-world applications.

## 7 Conclusion

In this paper, we introduced Safety Representation Ranking (SRR), a novel listwise ranking framework that leverages the internal representations of LLMs to select safe responses without altering the model’s decoding logic. Through contrastive training, SRR identifies safety-sensitive features within the LLM’s hidden states and uses them to rank candidate responses based on safety. Our method not only improves robustness against adversarial prompts but also generalizes well across different safety evaluation datasets. Furthermore, SRR demonstrates potential for addressing other alignment perspectives such as privacy and fairness. Experimental results indicate that SRR significantly reduces harmful outputs under attack while maintaining performance on benign tasks. Overall, SRR serves as a practical and effective safeguard module for LLM alignment, offering a new paradigm for enhancing the safety and reliability of LLMs in real-world applications.

## References

- [1]
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al.
Foundational challenges in assuring alignment and safety of large language models.
Transactions on Machine Learning Research, 2024.
- [2]
Yuntao Bai et al.
Constitutional ai: Harmlessness from ai feedback, 2022.
- [3]
Somnath Banerjee, Sayan Layek, Soham Tripathy, Shanu Kumar, Animesh Mukherjee, and Rima Hazra.
Safeinfer: Context adaptive decoding time safety alignment for large language models.
In AAAI, volume 39, pages 27188–27196, 2025.
- [4]
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini.
Large language monkeys: Scaling inference compute with repeated sampling.
arXiv preprint arXiv:2407.21787, 2024.
- [5]
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024.
- [6]
Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li.
Finding safety neurons in large language models.
arXiv preprint arXiv:2406.14144, 2024.
- [7]
Josef Dai, Xuehai Pan, Ruiyang Sun, et al.
Safe rlhf: Safe reinforcement learning from human feedback.
In ICLR, 2024.
- [8]
Angela Fan, Mike Lewis, and Yann Dauphin.
Hierarchical neural story generation.
In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018.
- [9]
Jessica Ficler and Yoav Goldberg.
Controlling linguistic style aspects in neural language generation.
In Proceedings of the Workshop on Stylistic Variation, 2017.
- [10]
Rasmus Boll Greve, Emil Juul Jacobsen, and Sebastian Risi.
Evolving neural turing machines for reward-based learning.
In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 117–124, Denver Colorado USA, 2016. ACM.
- [11]
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
In Advances in Neural Information Processing Systems, 2021.
- [12]
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.
The curious case of neural text degeneration.
In International Conference on Learning Representations, 2020.
- [13]
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi.
Learning to write with cooperative discriminators.
In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018.
- [14]
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.
Llama guard: Llm-based input-output safeguard for human-ai conversations.
arXiv preprint arXiv:2312.06674, 2023.
- [15]
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al.
Ai alignment: A comprehensive survey.
arXiv preprint arXiv:2310.19852, 2023.
- [16]
Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al.
Mistral 7b. arxiv.
arXiv preprint arXiv:2310.06825, 10, 2023.
- [17]
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez.
Pretraining language models with human preferences.
In ICML, 2023.
- [18]
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.
Contrastive decoding: Open-ended text generation as optimization.
In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
- [19]
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al.
Lipo: Listwise preference optimization through learning-to-rank.
arXiv preprint arXiv:2402.01878, 2024.
- [20]
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu.
Jailbreaking chatgpt via prompt engineering: An empirical study, 2023.
- [21]
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
In ICML, 2024.
- [22]
Swetasudha Panda, Naveen Jafer Nizar, and Michael L Wick.
Llm improvement for jailbreak defense: Analysis through the lens of over-refusal.
In Neurips Safe Generative AI Workshop 2024, 2024.
- [23]
Alberto Purpura, Gianmaria Silvello, and Gian Antonio Susto.
Learning to rank from relevance judgments distributions.
Journal of the Association for Information Science and Technology, 73(9):1236–1252, 2022.
- [24]
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson.
Safety alignment should be made more than just a few tokens deep.
arXiv preprint arXiv:2406.05946, 2024.
- [25]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners.
2019.
- [26]
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar.
Rewarding progress: Scaling automated process verifiers for llm reasoning.
arXiv preprint arXiv:2410.08146, 2024.
- [27]
Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv.
Does representation matter? exploring intermediate layers in large language models.
arXiv preprint arXiv:2412.09563, 2024.
- [28]
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li.
Reft: Reasoning with reinforced fine-tuning.
In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
- [29]
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang.
Interpretable preferences via multi-objective reward modeling and mixture-of-experts, 2024.
- [30]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models, 2023.
- [31]
Xuezhi Wang and Denny Zhou.
Chain-of-thought reasoning without prompting.
arXiv preprint arXiv:2402.10200, 2024.
- [32]
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al.
A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more.
arXiv preprint arXiv:2407.16216, 2024.
- [33]
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt.
Jailbroken: How does llm safety training fail?
In NeurIPS, 2023.
- [34]
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson.
Assessing the brittleness of safety alignment via pruning and low-rank modifications.
In ICML, 2024.
- [35]
Zeming Wei, Yifei Wang, and Yisen Wang.
Jailbreak and guard aligned language models with only few in-context demonstrations.
arXiv preprint arXiv:2310.06387, 2023.
- [36]
Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al.
Sorry-bench: Systematically evaluating large language model safety refusal.
In ICLR, 2025.
- [37]
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu.
Defending chatgpt against jailbreak attack via self-reminders.
Nature Machine Intelligence, 2023.
- [38]
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran.
Safedecoding: Defending against jailbreak attacks via safety-aware decoding.
In ACL, pages 5587–5605. Association for Computational Linguistics (ACL), 2024.
- [39]
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran.
SafeDecoding: Defending against jailbreak attacks via safety-aware decoding.
In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
- [40]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2.5 technical report.
arXiv preprint arXiv:2412.15115, 2024.
- [41]
Eliezer Yudkowsky.
The ai alignment problem: why it is hard, and where to start.
Symbolic Systems Distinguished Speaker, 4(1), 2016.
- [42]
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang.
Rest-mcts*: Llm self-training via process reward guided tree search, 2024.
- [43]
Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu.
General preference modeling with preference representations for aligning language models.
arXiv preprint arXiv:2410.02197, 2024.
- [44]
Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun.
Adversarial representation engineering: A general model editing framework for large language models.
arXiv preprint arXiv:2404.13752, 2024.
- [45]
Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi.
Alleviating hallucinations of large language models through induced hallucinations.
arXiv preprint arXiv:2312.15710, 2024.
- [46]
Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh.
Identifying and tuning safety neurons in large language models.
In ICLR, 2025.
- [47]
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng.
On prompt-driven safeguarding for large language models.
In International Conference on Machine Learning, pages 61593–61613. PMLR, 2024.
- [48]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
- [49]
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al.
Representation engineering: A top-down approach to ai transparency.
arXiv preprint arXiv:2310.01405, 2023.
- [50]
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models.
arXiv preprint arXiv:2307.15043, 2023.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Could you provide empirical evidence comparing the performance of SRR with directly ranking the final response or the final hidden state? This is critical to substantiate the claim that internal states capture more fine-grained safety signals.
2. Can you report comparative results on metrics like safety rate and benign task performance against more advanced baselines on the same datasets?
3. Can you compare the computational overhead of SRR with other inference-time aligning methods?
4. It seems that all LLMs are safety-aligned models. Does SRR still perform well for unaligned LLMs (e.g. uncensored models) as sampling refusal response from these models is more difficult?
soundness: 2
strengths: 1. This paper introduces a lightweight inference-time aligning method for LLMs, which does not need heavy training. While ranking responses itself is not new, using intermediate hidden states is an underexplored approach.
2. The paper is generally well-written, and the methodology is well-structured and described in detail.
summary: This paper introduces Safety Representation Ranking (SRR), a framework designed to enhance the safety alignment of LLMs during inference. The core premise of SRR is to leverage the intermediate-layer representations of LLMs to rank multiple candidate responses for a given prompt based on their safety. The method involves generating candidate completions through sampling and extracting their hidden state vectors from a selected intermediate layer of the LLM, and a lightweight, trainable Transformer-based ranking models trained with a KL-divergence based ranking objective. During inference, the ranking model can rate different responses of the LLM from an instruction, and therefore provide a safer response. The paper evaluates SRR across three base LLMs and three safety benchmarks demonstrating high ranking accuracy in distinguishing safe from harmful responses compared with the baseline. Experiments also show that the SRR generalizes effectively in cross-dataset evaluations and  other alignment perspectives like privacy and fairness.
weaknesses: 1. **The motivation of the proposed method is inadequate, which also limits the novelty of the paper.** The paper's central claim that ranking candidate responses using internal representations is superior to using final text is not sufficiently motivated or proven. It states that traditional reward models "may miss fine-grained safety cues embedded in the LLM’s state vectors," but provide no empirical evidence to support this critical claim. The entire motivation for using intermediate representations hinges on this unverified hypothesis. A direct and necessary ablation is missing: comparing the performance of SRR using internal states against another ranking model training on the same dataset but using only the final text embeddings of the responses. Without this comparison, it is impossible to conclude that the intermediate representations are the key factor for the observed performance, as opposed to, for instance, the listwise ranking objective itself or the specific architecture of the ranker.

2. **The experimental settings, including the metric and baseline are not convincing enough.** Firstly, the primary metric, "ranking accuracy," is an indirect measure compared to the safety rate or attack success rate under jailbreak scenarios, which is widely used for LLM safety research. A high ranking accuracy does not directly translate to a model that robustly refuses to answer harmful instructions in practice. Secondly, the choice of baselines is limited and fails to contextualize SRR within the current research landscape. Comparing solely against a GPT-2 based reward model is inadequate. To demonstrate its contribution, SRR must be compared against contemporary and strong inference-time defense methods such as ARGS [1] and SafeDecoding [2]. The absence of these comparisons leaves the reader uncertain of whether SRR offers a tangible improvement over existing, and potentially simpler, techniques.


3. **The analysis is not sufficient enough to demonstrate the advantages of SRR.** The paper lacks a thorough analysis of SRR, including (1) Computational Overhead: While the ranking model itself is small, the overall inference cost of SRR includes generating multiple candidate responses. The total computational overhead of this process, and how it compares to the cost of other defense mechanisms (e.g. SafeDecoding), is not analyzed. (2) Layer Selection Rationale: The decision to use representations from "the bottom 25% of the base model's layers" is presented heuristically without systematic validation. The paper does not explore how the choice of layer affects performance, nor does it justify why intermediate layers are better than later layers for this specific task. An ablation study varying the layer depth is essential to validate this key design choice and ensure the method's stability across different model architectures.

[1] ARGS: Alignment as Reward-Guided Search, ICLR 2024.

[2] SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding, ACL 2024.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: Please refer to the weaknesses section.
soundness: 2
strengths: The main strengths of this paper are:

1) The problem this paper studies is important: Aligning LLM's generation to be more safe.

2) The proposed approach is simple and easy to implement.

3) The experiments span three different LLM architectures, and a few datasets.
summary: This paper studies the safety of Large Language Models (LLM) generation and attempts to address it. It proposes a simple hook that ranks multiple generations of the LLM at a given received input prompt, and outputs the one that is the safest. The proposed approach is simple, and the experiments show potential positive performance gain.
weaknesses: Despite the paper's strengths, there are major weaknesses that need to be addressed before getting this paper accepted:

1) From the methodology side, I have the following two criticisms that need to be addressed:

1a) Leveraging the embedding of the last generated token as an embedding for the entire instruction/response seems incorrect and has too little information about the entire sequence. One should employ the embeddings for all instruction/response tokens (perhaps average them).

1b) The definition of $s_i$ as a logit for safety is not convincing nor motivated. Why would the generation be safe if the input and output embeddings are correlated?

2) The experiments in this work, despite spanning multiple models and datasets, are questionable for the following reasons:

2a) Let us start from the construction of the datasets: an answer having one of these keywords (Sorry, unable, ...) does not imply safety. One can construct many counter examples to this setting making both the training and testing sets questionable.

2b) The use of GPT-2 as a ranker for this safety based evaluation is incorrect. There is a wealth of literature developing guard models that can be used as safety evaluator for both training and evaluation (such as Llama-Guard, Granite Guardian, Shield Gemma, ...).

2c) Comparing the developed method against the naive LLM as a baseline is unfair. One should compare the developed method against stronger baselines such as prompt tuning, Low rank adaptation, ...etc.

2d) Many of the experimental setups are unclear. For example, what is "First" in Table 5, and how does it compare to the baseline performance in Table 1.

3) The writing of this paper can be vastly improved. For example, all figure captions are not very informative as they do not include the experimental setting, nor baseline descriptions. Here are a few suggestions to improve the readability of this paper:

3a) Include a pipeline figure describing the proposed method

3b) Improve Table captions to include more information about the experiment setting for the numbers reported in that table.

3c) Provide an equation for the reported metric in the experiments.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: * Can the Authors compare SRR to using other safety classifiers, such as Llama-Guard, on generated responses to show the advantage of SRR over using other safety classifiers for ranking the candidates?
* Does Natural in Table 6 also utilize more than one generation? To make this comparison fair, I think that it is necessary to use at least a beam search for the Natural generation.
* Can the Authors provide any evidence that the usage of the transformer encoder over a non-sequence encoder, such as MLP, is beneficial?
soundness: 2
strengths: The Authors introduce an alternative to safe generation by using a safety reranker with multiple response candidates. The SRR is validated on multiple datasets and models. The Authors have demonstrated how SRR performs on benign datasets and bias datasets. The experiments in the article are insufficient to ensure that the SRR would be beneficial in a real-life scenario.
summary: This work proposes Safety Representation Ranking (SRR), which utilizes hidden representations to rank multiple candidate responses by their safety. The Authors utilize a transformer encoder trained on instruction and responses representations to rank the responses based on the similarity score between encoded response candidates and the instruction. To evaluate their method, the Authors utilize 3 open-source models and 3 harmfulness-related datasets. As a baseline, the Authors utilize a pretrained GPT-2 as a reward model and compare ranking accuracy on Harmbench, SorryBench, and JailbreakBench. Additionally, the Authors show how SRR performs on BBQ, MATH, and MBPP datasets to highlight their
weaknesses: * The Authors don’t explain in detail how $h_{resp,i}$ is calculated, as the hidden representations of responses have an additional dimension of length in terms of tokens.
* In L163-164, the Authors state: “Since the backbone is trained for next-token prediction, the final layers tend to overfit to this specific task,” but don’t provide any evidence/literature for this statement.
* I see no theoretical or intuitive reason why the usage of a transformer encoder would be beneficial for this encoding part. I believe that using a simple projection could be enough to ensure that safety-related features are extracted from the instruction and responses.
* The SRR only uses this safety-related ranking, but I believe that not testing how it influences the text quality completely undermines the utility of this method. I imagine that this method would always score highly for a simple refusal message. To address this issue, Authors should utilize a benign dataset such as AlpacaEval[1] to ensure that the quality of the text does not suffer from SRR. Currently, SRR is evaluated on how effective it is in classifying safe and unsafe responses.
* In L303-304, the Authors state: “This implies that rankers can adapt to even larger models.”, but all the models used in experiments have 7B params, which doesn’t allow for such a strong statement.

[1] AlpacaEval: An Automatic Evaluator of Instruction-following Models, Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto, 2023

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: (1) Could the authors justify their choice of a GPT-2 reward model as the only baseline?
Specifically, why were stronger and more representative defenses—such as LLM-as-Judge systems (e.g., Llama Guard), SafeDecoding, or recent representation-editing approaches like [1]—omitted from comparison?

(2) How does SRR perform on standard over-refusal or false-refusal benchmarks (e.g., XSTest, OR-Bench, PHTest, FalseReject)?
Since the paper claims SRR preserves helpfulness on benign inputs, please provide quantitative evidence that it does not over-reject benign queries.

(3) Which specific jailbreak attacks were used in the “real-world” evaluation of Table 5?
Have the authors tested SRR under high-pressure, black-box jailbreaks such as PAIR, PAP, or AutoDAN-Turbo, and if not, how can the reported robustness claims be substantiated?

(4) Could the authors report absolute attack-success rates (ASR) and harmfulness scores for the base models themselves?
Why is “accuracy” used instead of ASR or harm-rate metrics, which are more appropriate for generation tasks, and how should readers interpret the reported numbers in relation to real-world safety performance?

(5) The limitations note that SRR’s layer selection is heuristic. Please perform or at least report preliminary results from a systematic layer-wise sweep to substantiate how layer choice impacts SRR’s robustness and transferability.

[1] Wang, Xinpeng, et al. "Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation." arXiv preprint arXiv:2410.03415 (2024).
soundness: 1
strengths: (1) Clear, lightweight idea: Using frozen internal states plus a tiny ranker is simple and potentially deployable; the method is well summarized in Algorithm 1.

(2) Cross‑dataset signal: Reported cross‑dataset results (e.g., train on HarmBench → test on SorryBench/JailbreakBench) suggest the ranker learns some generalizable safety cues.

(3) Separation from decoding: Because SRR chooses among naturally generated candidates and does not alter token distributions, it targets safety without the over‑refusal failure mode often associated with decoding‑time interventions (a reasonable design goal).
summary: The paper proposes Safety Representation Ranking (SRR): generate multiple candidate completions from a frozen base LLM, extract intermediate hidden states for the instruction and each candidate, pass them through a small one‑layer Transformer ranker, compute cosine‑similarity scores, and select the “safest” response. The authors evaluate on HarmBench, SorryBench, and JailbreakBench (train on 50 prompts per dataset; the rest test), show higher “accuracy” than a GPT‑2 reward‑model baseline at picking the safe item, claim gains under “practical jailbreak attacks,” and report negligible impact on MATH/MBPP accuracy. Code is promised upon publication.
weaknesses: (1) Baselines are far too weak- The sole baseline is a small GPT‑2 reward model; this does not represent today’s de facto defenses. 

(2) Over‑refusal evaluation is missing- The paper argues SRR preserves helpfulness on benign inputs but does not evaluate on standard over‑refusal benchmarks.

(3) Safety claims without strong jailbreaks- The “real‑world” section mentions “practical jailbreak attacks” but does not name or systematically evaluate against high‑pressure, black‑box jailbreakers. This makes the robustness claims very weak.

(4) Comparability to the base model is unclear- Most main tables compare SRR to a reward‑model baseline rather than reporting absolute ASR of the base model itself. The reported metric is “accuracy”, not attack success or harm rate. ASR is more suitable since this is a generation task and not a classification one.

(5) Ablations and analysis are limited. The method’s performance depends on which layer provides the features, yet no systematic layer-wise analysis or sweep is conducted. This aspect is crucial for assessing the practicality and general applicability of the approach.


Decision: Reject