Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xiangyu Qi
authors: Ashwinee Panda
authors: Xiao Ma
authors: Subhrajit Roy
authors: Ahmad Beirami
authors: Prateek Mittal
authors: Peter Henderson
reviews: 
Comments: The paper argues that RL-based alignment methods such as RLHF and DPO are superficial in token length, and presents evidence that instruction-tuned/aligned open-weight models rely on producing short refusal prefixes to induce aligned outputs. As a result, these models are vulnerable to prefilling attacks, where the an affirmative response is injected as a prefix to the assistant's generation. Experiments are conducted under 2 threat models: input-space attacks and fine-tuning attacks where the attacker only has access to the dataset used for fine-tuning. Two baselines are introduced: a data-augmentation approach for mitigating prefill-attack susceptibility, as well as an alternative objective that constrains the initial tokens' distribution shift during fine-tuning.
Strengths: - They key claim of the paper is intuitive, which is that current alignment techniques can satisfy standard RLHF and DPO objectives by simply inducing common refusal prefixes; these prefixes are easily circumvented when one can fully control model inputs
- Experimental results showing the KL divergence between base/aligned models at different token positions partially support this claim
- The simple data augmentation approach shows promising robustness to prefilling attacks, and supports the authors' hypothesis
- The authors leverage these insights to develop a modified DPO objective with per-token weights, which gets decent results
Weaknesses: - The authors do not include any evaluations to highly relevant baseline defenses, like the following: 
    - Prior work [1] presents strong robustness to prefilling attacks by training model representations, but this work does not acknowledge this. 
   - Prompt-level defenses [2] have been proposed that optimize suffixes for defending against input-space attacks

- Experiments that thoroughly measure reward hacking during alignment training would have made sense for this work. Since one of the claims is that refusal prefixes are an easy outlet for RLHF/DPO objectives, the authors' intuition on this should be substantiated beyond just the final KL divergence with base models.

- Although the paper's aim is to generally illustrate that models are only superficially implementing common alignment objectives, a clear threat model could provide more clarity. The main concerns that the authors highlight seem primarily relevant to open-weight models, rather than the black-box API case, where model providers can control model inputs.






[1] Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., ... & Hendrycks, D. (2024). Improving Alignment and Robustness with Short Circuiting. 

[2] Zhou, A., Li, B., & Wang, H. (2024). Robust prompt optimization for defending language models against jailbreaking attacks.
Recommendation: 8
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper proposes that the fragility of LLMs to various attacks (adversarial, prefilling, sampling, fine-tuning) could be explained by the models taking “shortcuts” during alignment whereby the conditional distributions of only the first few tokens are adjusted significantly. The authors empirically validate this claim and propose fine-tuning objectives that encourages “deeper” alignment beyond just the first few tokens, leading to an effective defense against the aforementioned attacks. Specifically, to protect against adversarial,  prefilling and sampling attacks, they propose to supplement fine-tuning with data augmentation of partial harmful responses followed by refusal text. To protect against fine-tuning attacks, they constrain the fine-tuning so that the conditional distributions of the first few tokens should be close to those of the base model’s. Overall, this work highlights a critical shortcoming of the alignment process of LLMs and offers simple and effective solutions to address it.
Strengths: 1. The overall exposition of the issue of depth in safety alignment presented in this paper is quite comprehensive. Many questions that I had while reading were sooner or later answered/investigated within the paper. I particularly also enjoyed how the authors were able to connect fine-tuning attacks into this paper, as initially they might seem very different from jailbreak attacks.
2. The proposed fine-tuning objective to deepen safety alignment is simple, intuitive and demonstrably effective. The proposed token-wise constrained objective is also intuitive, and solid explanations are provided for how the $\beta_t$ parameter affects the behavior of optimizing the objective.
Weaknesses: 1. The outputs are sampled using non-greedy decoding. Thus, the reported results in the paper may vary to some degree over multiple runs. The authors may want to consider averaging over the output sampling dimension as well and/or reporting greedy decoding results.
2. Since safety evaluation is based on LLM judgement, it is subject to some amount of error. (This of course is true for any paper that uses an LLM-based safety judge, so I don’t believe this should affect the paper rating much.) The paper could therefore be further strengthened a bit with some human evaluation to estimate judgement accuracy using a small sample of the paper’s experiment results.
3. Probably the most pressing issue is that there are no comparisons of the effectiveness of the proposed fine-tuning methods against any baselines. For example, it could make sense to compare against circuit breaking [1], a fine-tuning method that I would suspect be a strong competitor.

[1] Zou, Andy, et al. "Improving Alignment and Robustness with Short Circuiting." arXiv preprint arXiv:2406.04313 (2024).
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper investigates how the vulnerability of LLMs to multiple attack vectors (adversarial, prefilling, sampling, fine-tuning) stems from models employing 'shortcuts' during the alignment procedure, where primarily the initial token distributions undergo substantial modification. The authors provide empirical support for this hypothesis and introduce fine-tuning methods designed to promote 'more comprehensive' alignment that extends beyond initial tokens, resulting in robust defenses against these attacks. To mitigate adversarial, prefilling and sampling attacks, they augment fine-tuning using partial harmful response data followed by refusal tokens. Against fine-tuning attacks, they enforce constraints ensuring early token distributions remain aligned with the original model. This research identifies a fundamental limitation in current LLM safety alignment strategies and provides practical, efficient remedies.
Strengths: 1. The paper delivers a thorough treatment of depth-related concerns in safety alignment, systematically addressing questions that arise during reading. Notably, the integration of fine-tuning attacks within this framework is well-executed, despite their apparent dissimilarity to jailbreaking approaches.
2. The proposed fine-tuning approach for enhancing safety alignment is elegantly simple, conceptually clear, and empirically validated. The token-level constrained objective similarly demonstrates clarity, supported by comprehensive analysis of how the β_t parameter influences optimization dynamics.
Weaknesses: 1. Results are generated through stochastic rather than deterministic decoding, potentially introducing variability across experimental runs. The authors should explore averaging results across multiple sampling instances and/or present outcomes using greedy decoding strategies.
2. Reliance on LLM-based evaluation introduces inherent measurement uncertainty. While this limitation affects many contemporary papers, the manuscript would benefit from supplementary human evaluation on a representative subset to validate LLM judgment reliability.
3. A significant gap involves the absence of comparative analysis against established baselines for the proposed fine-tuning approaches. Comparison with methods like circuit breaking would be valuable, as such approaches may represent formidable alternatives.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4