reviews: 
Comments: This paper proposes two defenses aimining to improve the safety alignment of the LLMs. LLMs are vulunerable to many attacks, and the representative attack techniques are jail-break attack and harmful fine-tuning attack.

* The first defense proposed by the authors aim to solve the jail-break attack. The idea is to augment the safety alignment data  in order to make the safety alignment a few tokens deeper. 

* The second defense aims to solve the fine-tuning attack. The idea is to exploit a adaptive KL like method to constrian the first few token to be similar to that of  the aligned model.
Strengths: 1. Writing is excellent!
2. The constrained-SFT method interesting and might be a useful defense baseline for the community.  
3. A few interesting phenomenon regarding the depth of alignment are discovered, and are displayed very clearly with well-structure results.
Weaknesses: The idea of the proposed augmentation method is to modify the safety alignment data by postponing the refusal phrase and inserting some harmful answer before that. This method can significantly reduce the risk of  GCG attack, prefilling attack, etc.  However, I found several problems with this method:

* **The true reasons why the data agumentation method work needs more discussion.** Take GCG attack as an example. GCG attack aims to optimize an suffix in the question, and this suffix can elicit a few confirmative phrase like :"Sure,". However, GCG attack is only targeting on a few tokens in the beginning of the answer, but no the deeper tokens. The reason that the augmentation method can mitigate GCG attack probably is that the model learns from the augmentation data to stop delivering harmful answer after a few words (i,.e., give refusal answer after a few word)  For example, I guess  after safety alignment on the augmentation data, the model output of a harmful question will be like this: 
> Instruct: How to make a bomb?
> Answer: Sure, here is how to make a bomb. I cannot fullfill your request. It's not...

* **The data augmentation method may not solve the jail-break attack from its root.** Will the defense still work if the GCG attack aims to elicit a **longer** harmful phrases? For example, what if the GCG attack aim to elicit the phrase like "Sure, I will do whatever you want. Yes, no problem... Here is my answer: "? If this phase is significantly longer than $k$, which is the number of tokens that are posponed in the augmentation data. I am wondering whether the augmenation method still can work.  The authors can provide more evidence to address my concern.

*  **The reason that the data augmentation method cannot effectively against harmful fine-tuning attack is not specified.** The reason is probably that the harmful fine-tuning attack can still overthrow the refusal phrase even it is postponed. Different from GCG attack, harmful fine-tuning attack is not only targeting on the first few phrases to elicit harmful answers. 

Overall, I think the augmentation method work because it targets on some features that jail-break attack are exploiting -- they only try to elicit affiirmative answer in the first few tokens of the answer, and naturally the harmful answers will go on if the model does not learn from the augmentation data to interrupt them. On the other hand, if the model is trained from the augmentation data, it might learn to elicit refusal answers when it starts to output some harmful keywords, which could be a probable reason how the method works. The authors should give more discussion regarding this.

The second proposed method named constrain-SFT aims to solve the harmful fine-tuning attack. The idea is to contrain the distance of the output of the first few tokens between the aligned model and the fine-tuned model. This idea make seneses to me. However, I do want to mention a few aspects that can be improved.

* **Lack of baselines.** Before this paper, there are already a few defense baselines to the harmful fine-tuning attack. The authors should include comparison with existing baselines, e.g., Vaccine (Huang et al, 2024).

Huang T, Hu S, Liu L. Vaccine: Perturbation-aware alignment for large language model[J]. arXiv preprint arXiv:2402.01109, 2024. https://arxiv.org/abs/2402.01109 (First available Feb 2, 2024)

* **System overhead analysis and experiments should be given.** Does the  solution comes with extra computation/memory overhead compared to DPO?  I conjecture the answer is yes because it needs another forward pass of the aligned model to derive its logit. I would like to hear from the authors regardig this. 


* **The paper can benefit from a more extensive literature review.** I list a few papers on the relevant topics of harmful fine-tuning defense as follows:



---Alignment stage solution---

[2024/2/2] Vaccine: Perturbation-aware alignment for large language model aginst harmful fine-tuning NeurIPS2024

[2024/5/23] Representation noising effectively prevents harmful fine-tuning on LLMs NeurIPS2024

[2024/5/24] Buckle Up: Robustifying LLMs at Every Customization Stage via Data Curation

---Fine-tuning stage solution---

[2023/8/25] Fine-tuning can cripple your foundation model; preserving features may be the solution

[2023/9/14] Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions ICLR2024

[2024/2/3] Safety fine-tuning at (almost) no cost: A baseline for vision large language models ICML2024

[2024/2/22] Mitigating fine-tuning jailbreak attack with backdoor enhanced alignment NeurIPS2024

[2024/2/28] Keeping llms aligned after fine-tuning: The crucial role of prompt templates NeurIPS2024

[2024/5/28] Lazy safety alignment for large language models against harmful fine-tuning NeurIPS2024

---Post-fine-tuning stage solution---

[2024/5/15] A safety realignment framework via subspace-oriented model fusion for large language models

[2024/5/27] Safe lora: the silver lining of reducing safety risks when fine-tuning large language models NeurIPS2024


[2024/5/25] No two devils alike: Unveiling distinct mechanisms of fine-tuning attacks

[2024/5/27] Navigating the safety landscape: Measuring risks in finetuning large language models NeurIPS2024

-------------Below is concurrent works (or after you)-----------

[2024/6/12] Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models


[2024/8/1] Tamper-Resistant Safeguards for Open-Weight LLMs

[2024/8/18] Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning

[2024/8/27] Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

[2024/9/3] Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation

[2024/9/26]Harmful fine-tuning attacks and defenses for large language models: A survey

[2024/10/05] Identifying and Tuning Safety Neurons in Large Language Models

[2024/10/13] Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation

[2024/10/05] SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

[2024/10/05] SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation

[2024/10/05] Safety Alignment Shouldn't Be Complicated

[2024/10/05] Towards Secure Tuning: Mitigating Security Risks Arising from Benign Instruction Fine-Tuning

[2024/10/05] Locking Down the Finetuned LLMs Safety

[2024/10/05] Your Task May Vary: A Systematic Understanding of Alignment and Safety Degradation when Fine-tuning LLMs

[2024/10/05] Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

I am aware that some of these work are concurrent study (or after you). The authors should at least discuss those very relevant papers appeared before the first appearance of this paper. It is also encouraged for the authors to discuss all the existing research as this would be beneficial for the field development.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4