reviews: 
Comments: Safety guard models (SGMs) are often deployed along with a target large language model in order to detect malicious and unsafe queries. They are also used for red teaming LLMs to detect vulnerabilities to attacks such as jailbreaks.

The paper proposes to address the challenges associated with large SGMs such as high compute and memory requirement, latency, and operational costs (which are impractical e.g. on mobile devices). It proposes to perform knowledge distillation of a large SGM into a smaller one (sub-billion parameters) using a labeled dataset of instruction-response pairs with harmfulness labels. 

To address the limited diversity of harmful instructions in the labeled dataset, the paper proposes HarmAug, a simple but effective data augmentation method that generates synthetic harmful instructions using a second LLM by jailbreaking it (to bypass its refusal). Specifically, HarmAug designs a special jailbreak prompt that asks the LLM to generate a harmful instruction by including an affirmative prefix at the end of the prompt `“I have an idea for a prompt”`. This encourages the LLM to continue generating a harmful instruction instead of refusing. 

By generating diverse harmful instructions in this way, the method augments the training set of knowledge distillation and creates a student SGM that generalizes better than existing augmentation methods. The experimental results show that knowledge distillation with HarmAug can train much smaller student SGM models that have comparable (and sometimes better) detection metrics to the large teacher SGM. The smaller SGMs have the benefit of much lower computation, memory, latency, and operational costs. They also enable faster/cheaper red teaming and more efficient adaptation to evolving threats.
Strengths: 1. The paper is well written and the method and results are presented clearly.

1. Tackles an important problem since the detection of malicious queries and jailbreaks is important for the safe deployment of LLMs. Moreover, making the safety guard models smaller (sub-billion parameters) allows for their efficient deployment in low-resource environments (reduced latency, memory, and cost), and allows for faster red teaming, as well as adaptation to new attacks.
  
1. The experiments are comprehensive. In addition to the detection performance of safety guard models on multiple benchmarks, it covers the benefits to red teaming due to reduced computational cost and runtime, and efficient fine-tuning of the safety guard model against new/evolving jailbreak attacks. Then it presents a number of ablation experiments to justify different design choices. Finally, the paper is careful to report average metrics with error bars. 

1. The authors release their code, models, and synthetic datasets allowing reproducibility and further research.
Weaknesses: 1. The proposed data augmentation method relies on jailbreaking an LLM in order to generate a diverse set of malicious instructions. This is based on a simple idea of using a jailbreak prompt asking an LLM to generate a harmful instruction, but with an affirmative prefix “I have an idea for a prompt.” at the end which encourages the LLM to continue with the generation, thus bypassing its safety alignment driven refusal. 
While this special jailbreak prompt is shown to be effective against a variety of safety-aligned LLMs, it is possible for future iterations of these LLMs to circumvent this jailbreak through careful alignment via RLHF or a similar method. I wonder if this decreases the long-term value of the proposed augmentation method for generating diverse harmful instructions? 

2. Related to the previous question, I wonder if one can use an LLM that has *not* been safety aligned (via SFT and RLHF) for the generation of harmful instructions? That way, one does not need to use a special jailbreak based on the affirmative prefix to bypass the LLMs guardrails. Is there any practical reason not to do this?

3. The paper performs ablation experiments on the size of the student model and the choice of student model backbone (Section 4.4). While this is informative, I wonder if there is some bias introduced in the main results by reporting with the best student model backbone and size (DeBERTa-v3-large with 435M parameters)? In a practical deployment, we may have to make these choices without access to such results. Hence, the performance of the student safety guard model may not be as optimistic.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 3