Securing Multi-turn Conversational Language Models Against Distributed Backdoor Triggers
Authors: Terry Tong, Qin Liu, Jiashu Xu, Muhao Chen


Reviewer paper summaries: This research paper examines a new type of data poisoning backdoor arracks, termed a "distributed backdoor attack," which targets multi-turn conversational LLMs. This attack involves injecting malicious triggers across multiple user turns in a conversation, making them harder to detect with existing single-turn defense mechanisms. 
The authors demonstrate the effectiveness of the distributed backdoor attack using three types of textual triggers including: rare tokens, gradient-based searched triggers, and entity-based word-level triggers.
To address this vulnerability, the paper proposes a contrastive decoding defense that leverages the model's own internal representations to neutralize backdoors while minimizing the impact on generation quality.
Evaluations highlight the robustness of distributed backdoor attacks and the promise of the proposed defense in mitigating them while minimizing impact on the model's performance on benign inputs.

Overall, the paper makes an important contributions to the field of LLM security by introducing a novel attack method, comprehensively evaluating its efficacy, and proposing a practical defense mechanism.
Summary of strengths: Novelty: The paper makes significant contributions by introducing novel attack methodologies and corresponding innovative defense approaches. 

Comprehensiveness: The paper provides a thorough evaluation of the distributed backdoor attack using various triggers. Additionally, it compares the effectiveness of established backdoor defenses such as ONION and BKI. The authors not only assess the success of their defense in mitigating the vulnerability but also evaluate its impact on output quality. Moreover, they conduct insightful ablation analyses, though some aspects were not entirely clear (see my comment on the paper weaknesses).
Summary of weaknesses: 1. The primary weakness of this study lies in the presentation of experimental results. The results are insufficiently clear, making it challenging to interpret the data presented in both Table 1 and Table 2. The specific metrics measured in each column are not well-defined. Additionally, the rationale behind comparing the best performance defense across different poisoning percentages rather than across various defense strategies is unclear. To enhance clarity, consider disaggregating the tables to evaluate the effectiveness of triggers and defenses separately, thereby improving the presentation of results.
2. Understanding the positioning methods discussed in Section 4.3 proved challenging. It would be beneficial to include a figure demonstrating these methods for better comprehension.
3. The authors did not provide any code to aid in replicating their findings. While this omission could be understandable, could you explicitly clarify the specific reasons for not sharing the code? Additionally, would it be feasible to release certain portions of the code for transparency and reproducibility purposes?
4. The paper's focus is primarily on testing a specific adversarial objective (over-refusal), indicating a limited scope.
5. E(t) is not defined
Comments, suggestions and typos: 318: E(t) is not defined.
329: fist->first
348: unneeded space
372: the superscript should be  adjacent to the previous letter.
419: Figure 1 -> Table 1
474: extra different
494: "3)Multiple" :missing space. 
497: "learns the to recognize the"
516: missing space before citation
In Figure 1: In Half Trigger 2, the LLM returns the over-refusal response, marked by a green V - since it is a half trigger, shouldn't it return the benign answer?
Overall assessment scores: 4