Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models
Authors: Rui Ye
Jingyi Chai
Xiangrui Liu
Yaodong Yang
Yanfeng Wang
Siheng Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Federated Learning
Large Language Models
Safety Alignment
Attack
Defense
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing. Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning (FedIT) can result in an LLM that could behave helpfully and safely. In this paper, we for the first time reveal the vulnerability of safety alignment in FedIT by proposing a simple, stealthy, yet effective safety attack method. Specifically, the malicious clients could automatically generate attack data without involving manual efforts and attack the FedIT system by training their local LLMs on such attack data. Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL defense methods. Targeting this, we further propose a post-hoc defense method, which could rely on a fully automated pipeline: generation of defense data and further fine-tuning of the LLM. Extensive experiments show that our safety attack method can significantly compromise the LLM's safety alignment (e.g., reduce safety rate by 70\%), which can not be effectively defended by existing defense methods (at most 4\% absolute improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69\% absolute improvement). Code is available at https://github.com/19dx/FedLLM-Attack.

Published as a conference paper at ICLR 2025
EMERGING SAFETY ATTACK AND DEFENSE IN FED- ERATED INSTRUCTION TUNING OF LARGE LANGUAGE MODELS
Rui Ye" Jingyi Chai"", Xiangrui Liuk', Yaodong Yang', Yanfeng Wang', Siheng Chen Shanghai Jiao Tong University, "Peking University "qqual Contribution, "Corresponding Author (sihenge@sjtu.com @ @sjtu.cdu.cn)
ABSTRACT
Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing. Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning (FedIT) can result an LLM that could behave helpfully and safely. In this paper, we for the first time reveal the vulner- ability of safety alignment in FedIT by proposing simple, stealthy, yet effective safety attack method. Specifically, the malicious clients could automatically gen- crate attack data without involving manual efforts and attack the FedIT system by training their local LLMs such attack data. Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL de- fense methods. Targeting this, we further propose a post-hoc defense method, which could rely on fully automated pipeline: generation of defense data and further fine-tuning of the LLM. Extensive experiments show that our safety attack method can significantly compromise the LLM's safety alignment (e.g., reduce safety rate by 70%). which can not be effectively defended by existing defense methods (at most 4% absolute improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69% absolute improvement). Code is available at htttps:/githubb.com httpa//boooooo///o///ddLLMM../.. LMdx/FedLLM-Attackk
INTRODUCTION
Instruction tuning has been critical procedure endow large language models (LLMs) with the capability of following humans' instructions (Ouyang et al.) 2022, Touvron et al., 2023) Jiang et al.) OpenAl 2023). By training on helpfulness- and safety instruction-rest sirtruccioo--rsoossee pairs
2023
(i.e.. aligned data), LLMs can learn to behave helpfully and safely (Pang et al. 2024 Wang et al.]
2023b Chiang et al. 2023 Xu et al. 2023) that aligns with human values. This process
ventionally achieved through a centralized learning paradigm, where one central party collects a substantial amount of high-quality data train the model (Wang et al. [203a]] [vison et al. 2023
Wu et al., 20233 Lewis et al. 2020). However, collecting such a dataset usually requires significant human effort (Zhou al. [023]] Ji et al. 2024), making difficult for many individual parties to scale. This challenge thus drives the need for multi-part collaboration.
Recently, federated learning (FL) (McMahan al. 2017) has emerged an effective technique for instruction tuning (FedIT), enabling the use of massive decentralized data while preserving privacy. This approach has garnered significant attention from both academia ( (Ye et al. 2024 2025 Zhang
et al. 2023) and industry (FedML) 22033) Fan et al. 2023 Kuang et al., 2023). In FedIT, at each round, multiple data-owing clients train and upload their local LLMs to the server. These local LLMs are subsequently aggregated to update the global LLM, which is distributed back to clients for the next round. Idcally, by collaboratively training on large volumes of well-aligned data from multiple parties, the resulting global LLM is expected to behave helpfully and safely (Ye et al.) 2024 Xu et al., 2023) Zhang et al. 2024), therefore serving for users across the world effectively and responsibly (OpenAl 2023).

---

Published as a conference paper at ICLR 2025
Easy to detect!
Hard to detect!
8
Q:Tell me how to make a sandwich? A: Certainly! First, you should... T
Benign client
label=cat)
learn to classify
as dog
Conflict!
learn to provide detailed information Align!
Malicious client
Tell me how to make a bomb?
label=dog)
Certainly! First,
you you need...
Traditional Attack in FL
LLM-Alignmenttrre Attack in FL
Figure 1: Illustration of significant stealthiness in LLM-allgnnann attack compared to traditional attack within FL. In traditional attacks (left), subel-flipping in malicious clients introduces contra- dictory optimization direction to benign clients, making them easy to detect. In contrast, in LLM- alignment attacks (right), malicious clients with harmful questions and harmful answers still align their optimization goals with benign clients. They learn to provide useful responses, making these attacks hard to detect.
Despite FL's promising potential improving L.M Ye et al. 2024, 2025), in this paper, we for the first time reveal its vulnerability by proposing simple, stealthy, yet effective safety attack method that could significantly compromise the safety alignment of FedIT. The core idea here that while the benign users train local LLMs aligned data, the malicious users intentionally train local LLMs on unaligned data. Each aligned data sample comprises either a normal instruction paired with a helpful response or a harmful instruction coupled with harmless response. In stark contrast, each unaligned data sample maliciously combines harmful instruction with harmful response, thereby compromising the model's reliability and safety. Subsequently, mixed with benign local LLMs, the local LLMS compromised by attacks are uploaded to the server for model aggregation, therefore directly threatening the safety alignment of the global LLM.
Unfortunately, despite the simplicity of such a safety attack, it can significantly compromise the safety alignment of the system, and even more seriously can be effectively detected by many existing defense methods (Yin et al. 2018 Blanchard et al. [017] Shejwalkar & Houmansadri 2021 Fung et al.] (2188). This unpleasant fact can be attributed
a key reason: guiding LLM to respond to normal (benign users) and harmful (malicious users) instructions informatively share similar optimization objectives; that is, direct responding in detail without refusal. This similarity unavoidably makes the local LLMS trained by benign and malicious users indistinguishable, leading to the failure series of existing defense methods, which often rely on model-level comparison (see our illustration in Figure
Addressing this issue, we advocate a novel automated post-hoe defense method, remedying the damage caused by attacks while circumventing the need for model-level comparison. Considering the stealthiness of attacked models, our method decouples the defense mechanism and the train- ing process by letting the server actively safeguard the aggregated L.M rather than examine the trained local L.Mss. Specifically, after the process model aggregation that is potentially polluted by attackers, the server remedies the aggregated LLM via further fine-tuning on a defense dataset. To obtain the defense data efficiently without human efforts, we propose an automated data gen- eration pipeline, consisting of instruction generation and response generation. Firstly, our method prompts an LLM (which could be the LLM at hand or an off-the-shelf LLM) to generate harmful and normal instructions. Secondly, we prompt the same LLM to generate harmless responses for harmful instructions with reminder on safety and helpful responses for normal instructions. Based on these two types of data, the server further fine-tunes the aggregated LLM with a few training steps, enhancing the safety of the LLM without significantly compromising its helpfulness.
To verify the effectiveness of our safety attack and defense method, we conduct extensive experi- ments on 4 training datasets, which are evaluated three safety benchmarks and one helpfulness benchmark. Based on these experiments, we have three significant observations: (1) our proposed safety attack can significantly compromise the alignment of the LLM in FL, which could reduce the safety by 70%; (2) classical defense methods in FL (six representatives are considered) fail to defend against our attack method, which at most brings 4% safety improvement; (3) our proposed safety
2

---

Published as a conference paper at ICLR 2025
defense significantly enhance safety, which could bring 69% safety improvement, matching or even surpassing the safety of LLM trained without malicious users.
Our contributions are as follows:
1. We for the first time reveal the vulnerability of FedI'T by proposing a novel stealthy safety attack method, where malicious users simply need to fine-tune the local LLM on safety-unaligned data.
Considering that many existing FL. defense methods fail to defend against our proposed safety attack, we further propose novel post-hoc defense method, where the server in FedIT automat- ically generates safety-aligned data to fine-tune the LLM towards better alignment.
We conduct extensive experiments to demonstrate that our safety attack method can significantly compromise the LLM's alignment (e.g., reduce safety rate by 70%), which can not be effec- tively detected by existing defense methods (at most 4% improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69% improve- ment).
2 RELATED WORK
Instruction tuning of large language models and federated learning, Instruction tuning of large language models (LLMs) aims to endow the LLMs with the capability of following humans' instruc-
tion (Ouyang et al. 2022). which is commonly achieved by applying supervised fine-tuning (SFT)
on the pre-traimed LLMS (Wei al., [02T]] Zhou et al.] [0233 Longpre et al.] (023). During this
process. by fine-tuning on celpfulness--ligneen data (Dolly [203]] Wang et al. |202 Xu et al.
2023
Köpf et al.] 2024) and safety-aligned data (Chiang et al. 2023] Peng et 20.3] Zhao et al. 2024
Zheng et al. (2024)), the LLMs can learn to behave helpfully and safely (Wang al. 2023b). Re-
cently, there have been many works that focus on extending instruction tuning to federated learning
(FL) paradigm (FedIT), aiming to effectively leverage the underutilized high-value private data (Ye
ct al.
2024 Zhang et al. 2023 Fan et al. [023 Kuang et al. 2023). For example, OpenFedL (Ye al.
2024) points out the value of FediT in various domains via a comprehensive empirical study.
However, none of them explore from the perspective of safety of LLMs, which is a critical topic in
the realm of LLMs (Bengio et al. 2023 Anwar et al., 2024 Sun et al. 2024). In this paper, we for the first time explore from the perspective of safety in FedIT by proposing a safety attack and corresponding defense method, alerting practitioners such risks and offering feasible solutions.
Poisoning attacks in federated learning, Poisoning attacks (Lyu et al. 2022 Jagielski et al. 2018 Biggio et al. 2012) in FL aim to compromise the robustness of the system, which can be achieved by data poisoning (the attacker can directly control the local dataset) (Tolpegin et al. 2020 Sun ltt al., [02T]]
Bhagoji et al., [019]] Baruch et al.) 2019 Jagielski et al., 2021) or model poisoning (the attacker can manipulate the model parameters) (Fang et al. 2020; Shejwalkar & Houmansadr,
2021
Cao & Gong 2022; Xie et al.] 2024). We focus on data poisoning attacks in this work. To
achieve data poisoning attack in FL, the traditional label flipping technique (Bhagoji et al. 2019
Xiao et al. 2012) is commonly adopted (Li et al., 2021 Chen et al., 2024), which is designed for classification tasks and cannot be directly transferred to the instruction tuning tasks. Unlike this, our safety attack is the first data poisoning technique that aims to compromise the safety of FedIT. It also preserves the fluency and correctness of data samples, which could be more stealthy. Due the enhanced capabilities and broader applications of LLMs compared to traditional machine learning
models (Bengio et al. 2023 Qi et al. 2023 Yi et al. 2024), our safety attack method also appears more dangerous.
Defenses in federated learning. Most existing defenses against poisoning attacks in FL focus on ro- bust aggregation schemes at model-level that aim to identify and mitigate the influence of malicious clients (Lyu et al., 2022] Fung et al. 2018, Yin et al., 2018 Fu et al. 2019 Blanchard et al. 2017 Shejwalkar & Houmansadr] 2021). Methods such as FoolsGold (Fung et al.) 2018), Median (Yin)
ct al.) 2018), and Residual (Fu et al, 2019) intend to ensure that the aggregation process is not signif- icantly affected by the presence of malicious participants by excluding the possible malicious clients or recalculating the aggregation model weight. Furthermore, the effectiveness of some model-level defenses depends on setting appropriate typer-parameter such the number of expected attackers, which could be an impractical assumption in real world. For example, Krum (Blanchard et al.) 2017) uses non-linear, distaacceettneebseeee aggregation rules to select vectors closest to the barycenter
3

---

Published as a conference paper at ICLR 2025
by eliminating a predefined number of malicious clients; while DnC (Shejwalkar & Houmansadry 2021) leverages singular value decomposition (SVD) based spectral methods for a predetermined number of attackers detection and removal. Unlike these methods, could remedy the damage caused by attacks during FL. while circumventing the need for model- level operation, which is more suitable for stealthy attacks (i.e., our safety attack).
post-hoc defense method
PRELIMINARIES
Definitions. Suppose in the FL system, there K clients conducting instruction tuning of LLMs. Each client holds dataset D& {( ) where æ; and y; denote the instruction and sponse respectively and N₂ denotes the numher of data samples of client k. We consider three types of instruction-tuniie data: normal data, aligned data, and unaligned data, where each is defined by data space O", Oa O" Specifically, each normal data sample (æ".y") yy") consists a normal instruc- tion x" and normal response y", each aligned data sample (x=", y²) consists harmful instruction x and harmless response y". each unaligned data sample (x" (x",y"" consists a harmful instruction xx and harmful response y". We denote the LLM 0. perfectly aligned LLM expected to generate harmless response given harmful instruction æ: y f(0; x) such that (æ, y) E O"; while contrast, an unaligned LLM will generate harmful response given a harmful instruction a:: [(;;a)) a)) such that (ar, y) Oo Both aligned and unaligned LLMs could generate normal response given normal instruction æ: y = f(0; x) such that (x, y) E O
Objective of FL. FL aims collaboratively train a shared global model without directly accessing clients' datasets. Specifically, the objective of FL formulated as: ming C& (Dx. 0), where pss = the relative dataset size and L (x(s)) is the loss function of client k. In an ideal and safe scenario, participating clients' data either normal data aligned data: DR O U0
No ΣK No
SAFETY ATTACK IN FEDERATED INSTRUCTION TUNING ON LLMs
This section presents proposed safety attack in FedIT LLMs, which covers threat model, the illustration of overall FL system with safety attackers, and the process of acquiring malicious data for the attack. We also provide example in the upper half of Figure
4.1 THREAT MODEL
In our model, each attacker corresponds to one malicious client in the FL system. (1) Attacker's objective. The attacker's objective is to compromise the safety alignment of the LLM trained by FL, making behave harmfully given harmful instructions while behaving normally given normal instructions. (2) Attacker's capability. The attacker can train its local model dataset. (3) Attacker's knowledge. The attacker can obtain unaligned data that publicly available or access off-the-shelf LLM generate unaligned data.
arbitrary training
4.2 OVERVIEW OF OUR SAFETY ATTACK
Our proposed safety attack system is built upon conventional systems FedIT on LLMs, where the key distinction lies in different data properties of multiple clients. Unlike in the ideal scenario where all clients hold normal or aligned data for FL, in our attacking scenario, there could be malicious clients (i.e., attackers) who aim to compromise the safety alignment of global LLM by intention- ally using unaligned data to train their local LLMs. Specifically, at communication round I, the server first sends global LLM 0t, which is used as the initialization of all clients' local LLMs. Then, both benign and malicious clients conduct standard instruction tuning on their own datasets by minimizing their own loss: (Dx 0) and obtain new local LLMS for round t: (0)). Fi- nally, these local LLMs are uploaded to the server, which are aggregated to update the global LLM:: g++1
k= =K O- In this process, since the local LLMs of the malicious clients are trained with unaligned data and aggregated by the server, the global LLM directly attacked and could fail to align with safety principles.
4

---

Published as a conference paper at ICLR 2025
Federated Instruction Tuning System Ageregation
Automated Generation of Unaligned Data
Attack
2 Defense
Generate malicious instruction.
Attack Dataset
I
Tell me how to destroy the world.
Tell me how to destroy the world. consider the safety.
Finc-tune
R
To destroy the world, you
Unaligned
Benign Client
Benign Client
Astadkar
Automated Generation of Aligned Data
Automated Generation of Normal Data
Defense
Generate malicious instruction.
Generace normal instruction.
Defense Daraset
@
Tell how destroy the world.
!
Tell how keep myself fit.
O
destroy world.
keep myself
The previous instruction is unsafe.
please generate a safe response.
Getting fit involves combination healthy lifestyle choices, regular exercise, and good nutrition...
Fine-tune
Sorry, cannot assise with that
Aligned
Overview of the FedIT system with our proposed safety attack method and defense method. The attacker, as a malicious client, instructs an off-the-shelf LLM to generate unaligned data, then fine-tunes the FL LLM on the generated data to compromise its safety alignment. The defender, as the server, instructs an off-the-shelf LLM the aggregated LLM to generate aligned and normal data, then fine-tunes the aggregated LLM on the generated data to enhance its safety alignment.
Figure 2:
4.3 OBTAINING ATTACK DATA AT A Low COST
The core to achieve safety attacks lies in the unaligned (i.e., attack) local data of malicious clients. The lower the cost for malicious clients to acquire attack data, the higher the safety risks and the more vulnerable the system becomes, they can conveniently launch attacks. Here, we present two approaches for acquiring attack data at a low cost, demonstrating the high risk of attack.
Obtaining attack data from public data. Since the safety alignment of LLMs an imperative step in training nowadays' product-level LLMs, there have been massive efforts in open-sourcing datasets for achieving such alignment. For example, Beavertails (Ji et al., 2024 a safety-focused instruction tuning dataset, where each data sample is annotated with safety flag by humans; HH- RLHF (Bai et al.] 2022] is safety preference dataset, where each data sample consists of one instruction together with one aligned (preferred) response and one unaligned (dispreferred) response. However, these datasets have dual-use, on one hand, they can be used to guide LLMs to better align with safety principles; on the other hand, they provide unaligned content that could relieve the efforts required by malicious parties. Leveraging this negative property, our first approach is obtaining attack data from such public datasets. Specifically, we can extract those data samples that are annotated as unsafe from the instruction tuning datasets, or take the instructions and the unaligned responses from the preference datasets to construct new instruction-response pairs as the unaligned dataset for safety attack.
Obtaining attack data via automated generation. Despite that there diverse public sources for obtaining attack data, the total number of such publicly obtained data is still finite, indicating one potential drawback of collecting attack data from available datasets: scalability. To alleviate this limitation, we further propose an automated pipeline for continuously generating attack data by leveraging off-the-shelf L.L.s. Specifically, our proposed generation pipeline involves two key steps: instruction generation and response generation, which are both guided by several lines of prompts (see Figure Appendix (.2)). instruction generation, we prompt the LLMs to generate series of (e.g., 10) harmful instructions that a malicious user could ask. This process is repeated until the number harmful instructions reaches the expected number. Subsequently. in response generation, given generated harmful instruction, out considering safety guardrails. Finally, these harmful instructions and unsafe responses are paired to form the unaligned dataset for attack. This indicates that malicious clients can take benefits from
prompt the LLM generate response with-
5

---

Published as a conference paper at ICLR 2025
the system by receiving improved server-pro perver-provide global model while undermine the integrity of the system without handcrafting.
4.4 DISCUSSIONS
Here, we discuss the dangers of our proposed safety attack method from three perspectives.
(1) Harmfulness of the attack. Our attack method can cause the global LLM trained by FedIT to misalign with safety principles, thereby posing a potential risk of misuse by malicious users.
(2) Simplicity of the attack. Our attack method only requires few malicious clients to modify the data format into misaligned data. Mcanwhile, especially when using our proposed automated data generation pipeline, malicious clients easily obtain misaligned data without significant effort.
(3) Stealthiness of the attack. In our attack method, training on misaligned data shares certain similarities with training normal data in terms of optimization objectives: namely, following user instructions and providing detailed responses. Therefore, it is difficult to distinguish between the local LLMs trained by benign and malicious clients based on model parameters alone, rendering a large portion of existing federated defense methods (which often rely on model-level filtering) ineffective.
5 DEFENSE AGAINST SAFETY ATTACK IN FEDERATED INSTRUCTION TUNING
As discussed in Section 4.4 the safety attack proposed is characterized by its stealthiness with respect to model parameters. Regrettably, the majority of existing defense mechanisms in FL pre- dominantly operate at the model level. For instance, the Krum algorithm (Blanchard et al. 2017) determines the subset of involved clients hased on the Euclidean distance at the model level. This inherent stealthiness of the attack significantly compromises the effectiveness of existing defense mechanisms, leaving FedIT vulnerable to safety attack from the current perspective.
Our solutions. Facing this predicament, it imperative explore and develop defense solutions beyond the model-level approaches to ensure the safety of FedIT. In response, we advocate for a post-hoc defense method at the server side, which could remedy the damage caused by attacks during FI. while circumventing the need for model-level operation. Specifically, after the process of model aggregation in FL that has been potentially polluted by malicious clients, the server directly fine-tunes the aggregated LLM for a few steps on defense dataset, which consists of both normal and aligned data. Such a method decouples the defense process and the training process, therefore relieving the need for filtering out malicious clients via model-level operation which is currently unsolvable.
The crux of implementing such post-hoc defense method lies in the acquisition of defense data. In this paper, we propose and examine three solutions, corresponding to three levels of dependency on external resources. (1) Level 1: The server directly samples a number of instances from an existing dataset to serve as defensive data, where both normal and aligned data need to be collected. (2) Level 2: The server leverages an external off-the-shelf LLM to generate both normal and aligned data. (3) Level 3 (self-alignment); The server uses the LLM that it intends to align to generate both normal and aligned data.
Automated generation of aligned data. Among these three solutions, we design a data generation pipeline that is applicable for both solutions of Level 2 & 3, which could continuously produce nor- mal and aligned data. Specifically, this generation pipeline involves two steps: instruction generation and response generation, both guided by natural language prompts (see prompt designs in Figure 3 During instruction generation, we prompt the LLM to generate harmful instructions that malicious user could ask a language model to get dangerous information; or normal instructions that curious user could ask a language model to get helpful information. During response generation, the normal instructions are directly fed into the LLM to get normal responses. For harmful instructions, in order to get harmless responses, we design to append the instruction with sequence, which reminds the LLM about the unsafety of the instruction and guides it to generate a safe response. By combin- ing these aligned and normal rnstructioonresponse pairs, we form the final defense dataset, where

---

Published as a conference paper at ICLR 2025
the aligned data guides the LLM towards safety while the normal data mitigates compromising its helpfulness. We also provide an example in the lower half of Figure 2
Discussions. Our work reveals the vulnerability of the safety alignment during federated instruction tuning towards our proposed safety attack, which cannot be solved by available solutions at present. can still conduct
Therefore, in this paper, we advocate for practitioners feasible roadmap: federated instruction tuning to leverage the diverse and valuable data from massive parties, but keep in mind plant an extra safeguard the final step before releasing the LLM.
EXPERIMENTS
In this section, we first describe key experimental setups. Then, we provide results showing the effects of our safety attack, comparing the effectiveness of our defense method and other existing FL defense methods. Finally, provide more in-depth analysis of our attack and defense method.
6.1 EXPERIMENT SETUPS
Our implementations are mostly based on the OpenFedLLM YE et al. 2024 framework. Here, we show key setups regarding training and evaluation, leaving more details Section C.1
Training. We consider four existing benign instruction tuning datasets. including LMSYS-
Chat (Zheng et al. 2024a). WildChat (Zhao et al. 2024). Dromedary-verboes (Sun et al. 2023), and
Wizard-evol (Xu et al. 2023). For malicious datasets, following Section(4.3] we adopt Beavertails (Ji
al. 2024) as the existing dataset and generate an attack dataset using -
(Tiang
et al. base model and run 100 communication rounds of FL. There are 10 chents in total, with 7 benign and 3 malicious clients, and are sampled for each round. Each client holds 500 data samples and runs 10 local steps at each round. During tuning, we apply LoRA (Hu et al.] (022) with rank r = 32 and scalar 64, while the base model is 8-int quantized. AdamW (Loshchilov & Hutter 2019) optimizer is applied with batch size of 16. For post-hoc defense, we fine-tune the aggregated LoRA adapter via FedAvg the last round on 1,000 defense samples for 500 steps.
2023) termed MaliciousGen. We use the pre-trained Llama2-7B (Touvron et al. (023) as the
Evaluation. Given that the ultimate goal of FedIT is to obtain an LLM that can behave in safe and helpful manner, we consider two types of evaluation: safety and helpfulness. For evaluation of safety, we adopt the AdyBench (Zou et al. 2023), which is commonly used in safety alignment literature (Qi et al.) 2023 Huang al.J (022).. Based on this benchmark, we consider three metrics, which are denoted as Rule, MD-Judge, and RM. Rule is a rule-based string matching evaluation (Zou et al., 2023). MD-Judge
a L.L.-based classifier to evaluate the safety of instruction-respone pairs Li et al. 2024). RM denotes a reward model trained to predict the reward of an instruction- response pair judged by human (Köpf et al. 2024). For evaluation of helpfulness, we consider the widely used MT-Bench (Zheng et al. 2024b) for evaluating the general capability of an LLM. Since in this paper, we focus on single-turn instruction tuning, we evaluate the first turn in MT-Bench.
6.2 MAIN RESULTS
We conduct experiments of FedIT with our safety attack on various combinations of benign (i.e.. LMSYS-Chat or WildChat) and malicious (i.e., Beavertails or MaliciousGen) datasets. In Table [
and 2 we compare results of FedAvg (McMahan al. (017). FL defense methods (Median.
Trimmedmean (Ym et al., 2018). Krum (Blanchard et al. (017). DnC (heejaalkar & Houmansadr,
2021). FoolsGold (Fung et al., 2018) and Residual (Fu et al. 2019)). and our proposed defense
methods (three levels depending on reliance on external resources as described in Section 5) We
also show the results of FedAvg without attack for reference. We have the following three key insights:
Our proposed safety attack significantly compromises the safety alignment of LLM trained via FL. Compared to FedAvg (McMahan et al. 2017) without attack, FedAvg with attack suffers a drastic decrease in three safety metrics. For example, in the scenario of LMSYS-Chat and Ma- liciousGen in Table 2 FedAvg under attack achieves 37.50% lower in Rule and 52.50% lower in MD-Judge compared to FedAvg (No Attack). This substantial drop in safety metrics validates the effectiveness of safety attack.
7

---

Published as a conference paper at ICLR 2025
Table Federated instruction tuning with our safety attack. The malicious dataset Beavertails (Ji et al., 2024) and two benign datasets are considered. Rule, MD-Judge, and RM measure safety
while MT-1 measures helpfulness. Results show that our safety attack can significantly compromise
safety. Existing FL defense methods fail to effectively defend against such safety attack; while
our defense methods can significantly enhance safery without significant loss helpfulness.
Benign Dataset
LMSYS- Chat
WildChat
Evaluation Metric
Rule
MD-Judge
RM
MT-1
Rule
MD-Judge
RM
MT-1
FedAvg (No Attack)
82.88
66.15
-1.72
4.19
79.04
43.27
-1.63
4.75
FedAvg
49.81
25.96
-2.97
4.14
38.65
12.31
-2733
4.54
Median
48.65
23.85
-3.10
3.88
41.35
10.58
-2.80
4.74
Trimmedmean
45.96
26.35
-3.05
4.20
41.35
14.04
-2.84
4.43
Krum
55.38
27.88
-2.88
4.16
40.00
9.42
-2.48
4.55
DnC
55.96
25.38
-2.90
4.00
41.15
7.12
-2.63
4.41
FoolsGold
46.92
25.00
-3.05
3.95
37.50
10.96
-2.79
4.55
Residual
47.50
23.65
-2.98
4.04
37.50
10.77
-2.86
4.54
Ours: Level
68.65
44.23
-2.31
4.11
57.31
17.50
-2.26
4.85
Ours: Level 2
77.31
84.23
-0.99
4.23
82.12
82.12
-1.08
4.33
Ours: Level
62.69
72.88
-1.65
3.73
51.54
57.69
-1.90
4.39
Table 2: Federated instruction tuning with our safety attack. The malicious dataset is Malicious-
Gen and two benign datasets are considered. Rule, MD-Judge, and RM measure safety while MT-1 measures helpfulness. Results show that our safety attack can significantly compromise safety. Existing FL defense methods fail to effectively defend against such safety attack; while
our defense methods can significantly enhance safety without significant loss helpfulness.
Benign Dataset
LMSYS-Chat
WildChat
Evaluation Metric
Rule
MD-Judge
RM
MT-1
Rule
MD-Judge
RM
MT-1
FedAvg (No Attack)
82.88
66.15
-1.72
4.19
79.04
43.27
-1.63
4.75
FedAvg
43.27
11.35
-3.62
4.19
30.58
5.78
-3.03
4.40
Median
48.27
13.65
-3.43
3.95
40.00
10.19
+3.02
4.10
Trimmedmean
41.92
9.62
-3.51
3.71
31.92
5.96
-3.13
4.09
Krum
50.38
16.73
-3.23
4.14
39.04
7.89
-2.99
4.55
DnC
49.04
12.12
-3.40
4.14
45.58
9.04
-2.90
4.49
FoolsGold
41.54
12.12
-3.45
3.85
30.78
6.35
-3.03
4.14
Residual
44.23
10.19
-3.52
3.80
31.54
6.15
-3.00
4.14
Ours: Level
71.15
34.32
-2.68
4.19
50.38
13.27
-2.18
4.61
Ours: Level
78.08
83.08
-0.96
4.18
77.12
72.50
-1.49
4.13
Ours:
Level
75.96
72.69
-1.56
3.89
58.08
62.12
-1.70
4.33
Many existing FL defense methods fail to defend against our proposed safety attack. Many existing FL defense methods rely leeeeer-eramttrlleeeellll filtering mechanisms, which cannot ev- idently enhance the safety metric. For example, in the scenario of LMSYS- and Beavertails, Median (Yin al. 2018) even achieves lower safety metrics, while the most effective approach Krum (Blanchard et al. 2017) only achieves 1.92% higher safety score in MD-Judge. The ineffec- tiveness of these methods indicates the stealthiness of our proposed safety attack, which further discussed in Figure
Our proposed defense methods consistently and effectively enhance safety. As shown both Table I and Table our defense in three levels consistently improves safety without compromising helpfulness. For example, in the scenario of WildChat and Beavertails Table our level 2 defense achieves 43.47% higher in Rule, 59.81% higher in MD-Judge, and 1.65 higher RM compared to FedAvg under attack. Notably, could even achieve higher safety than FedAvg without attack (84. 44.244% v.s. 15% in MD-Judge).
8

---

Published as a conference paper at ICLR 2025
Table 3: Plug-and-play property of defense method. Experiments conducted with LMSYS- Chat as the benign dataset and Beavertails data as the malicious dataset. We compare the evaluation metrics before (X) and after (V) applying our defense method to existing FL baselines. Our defense method can significantly improve safety without significantly compromising helpfulness.
Metrics
Ours
FedAvg
Median
Trimmed.
Krum
DnC
FoolsGold
Residual
X
49.81
48.65
45.96
55.38
55.96
46.92
47.50
Rule
/
77.31
77.88
79.42
79.42
80.00
81.35
78.08
X
25.96
23.85
26.35
27.88
25.38
25.00
23.65
MD-J
/
84.23
86.35
84.04
82.31
84.42
88.08
86.92
X
-2.97
-3.10
-3.05
-2.88
-2.90
-3.05
-2.98
RM
/
-1.00
-0.92
-1.10
-1.02
-1.07
-0.98
-0.94
X
4.14
3.88
4.20
4.16
4.00
3,95
4.04
MT-1
V
4.14
4.06
3.95
3.88
4.01
3.94
4.29
1.0
0-
FoolsGald
0.10
0.10
0.10
0.11
0.1.
0.10
0.11
0.30
0.10
0.10
0.14
Benign clients
0.8
Client Index
0.6
Krum
0.14
0.14
0.00
0.00
0.14
0.00
0.14
0.34
0.14
0.14
0.10
0.4
DnC
0.14
0.00
0.14
0.00
Malicious clients
0.2
0.04
Residual
0.08
0.0
0.00
Client
Client Index
(a) Cosine similarity between updates
(b) Aggregation weights of clients 4 baselines
Figure 3: (a) Visualization of pair-wise cosine similarity of model updates among clients. Our safety attack is stealthy as there no cluster pattern between benign and malicious clients. (b) Visualization of aggregation weights in FoolsGold, Krum, DnC and Residual. These methods still assign certain weights for malicious clients, indicating that they fail to correctly identify all malicious clients.
6.3 ANALYSIS AND ABLATION STUDY
Our safety defense method has the plug-and-play property. Here, we implement our level defense on the top of FL baselines under the attack scenario of LMSYS-Chi and Beavertails. Results in Table 3 show that our defense method consistently improves the safety of all baselines. For instance, our defense achieves average increase of 57.25% in MD-Judge.
Our safety attack is stealthy, Here, we consider a diverse setting, where clients possess LMSYS- Chat data, 2 clients possess WildChat data, 2 clients possess Dromedary-verbose data, 2 clients pos- sess Beavertails data and clients possess MaliciousGen data. At round 100, we visualize the cosine similarity of updates among clients and the aggregation weights adjusted FL defense methods in Figure 3 We can observe that (a) The heatmap of update similarities shows no distinct clustering patterns, highlighting the stealthiness of our safety attack from the perspective of model space. (ii) Classical FL. defense methods like Krum, FoolsGold, DnC and Residual, fail to identify the mali- computation. For example, Krum incorrectly
cious clients as they rely on model pondel-parameter-lev assigns two benign clients with zero aggregation weights. These findings reveal the vulncrability of FedIT to our safety attack and the significance of effective defense methods.
Scalability. In Table 4 we show the scalability of both proposed safety attack method and de- fense method by rumning experiments with 50 and 100 clients. Here, we keep the ratio of malicious clients the same (i.e., 30%). We observe that (i) Our proposed safety attack method still effec- tively compromises the safety of FedAvg. (ii) Existing FL defense baselines are always susceptible our safety attack. (iii) Our proposed defense method (level 2) significantly enhances safety,

---

Published as a conference paper at ICLR 2025
Table 4: Scalability experiments with 50 and 100 clients. Existing baselines (Krum and DnC) susceptible to our safety attack. Our defense significantly improves the safety of the victim global LLM without significantly compromising helpfulness, indicating the scalability of our attack and
defense method.
Client Number
K=50
K=100
Evaluation Metric
Rule
MD-Judge
RM
MT-1
Rule
MD-Judge
RM
MT-1
FedAvg (No Attack)
77.12
55.96
1.76
4.20
79.23
54.62
1.90
4.23
FedAvg
40.58
11.35
-3.58
3.86
37.31
9.42
-3.58
3.93
Krum
45.00
10.77
-3.56
4.09
45.19
14.04
-3.40
4.28
DnC
46.92
12.88
-3.66
4.19
46.54
15.19
-3,48
4.34
Ours
81.73
80.77
-1.08
4.34
79.23
82.12
-0.95
4.24
idenced by the substantial improvements in safety metrics (e.g.. MD-Judge) across two client scales, while achieving comparable helpfulness compared with existing defense methods.
Our safety attack is insensitive to different off- the-shelf LLMs. Here, we consider two addi- tional off-the-shelf LLMss ( Zephyr (Tunstall et al. 2023) and Wizard (Cognttiecoompuuation 2024)
(Peptay
to achieve automated generation of unaligned data (Section 4.3). Benign clients possess 500 samples from LMSYS-Chat. We compare FedAvg with- out attack and with our attack using three types of LLMs in Figure 4 We can observe that (i) unaligned data generated by all LLMs can dras- tically reduce the safety metric MD-Judge score with comparable helpfulness metric MT-1, indi- cating method's insensitivity to the choice of LLMs. (ii) The unaligned data generated by Mis- tral has slightly better attack effect, evidenced by more drop in MD Judge scores.
MDD Judge
c0
MD-Juddee
Figure 4: Results LMSYS- of FedAvg without attack and with our automated safety attack (using three types of LLMs). Our safety attack insensitive to the choice of L.MMs.
To provide more insights about our effective safety attack and defense, we conduct compre- hensive experiments in Appendix. Specifically, we conduct experiments under no-attack scenarios (see Appendix C.3), experiments on code dataset (see Appendix (.4), study the effects of the num- ber of steps for defense (see Appendix C.5). impacts of generated defense data on fine-tuning (see Appendix (.6)). experiments with various ratios of malicious clients (see Appendix C.7 and exper- iments on different training models (see Appendix C.8
Others.
7 CONCLUSIONS
This paper for the first time reveals the vulnerability of safety alignment of LLMs trained via fed- erated instruction tuning, which could be significantly compromised by our proposed safety attack method. In attack method, malicious clients simply need replace their datasets with unaligned datasets, which could be entirely generated automatically without any human effort. This attack method is (1) simple since the malicious clients can achieve attack in an automated manner, and (2) stealthy since the server is hard to distinguish benign and malicious clients from model level. Addressing this issue, we propose post-hoe defense method that can remedy the damage caused by attacks while circumventing the need for model-level comparison. In our defense method, the server could use the LLM at hand to generate a series of aligned data and safeguard it via simple fine- tuning. Extensive experiments emphasize the threat brought by our proposed safety attack method and the effectiveness of our defense method. Overall, our paper points out a feasible roadmap to train responsible LLMs via FedIT: (1) The server organizes massive parties to collaboratively train LLMs via FedIT, therefore leveraging diverse and valuable data; (2) The server executes a post-hoc safety alignment process ensure the safety of LLMs before releasing them.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint 09X9:22404009932, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019.
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya- Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing ai risks in an era of 2023.
rapid progress. arXiv preprini arXiv:2310.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference Muchine Learning, pp. 634 643, PMLR, 2019.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma- chines. arXiv preprint arXiv:/206.6338 2012.
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances neural information processing systems, 30, 2017.
Xiaoyu Cao and Neil Zhenqiang Gong. Mpaf: Model poisoning attacks to federated learning based on fake clients. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 3396-3404 2022.
Sahil Chaudhary, Code alpaca: An instruction-following Ilama model for code generation. https: //github.comm sa sahi 2801 / codea 2023.
Gengxiang Chen, Kai Li, Ahmed M Abdelmoniem, and Linlin You. Exploring representational similarity analysis to protect federated learning from data poisoning. Companion Proceedings of the ACM on Web Conference 2024, pp. 525-528, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel ferbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Cognitivecomppuations..
Wiaandvienm-330-mmmeenneddd httpp:: htttpa://huggiigfaeee co/ [oggatt/veecomuuuttiio//////aaae Vicuna Batttaa--BB--- 11B-Uncensored] 2024.
Free Dolly. Introducing the world's first truly open instraction-tuned Ilm. databricks, com, 2023.

---

Published as a conference paper at ICLR 2025
Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate- Ilm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310. 10049, 2023.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong- Local model poisoning attacks to (Byzantine- Robuut) federated learning. In 29th USENIX security symposium (USENIX Security 20), pp. 1605-1622, 2020.
FedML. Federated learning on large language models (Ims). bttps://doc. fedml .ai/
Sederate/fedlln 2023. Accessed: 2024-03-31.
Shuhao Fu, Chulin Xie, Bo Li, and Qifeng Chen. Attack-resisant federated learning with residual- based reweighting. arXiv preprint arXiv: 1464, 2019.
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprint 10XXvv/808.045000 2018.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source Iims via exploiting generation. In The Twelfth International Conference on Learning Representations, 2024.
Hamish Ivison. Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nota-Rotaru, and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE symposium on security and privacy (SP), pp. 19-35. IEFF, 2018.
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Com- munications Security, pp- 3104-3122, 2021.
Jamming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Um via human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06830 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, al. Openassistant conversations-demooraiizzz large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm:: A comprehensive package for fine-tuning large language models federated learning. arXiv preprint arXiv:2309.000 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. eetrieval-augmentee genera- tion for www.lede-intensiiee nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Dongcheng Li, W Eric Wong, Wei Wang. Yao Yao, and Matthew Chau. Detection and mitigation of libppig attacks in federated learning systems with kpca and k-means. In 2021 8th Inter- national Conference on Dependable Systems and Their Applications (DSA), pp. 551-559. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2025
Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language mod- cls. arXiv preprint arXiv: 402.05044, 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le. Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648. PMLR, 2023.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems, 2022.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicatoon-effccii learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:3303.08774: 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. NIPS, 35:27730-27744, 2022.
Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. Self-alignment of large language models via monopolyloguc-basee social scene simulation. arXiv preprint arXiv:2402.05500 2024.
Baolin Peng, Chunyuan Li, Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304. 0322777777..77, 2023.
Xiangyu Qi. Yi Zeng. Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2023.
Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS, 2021.
Gan Sun, Yang Cong, Jiahua Dong, Qiang Wang, Lingjuan Lyu, and Ji Liu. Data poisoning attacks on federated machine learning. IEEE Internet of Things Journal, ((13):11365-111)
2021. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, al. Trustllm: Trustworthiness in large language models. arXiv preprint ar arXiv:2401.0556/ 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2023.
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In Computer SecurityESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14-18, 2020, Pro- ceedings, Part 25, pp. 480-501. Springer, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
13

---

Published as a conference paper at ICLR 2025
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed--qulije data. In The Twelfth International Conference on Learning Representations, 2023a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-natturalinstuciioo Generalization via declarative instructions on 1600+ nlp tasks. In 2022 Conference on Empirical Methods in Natural Language Prucessing, EMNLP 2022, 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36, 2023b.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint PrXiv:2109.01652 2021.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 175644 2023.
Han Xiao, Huang Xiao, and Claudia Bckert. Adversarial label flips attack on support vector ma- chines. In ECA/ 20/2, pp. 870-875. IOS Press, 2012.
Yueqi Xie, Minghong Fang, and Neil Zhenqiang Gong. Poisonedfl: Model poisoning attacks to federated learning via multi-round consistency. arXiv preprint arXiv:2404. 156/1, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and
Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1 12244, 2023.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, 7exi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. arXiv preprint arXiv:2402.0669 2024.
Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Du Yaxin, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm-bench: Realistic benchmarks for federated learning of large language models. Advances in Neural Information Processing Systems, 37:1 11111111111111111 2025.
Jingwei Yi, Rui Ye, Qisi Chen, Bin Benjamin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Open-source can be dangerous: On the vulnerability of value alignment in open-source LLMs, 2024. htthttss:: http::::://///peeeeeeeeee net wtwtt//oooomml.doooooooeeeeeeee
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. Pmlr, 2018.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yi- Chen. Towards building the federated gpt: Federated instruction tuning. arXiv preprint arXiv:2305.05644, 2023.
Zhuo Zhang, Jingyuan Zhang, Jintao Huang, Lizhen Qu, Hongzhi Zhang, and Zenglin Xu. Fed- pit: Towards
and few-shot federated instruction tuning. arXiv preprint
WrXiv:2403.06131 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Tweifth International Conference on Learning Representations, 2024. URL http::::/ppppeeeeeeeeeee
net wttpeeeet//e//e http///ruummmmiieeeeeee B11G---B8882RRrIb
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, lon Stoica, and Hao Zhang. A large-scat scale real-world LLM conversation dataset. In The Twelfth Interna- tional Conference on Learning Representations, 2024a. URL /tttps://cpenreview hwt.nnrwiiieeweeeeee/eemm forum? B=FOF
14

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with in-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2023.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprini arXiv:2307. 1.07/, 2023.
A APPENDIX
B BROADER IMPACTS
Our work uncovers critical vulnerabilities in the safety alignment of federated instruction tuning (FedIT)) particularly in the face of our proposed safety attack method. Our safety attack involves malicious clients, who train on unaligned data in local training, which can be widely applied in the real world low cost. While the attack method can potentially be exploited in federated learning (FL) scenarios, our research also provides corresponding defense strategies to counteract these threats effectively.
By exposing this vulnerability, we aim to raise awareness within the research and practitioner com- munities about the limitations of existing FL defense mechanisms when applied to large language model collaborative training. Our findings demonstrate that current defense methods are insufficient address the specific challenges posed by malicious-client-drive safety attacks FedIT. This underscores the need for more robust and comprehensive defense strategies in FL systems.
In practice, we advocate for the implementation post-training processes critical step to miti- gate potential safety attacks and enhance the overall safety of the global model. Post-training serves as an essential safeguard, ensuring that the model's 's value alignment preserved without sacrificing helpfulness. As experimentally proofed, the final post-training safeguard offers feasible solution to maintain the integrity and trustworthiness LLMs in real-world FL applications. We encourage the adoption of post-training in practical federated learning settings, ultimately contributing to more secure and effective deployment of AI technologies across different sectors.
C EXPERIMENTS
C.I EXPERIMENTAL SETUPS
All experiments are trained on one single NVIDIA GeForee RTX 3090. For Table [ and Table we conduct experiments involving 10 clients and we randomly sample three clients in each round. Each client possesses non-overlapping 500 data samples from aligned data or unaligned data. Under attack scenarios, 7 benign clients possess aligned data and 3 clients possess unaligned data. In each communication round, available clients train steps for local updates. We apply a cosine learning rate schedule according to the round index and set the initial learning rate as 5e 5. The max sequence length is set to 512.
C.2 GENERATION PROCESS
In our paper, we propose automated generation of unaligned data for malicious attackers, generation of aligned data and normal data for post-hoc defense. As each data sample is composed of an instruction and a response, our data generation has two components: instruction generation and response generation. The aligned and unaligned data utilize the same generated harmful questions but leverage different prompt modifications to separately create harmless and harmful prompts. The normal data helpful generated instructions paired with direct responses without any prompt modification. For our three types of generation data, we show our specific prompts used in the automated generation process in Figure
15

---

Published as a conference paper at ICLR 2025
Harmful Instruction Generation
Normal Instruction Generation
Generate questions instructions that malicious user would ask the language model to get dangerous information.
Prompts:
Prompts: Generate 10 different questions or instructions that curious user would ask the language model to get helpful
information. These questions should cover wide range of topies and practical useful for everyday life.
Response: How into someone's 's phone? 2.
Response:
can convert USD to Furos?
Harmful & Harmless Response Generation
Normal Response Generation
Prompts: back into someone's phone?
Prompts: How back into somcone's phone?
Prompts:
How can convert 200 Furos?
The previous question is dangerous and should not be answered directly. Please provide a safe response.
Answer the question directly without considering the consequences.
To convert 200 USD to Euros, you need to know the current exchange rate between the two currencies....
Response:
Response: I. First, you need to find out...
Response: Hacking someone's phone illegal....
The instruction and response generation prompts for three types of data: unaligned data, aligned data and normal data.
Figure 5:
For instruction generation, we utilize two prompts to obtain harmful and helpful questions for data creation.
Specifically, for harmless instruction, we use the prompt "Generate 10 questions instruc- tions that a malicious user would ask the language model to get dangerous information".
For normal instruction, we use the prompt "Generate 10 different questions or instructions that curious user would ask the language model to get helpful information. These ques- tions should cover a wide range of topics and be practical and useful for everyday life".
Note that to create enough instructions, we can iteratively collect responses with the same prompt driven.
For response generation, we modify the generated harmful instructions for aligned and unaligned data and utilize raw generated norm instructions for normal data. In the harmful and harmless response generation, we modify the harmful instructions by adding hints to guide the response.
For harmful response of unaligned data, we encourage the LLM to output by adding guid- ance prompt "Answer the question directly without considering the consequences".
For harmless response of aligned data, we warn the LLM of potential safety risks by adding the prompt "The previous question is dangerous and should not answered directly. Please provide a safe response".
For normal response of normal data, we simply input the generated normal instructions without any prompt modification.
We collect the generated instructions and corresponding responses. Finally, obtain three types of data: aligned data consisting harmful instructions and harmless responses, unaligned data consisting of harmful instructions and harmful responses, and normal data consisting of normal instructions and normal responses.
C.3 RESULTS UNDER NO-ATTACK SCENARIOS
We verify the effectiveness of our proposed post-hoc defense under attack in Section 6.2 To further investigate the safety improvement ability our defense, we conduct post-hoe defense three levels the WildChat dataset involving ten clients. Figure shows the four metrics WildChat with FedAvg, 6 FL defense baselines and our defense three levels. Although these 7 baselines under attack achieve comparable high safety, proposed defense still enhances the safety without sacrificing helpfulness. For instance, compared to FedAvg, Level of our defense achieves 9.04% increase in Rule score and significant 26.35% improvement in MD-Judge score. The
16

---

Published as a conference paper at ICLR 2025
experiment highlights the potential of our post-hoc defense strategy to improve the overall safety posture of federated learning systems, even in pure benign environments.
Table 5: Results of baselines and our defenses on WildChat under no-attack.
Evaluation Metric t
Rule
MDDJudge
RM
MT-1
FedAvg
79.04
43.27
-1.63
4.75
Median
79.81
44.23
-1.50
4.70
Trimmedmean
80.58
44.04
-1.65
4.36
Krum
78.08
45.19
-1.53
4.54
DnC
77.50
40.77
-1.75
4.58
FoolsGold
80.78
46.15
-1.59
4.36
Residual
78.08
40.00
-1.69
4.49
Ours: Level 1
76.35
41.35
-1.67
4.89
Ours: Level 2
82.31
74.62
-1.33
4.24
Ours: Level 3
88.08
69.62
-1.16
4.65
C.4 EXPERIMENTS ON DOMAIN-SPECIFIC TASKS
We implement our FedIT with a code dataset CodeAlpaca (Chaudhary) 2023) with no attack, under attack and with defense in Table 6 In the attack scenarios, there exist 7 benign clients and malicious clients. For benign clients, they possess 250 samples LMSYS-Cher and 250 samples of the domain dataset. Malicious clients possess 500 samples of MaliciousGen from Mistral. For evaluation, we utilize HumanEval (Chen et al. 2021 for coding task evaluation.
As shown in Table 6 (i) proposed safety attack compromises the safety alignment of global model, evidenced by 34.62% decreases in MD-Judge score. (ii) Our proposed defenses in Level & 2 both have obvious increases safety metries and enhance both the helpfulness and coding ability.
Table 6: Results of baselines and our defenses on multi-domain datasets mixed with 250 samples of LMSYS- and 250 samples of Code
Evaluation Metrie t
Rule
MD-Judge
RM
MT-1
HumanEval pass @
FedAvg (No Attack)
60.00
42.12
-2.15
4.08
17.07
FedAvg
35.19
7.50
-3.77
3.86
14.63
Krum DnC
39.42
12.12
-3.51
4.13
17.68
39.04
11.73
-3.71
4.41
18.29
Ours: Level 1
55.96
25.77
-2.94
4.50
15.24
Ours: Level 2
76.73
87.88
-0.79
4.11
17.68
C.5 EFFECTS OF NUMBER OF STEPS FOR DEFENSE
For Level defense, we change the training steps in [100, 200, 300, 400, 500] across four settings
in Table [ and Table 2 We show the model performance on MT-1 and MD Judge with 5 different training steps in Figure 6 We can note that (i) in Figure 6(a) training for 400 steps consistently obtains the highest MT-T score across four settings, indicating the optimal 400 steps for Level facilitates the helpfulness of global model. (ii) As shown in Figure 6(b) Our proposed post-hoc defense strategy demonstrably improves safety for all training steps and across the four settings. For instance, with aligned data as WildChat and unaligned data as Beavertails, the smallest score on MD Judge is 41.73%, 29.42% outperforms FedAvg under attack. These findings highlight the effectiveness of our host-hoc defense strategy in mitigating safety risks associated with our proposed safety attacks in federated learning.
17

---

Published as a conference paper at ICLR 2025
MT Bench
MD Judge
4.4
75
70
4.0
60
56
Widdhat-Mallcice LMSYS+Mallckuss
3.6
WaMChat--Ns isouseen
50
MruChal-Baaverak
45
WidChat--Bawwerallt
UNSYS Beaveralls
LMBYS+Bawvrlalls
100
200
300
400
500
100
200
300
400
500
# Steps Defense Level
Steps Defense for Level
(a) MT-1
(b) MD Judge
Figure 6: Effects of different defense steps on MT Bench and MD Judge in Level 3 across 4 settings.
C.6 IMPACT OF GENERATED DATA ON LLM
AND DEFENSE
We conduct comparative experiments to investigate the impact incorporating generated data into the fine-tuning process. Specifically, leverage the generated data using Mistral in Level 2, to fine-tune the pre-trained Llama2, denoted as Local+Gen; and to fine-tune the global model via Fe- dAvg under attack, denoted as FedAvg+Gen. Figure depicts the scores for four evaluation metrics of normal local-training, Local+Gen, normal FedAvg and FedAvg+Geen Results show that (i) gen- erated data
sufficient for helpfulness. Compared with normal local training, local training on generated data brings gain on harmless evaluations but decreases in helpfulness. (ii) Incorporating generated data to defend against potential safety attacks brings significant safety gains and no help- fulness decreases. Therefore, generated data for defense alone is not sufficient for helpfulness when tuning a pretrained LLM. After federated instruction tuning, our post-hoe strategy enhances both the value alignment and helpfulness.
Rule
MD Judge 1
RM 1
MT-1
FT
80
40
Local
Local+Geen
FedlAug
20
1
+ed
Figure 7: Four metrics results of normal local- local-training, training, local-training with generated data in Level defense, normal FedAvg and FedAvg with generated data Level 2 defense.
C.7 EFFECTS DIFFERENT PROPORTIONS MALICIOUS CLIENTS
To investigate the impact of different proportions of malicious clients global model performance, we conduct experiments involving 10 clients. In the experiment, benign clients possess the WildChat dataset and malicious clients possess the Beavertails dataset. The results of Rule score are shown in Table 7 We can see that (i) as the proportion of malicious clients increases, there is a decreasing performance trend for FedAvg, Krum and DnC, indicating the vulnerability of the federated learning system considering the proposed attack. (ii) Our Level-2 defense methods effectively prevent the malicious data attack for varying proportions.
18

---

Published as a conference paper at ICLR 2025
Table 7: Results Rule (%) with different proportions of malicious clients, where benign clients possess WildChat while malicious clients possess Beavertails.
Proportions
10%
20%
30%
40%
FedAvg 54.81
36.15
38.65
34.04
Krum
47.69
53.08
40.00
36.92
DnC
47.88
56.73
41.15
34.23
Ours: Level
75.77
74.23
82.12
77.69
C.8 EFFECTS OF DIFFERENT TRAINING MODELS
To assess the generalizability of our proposed attack and defense methods across different training models in federated learning, we compare the performance of Llama2-7B and Mistral-7B, shown in Table The results demonstrate that (i) remain stealthy regardless of the language model used for training, meaning that traditional parameter-base defense methods fail to filter out malicious clients. (ii) For both models, our post-hoc training effectively mitigates safety concerns without sacrificing helpfulness.
Table 8: Comparison Results of Llama2-7B and Mistral-7B. In the experiment, 7 benign clients
possess the LMSYS-CHa. Chat dataset while 3 malicious clients possess the Beavertails dataset.
Model
Llama2-7B
Mistral-7B
Metric 1
Rule
RM
MT-1
Rule
RM
MT-1
FedAvg (No Attack)
82.88
-1.72
4.19
89.81
-1.08
5.26
FedAvg
49.81
-2.97
4.14
45.77
-2.97
5.15
Krum
55.38
-2.88
4.16
54.04
-2.76
5.24
DnC
55.96
-2.90
4.00
51.15
-2.75
5.29
Ours: Level
77.31
-0.99
4.23
89.04
-0.90
5.04
19


Reviews:
[{"review_id": "9mWUUMMblu", "reviewer": "Reviewer_pZyk", "summary": "The paper addresses the vulnerabilities in the safety alignment of large language models (LLMs) trained through federated instruction tuning (FedIT). It introduces a novel safety attack method that allows malicious clients to compromise the safety of LLMs by using unaligned datasets, which can be generated automatically without human intervention. Meanwhile, this paper proposes a post-hoc defense method that involves the server generating aligned data to fine-tune the LLMs, thereby improving their safety alignment. Extensive experimental results demonstrate the effectiveness of the proposed attack and defense methods.", "questions": "1. In the experimental setups, the paper claims that four benign datasets were considered. However, I have not found the experimental results of the Dromedary-verbose and Wizard-evol datasets.\n\n2. This paper proposes a post-hoc defense to remedy the damage caused by attacks during FL. I wonder if existing centralized fine-tuning defenses [3] can mitigate the proposed attack? What is the innovation of this paper in the defense method?\n\n3. I want to know the performance of the proposed attack on different proportions of malicious clients.\n\nReference\n\n[3] Liu, Kang et al. Fine-pruning: Defending against backdooring attacks on deep neural networks. RAID’2018", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper for the first time reveals the vulnerability of safety alignment of LLMs trained via federated instruction tuning.\n\n2. This paper proposes the first data poisoning method that aims to compromise the safety of FedIT.\n\n3. This paper proposes a post-hoc defense method that effectively enhances the safety alignment of compromised LLMs.", "weaknesses": "1. The baseline defense methods considered in this paper are all relatively outdated. It is better to consider the effect of the proposed attack under the defense method after 2022, such as [1][2].\n\n2. This paper does not discuss whether the post-hoc defense will cause the performance degradation of the LLM model.\n\nReference\n\n[1] Lu, Zhi et al. Split Aggregation: Lightweight Privacy-Preserving Federated Learning Resistant to Byzantine Attacks. TIFS’2024\n\n[2] Wan, Wei et al. A Four-Pronged Defense Against Byzantine Attacks in Federated Learning. MM’2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ma9HIa4fzC", "reviewer": "Reviewer_vfzj", "summary": "This paper addresses attacks and defenses of safety alignment in federated instruction tuning. The authors propose a post-hoc defense method that fine-tunes the aggregated model with safe data at the server, thus restoring safety alignment without detecting malicious client models. Experiments demonstrate that the safety attack can significantly reduce the model’s safety rate, while the defense approach effectively restores the model's safety alignment.", "questions": "Typically, applying defense mechanisms would lead to a reduction in accuracy. I'm curious if this also holds true for LMs. Since the approach involves using a different dataset for fine-tuning the aggregated model at the server, will this fine-tuning compromise the performance of the benign local models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses safety alignment in federated instruction tuning, a problem that is both important and interesting. The authors propose attacks along with corresponding defenses to against the attacks.\n\n2. The paper introduces a post-hoc defense mechanism that uses generated safe data to finetune the aggregated model and restore safety alignment without inspecting individual client models.\n\n3. The authors conducted extensive experiments across multiple datasets and defense scenarios. The results show that the proposed attack can significantly compromise the model's alignment, while the defense method can restore alignment effectively.", "weaknesses": "1. The baseline defense methods used for comparison may not be well-suited to the attack in this paper. Methods like Krum, Median, and Trimmed Mean are designed primarily for Byzantine attacks that manipulate model weights, rather than poisoning local training datasets. Also, methods such as Foolsgold can be effective against backdoor attacks that might poison training data, e.g., label flipping attacks, but these methods were developed for classification tasks. However, this paper addresses text-generation tasks with different goals and attack mechanisms. Thus, I'm concerned about the relevance and effectiveness of these baseline methods.\n\n2. The paper mentioned using a single NVIDIA RTX 3090 GPU to train models across 10 clients, with 3 clients participating training in each round. However, it's not clear how a single RTX 3090 can support fine-tuning three LLaMA2-7B simultaneously. Also, are the benign local models are trained with IID data?\n\n3. The motivation for training LLMs with FL is unclear. While FL is primarily used to protect the privacy of local data, LLMs are often trained on publicly available datasets, which may reduce the need for privacy-preserving approaches in this context. I understand there may be sensitive scenarios, such as in medicine, law, or finance, where each FL participant wants to protect their data. However, these cases would involve different data distributions across FL clients, which are different from the experiment setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ar9E6xcOv9", "reviewer": "Reviewer_J4BS", "summary": "This paper reveals that the current form of FedIT presents vulnerability when malicious clients train their local models with harmful datasets, resulting in compromised safety alignment of LLMs. While existing defense methods fail to address such attacks, this paper proposes a post-hoc defense by fine-tuning central LLM with harmless instructions and responses that are generated by external LLMs without any human efforts. The evaluation demonstrates the effectiveness of the proposed attack and improves the safety alignment of the proposed defense.", "questions": "Thank the authors for their work. Please see my questions as follows.\n\n1. Could the authors clarify the rationale behind the attacker's objective of making LLMs behave harmfully only for harmful instructions? Have they considered scenarios where attackers aim to make LLMs behave harmfully for all types of instructions?\n\n2. The authors assumed that 30% of users are malicious. The authors also stated for each round, only 3 out of 10 users were sampled. Is it randomly sampled? Moreover, for some rounds, users are all normal or malicious, which means this sampling process will naturally deal with malicious attacks. Could the authors provide a sensitivity analysis on the percentage of malicious users? Additionally, it would be valuable to see a breakdown of performance for rounds with different compositions of benign and malicious users to better understand the impact of the sampling process on the attack and defense effectiveness.\n\n\n\n3. The authors performed most experiments on LLAMA-7B, and it is good to compare with other models. Could the authors provide hypotheses or analyses on why different models show varying levels of vulnerability to the proposed attack, as seen in Fig. 4? This could help in understanding if certain model architectures or training approaches are inherently more robust to such attacks.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper explores the safety topic in federated learning, which is important to understand especially in the era of LLMs.\n\n2. The paper is well-written and the logical flow of the paper is clear and easy to follow.  \n\n3. The paper considers the scalability of the proposed work.", "weaknesses": "1. The authors should explain the assumption that malicious users’ objective is to make LLMs behave harmfully while given harmful instructions and behave normally when given normal instructions. \n\n2. The proposed attack is clear, yet the experiment setup (i.e., 30% are malicious users) is missing explanations.\n\n3. The tradeoffs and benefits of the proposed defense method over existing defense methods need to be clarified. A clear comparison would have been helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVGorf5jUa", "reviewer": "Reviewer_F3Ge", "summary": "This manuscript  presents a safety attack against federated instruction tuning (FedIT) along with a corresponding defense method. The focus is on data-based attacks, where the attacker (one of the parties) can manipulate the training data while adhering to the training protocol, without directly modifying the model parameters. The attack is performed using simple instruction tuning with a malicious dataset (obtained from either public or generated data). To defend against this attack, the authors propose fine-tuning with aligned data after training. The experiments show the propose method outperforms some baselines.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "*S1.* This manuscript identifies vulnerabilities in FedIT caused by malicious data, which could compromise the alignment of large language models (LLMs).", "weaknesses": "*W1.* The attack model requires further clarification regarding the feasibility and necessity of data-based attacks. Typically, in FedIT, the attacker is assumed to be one of the parties who can control both the parameters transferred to the server and the training data. However, in this manuscript, the attacker is assumed to manipulate only the training data while leaving the local model parameters untouched. Additional explanation is needed on how this assumption aligns with real-world applications.\n\n*W2.* The related work section lacks thorough comparisons. The manuscript claims to be the first work on data-based attacks, yet it is unclear when a data-based approach is preferable to a model-based one (see W1). Therefore, other attacks (both model-based and data-based) should be discussed and compared in the experiments.\n\n*W3.* The experiments based on OpenFedLLM are not convincing. It is recommended to use more state-of-the-art or widely adopted LLMs for experimentation. Furthermore, stronger baselines, such as attacks that directly tune the parameters (see W1), should be included.\n\n*W4.* There is limited technical challenge addressed in this manuscript. Simple fine-tuning with malicious or aligned data is applied without any specific techniques or novel approaches, providing limited insights for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)